Node.js Streams: How They Work & When to Use

Node.js streams are objects that let you read or write data piece by piece, instead of loading the entire data into memory at once. You use them to process large files, network responses, or any data that arrives over time, while keeping memory usage low. This article explains what streams are, how they work, the different types, and how to use them effectively.

Formula

The calculator above is a pace calculator, not a stream calculator, but it demonstrates the kind of streaming data processing you might do with Node.js. The formula for pace is:

pace = time / distance

Where time is in minutes and distance is in kilometers. The result is the average time per kilometer. The speed is calculated as distance / (time / 60) to get kilometers per hour.

Worked Example

Using the default values: time = 50 minutes, distance = 10 km.

1. Pace = 50 / 10 = 5.00 min/km
2. Speed = 10 / (50 / 60) = 10 / 0.8333 = 12.00 km/h

The calculator shows 5.00 min/km and 12.00 km/h.

What Are Node.js Streams?

In Node.js, a stream is an abstract interface for working with streaming data. Instead of reading a whole file into memory, a stream lets you process data as it arrives. This is crucial for handling large files, like videos or logs, because reading a 2 GB file into memory would likely crash your application. Streams are used throughout Node.js: for HTTP requests and responses, file reading and writing, and process input/output.

Types of Streams

Node.js has four fundamental stream types:

• Readable – sources from which data can be read (e.g., fs.createReadStream, HTTP request).
• Writable – destinations to which data can be written (e.g., fs.createWriteStream, HTTP response).
• Duplex – both readable and writable (e.g., a TCP socket).
• Transform – a duplex stream that can modify data as it passes through (e.g., zlib.createGzip).

How Streams Work

Streams work by emitting events and using buffers. When you read from a readable stream, it emits data events with chunks of data. The size of each chunk is determined by the internal buffer size (default 64KB for files, but can vary). You can also use pipe() to connect a readable stream to a writable stream, automatically managing flow and backpressure.

Backpressure is a key concept: if the writable stream cannot accept data as fast as the readable stream produces it, Node.js will pause the readable stream to avoid memory issues. The pipe() method handles this automatically, but if you use events manually, you need to respect the return value of write().

How to Use Streams

Here’s a simple example of copying a file using streams:

const fs = require('fs');
const readStream = fs.createReadStream('input.txt');
const writeStream = fs.createWriteStream('output.txt');
readStream.pipe(writeStream);

To transform data, you can use a transform stream:

const { Transform } = require('stream');
const upperCase = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  }
});
process.stdin.pipe(upperCase).pipe(process.stdout);

This reads from standard input, converts to uppercase, and writes to standard output.

Common Problems and Mistakes

• Not handling errors – streams emit ‘error’ events; if you don’t listen, your app may crash.
• Ignoring backpressure – when writing manually, check the return of write() and wait for the ‘drain’ event.
• Using strings without encoding – by default, streams work with Buffers. If you need strings, set an encoding like readStream.setEncoding('utf8').
• Piping to a writable that is already closed – always handle ‘close’ or ‘finish’ events appropriately.

Streams vs. Other Methods

Streams are not always the best choice. For small files (a few KB), using fs.readFile is simpler and faster. Streams shine when you have large data or need to process data incrementally. They also allow you to start processing before the entire data is available, which is useful for real-time applications like chat or video streaming.

Safety and Best Practices

When working with streams, always ensure you handle errors to avoid unhandled exceptions. Use pipeline() from the stream module (available in Node.js 10+) instead of pipe() for better error propagation:

const { pipeline } = require('stream');
pipeline(readStream, writeStream, (err) => {
  if (err) console.error('Pipeline failed', err);
  else console.log('Pipeline succeeded');
});

Also, be aware of memory usage: even with streams, if you don’t consume data quickly, buffers can grow. Always set appropriate highWaterMark values if you need different buffer sizes.

Frequently Asked Questions

What is the difference between pipe() and pipeline()?

pipe() is the older method, but it does not forward errors from the writable stream to the readable stream. pipeline() was introduced to handle errors properly and also cleans up streams when an error occurs. It is recommended to use pipeline() in modern Node.js.

How do I read a stream as a string?

You can collect the chunks and join them, or use the setEncoding() method to get strings instead of Buffers. For example: readStream.setEncoding('utf8').

Can I pause and resume a stream?

Yes, readable streams have pause() and resume() methods. When paused, the stream stops emitting ‘data’ events. This is useful for controlling flow.

What is the default buffer size (highWaterMark)?

For readable and writable streams, the default is 64KB (65536 bytes). For process.stdout and process.stderr, it’s usually 0, meaning they are unbuffered.

Are streams only for files?

No, streams are used for many things: HTTP requests and responses, TCP sockets, child process stdin/stdout, and even custom data sources. They are a general-purpose abstraction.

How do I create a custom readable stream?

You can extend the Readable class and implement the _read() method, or use the constructor with a read method. For example: new Readable({ read() { this.push('data'); this.push(null); } }).

Related

• Unhide Chat in Viber: Step-by-Step Guide
• Retries, Timeouts, and Circuit Breakers: When to Use Each
• Clipboard Access Denied: Error Explained

in t | Finish time | min | 50 | 0
in d | Distance | km | 10 | 0 | 100000
out pace | Average pace | min/km | t / d | 2
out speed | Average speed | km/h | d / (t / 60) | 2