Buffers, Streams & Backpressure — Node.js Course

What you'll learn

Streams keep memory bounded while data moves through a system. Work with bytes explicitly, transform incrementally, respect backpressure, and use pipeline() so failure and cleanup propagate across the chain.

By the end of this lesson, you'll be able to:

Handle buffers and encodings
Compose readable, transform, and writable streams
Respect backpressure
Propagate cancellation and errors

Core mental model

Node.js becomes easier when you separate the JavaScript language from the runtime and the operating-system capabilities it exposes. Use this table as a decision guide.

Concept	What it means	Decision rule
Buffer	A fixed sequence of bytes	Specify encoding whenever bytes become text
Backpressure	A slow consumer asks a producer to pause	Honor write() false or use pipeline
Transform	A stream mapping chunks to chunks	Keep transformations incremental and bounded

Professional workflow

Build and verify Node.js programs from the terminal in small, observable steps.

Define the streaming pipeline boundary: inputs, outputs, invariants, ownership, and expected failures.
Design the data or message contract before choosing implementation details.
Implement the smallest correct path with dependencies passed explicitly.
Add validation, failure translation, cleanup, and concurrency behavior.
Verify the boundary with realistic data and at least one adversarial case.
Measure or observe the behavior before optimizing or extracting abstractions.

Keep the feedback loop short

Run the smallest useful command after every meaningful change. Read the complete error message before editing again, and keep inputs and outputs visible while you learn.

Guided code lab

Process a large file without buffering it

pipeline() connects completion and failure while the transform keeps only one logical record at a time.

import.js

import { pipeline } from 'node:stream/promises';
import { createReadStream, createWriteStream } from 'node:fs';

await pipeline(
  createReadStream('events.ndjson'),
  splitLines(),
  validateAndRedactEvents(),
  createWriteStream('safe.ndjson'),
  { signal: abortController.signal },
);

Production practice

Contract

Each pipeline defines accepted bytes, encoding, maximum record/chunk behavior, output, cancellation, and partial-output cleanup.

Verification

Use tiny highWaterMark values, slow consumers, malformed chunks, midstream aborts, read/write failures, and files larger than memory.

Operations

Track bytes, throughput, duration, errors, and aborts; cap record size and remove incomplete artifacts.

Common failure mode

Reading an entire upload with readFile or concatenating every chunk defeats streaming and can exhaust memory.

Independent workshop

Build an NDJSON import/export pipeline with validation and progress.

Your finished workshop must include:

Incremental parser
Bounded record size
Transform stage
pipeline() composition
Abort support
Slow-consumer test

Definition of done

Run the happy path and at least two edge cases, keep responsibilities separated, and add a short README explaining how to run the program.

Recap & quick check

Key takeaways

Buffers are bytes
Encoding is explicit
Streams bound memory
Backpressure coordinates speed
pipeline propagates failure

Quick check

1. What does write() returning false mean?

2. Why prefer pipeline()?

3. What breaks bounded memory?

Next: Event Loop, libuv & Async Internals