Files, Buffers & Streams — JavaScript Course

What you'll learn

Node file APIs handle small data conveniently; streams handle large or continuous data with bounded memory. Backpressure keeps producers from overwhelming consumers.

By the end of this lesson, you'll be able to:

Use promise-based file operations
Explain buffers and encodings
Build backpressure-aware pipelines

Core mental model

Use this decision table as a compact reference. Focus on what each tool means and when it earns its place in production code.

Concept	What it means	Decision rule
Buffer	Raw byte sequence	Decode only when bytes represent known text
Stream	Incremental readable or writable data source	Use when data may be large or continuous
Backpressure	Consumer signals that producer should slow	Use pipeline instead of manual unchecked writes

Professional workflow

Build the behavior in small, observable steps. Each step should leave something you can inspect or test.

Describe the file and stream pipeline boundary: inputs, outputs, state, timing, and expected failures.
Implement the smallest correct path with names that expose intent.
Add edge cases and failure handling before introducing abstractions.
Verify behavior with realistic data and one deliberately adversarial example.
Refactor only after the observable behavior is protected.

Make behavior observable

Before optimizing or abstracting, make inputs, outputs, state changes, timing, and failure paths visible. JavaScript becomes much easier to reason about when hidden work is exposed.

Guided code lab

Read and write a small JSON file

Explicit UTF-8 decoding produces text; a temporary path can support atomic replacement.

settings.js

import { readFile, writeFile } from "node:fs/promises";
const settings = JSON.parse(await readFile("settings.json", "utf8"));
settings.lastCourse = "javascript";
await writeFile("settings.json.tmp", JSON.stringify(settings, null, 2));

Transform a large file safely

pipeline connects error handling, cleanup, and backpressure.

pipeline.js

import { createReadStream, createWriteStream } from "node:fs";
import { Transform } from "node:stream";
import { pipeline } from "node:stream/promises";
const uppercase = new Transform({
  transform(chunk, encoding, callback) {
    callback(null, chunk.toString().toUpperCase());
  },
});
await pipeline(createReadStream("input.txt"), uppercase, createWriteStream("output.txt"));

Production practice

Contract

Make the file and stream pipeline boundary explicit with validated inputs, structured outputs, owned resources, and stable failures.

Verification

Exercise normal work, invalid input, dependency failure, concurrency, and graceful cleanup in automated tests.

Operations

Use structured logs, health signals, timeouts, and configuration that can change without editing source code.

Common failure mode

Reading an unbounded upload with readFile or concatenating every chunk defeats streaming and can exhaust process memory.

Independent workshop

Create a streaming CSV course-progress report.

Your finished workshop must include:

Incremental input processing
A transform stage
Pipeline error handling and partial-output cleanup

Definition of done

Demonstrate the happy path and at least two edge cases, keep responsibilities separated, and add a short note explaining one design choice.

Recap & quick check

Key takeaways

Buffers represent bytes
Encoding turns bytes into text
Streams bound memory
pipeline coordinates errors and backpressure

Quick check

1. When should you prefer a stream?

2. What is backpressure?

3. What does a Buffer contain?

Keep the workshop. Later modules deliberately build on these decisions, so each exercise can become part of your final portfolio architecture.