Queues & Background Jobs — Node.js Course

What you'll learn

Queues move durable work out of request latency, but most brokers deliver at least once. Design idempotent handlers, bounded retries with backoff, dead-letter inspection, scheduling, and observable worker shutdown.

By the end of this lesson, you'll be able to:

Model job envelopes and states
Make handlers idempotent
Classify retryable failures
Operate retries and dead letters

Core mental model

Node.js becomes easier when you separate the JavaScript language from the runtime and the operating-system capabilities it exposes. Use this table as a decision guide.

Concept	What it means	Decision rule
At-least-once	A job may be delivered again	Make effects idempotent using stable job or business keys
Backoff	Increasing delay between attempts	Add jitter and cap attempts to avoid synchronized retry storms
Dead letter	A terminally failed job retained for inspection	Store reason and safe replay workflow

Professional workflow

Build and verify Node.js programs from the terminal in small, observable steps.

Define the job delivery boundary boundary: inputs, outputs, invariants, ownership, and expected failures.
Design the data or message contract before choosing implementation details.
Implement the smallest correct path with dependencies passed explicitly.
Add validation, failure translation, cleanup, and concurrency behavior.
Verify the boundary with realistic data and at least one adversarial case.
Measure or observe the behavior before optimizing or extracting abstractions.

Keep the feedback loop short

Run the smallest useful command after every meaningful change. Read the complete error message before editing again, and keep inputs and outputs visible while you learn.

Guided code lab

Claim an idempotency key atomically

A stable business key ensures a retried notification does not produce duplicate effects.

send-receipt.js

export async function handleReceipt(job) {
  const key = 'receipt:' + job.orderId;
  const claimed = await idempotency.tryBegin(key, job.id);
  if (!claimed) return { duplicate: true };
  try {
    await mailer.sendReceipt(job.orderId);
    await idempotency.complete(key);
  } catch (error) {
    await idempotency.releaseOrFail(key, error);
    throw error;
  }
}

Production practice

Contract

Each job declares schema version, stable identity, idempotency key, timeout, retry class, maximum attempts, and terminal handling.

Verification

Deliver duplicates, crash after side effect, time out dependencies, reorder jobs, poison a payload, and terminate a busy worker.

Operations

Track queue depth, oldest age, run time, attempt count, failure class, and dead letters; drain gracefully during deploy.

Common failure mode

Retrying every error forever converts permanent bad input into a self-sustaining outage.

Independent workshop

Build a durable email/export worker for the task platform.

Your finished workshop must include:

Versioned job envelope
Idempotent handler
Retry classifier
Backoff with jitter
Dead-letter workflow
Queue dashboards

Definition of done

Run the happy path and at least two edge cases, keep responsibilities separated, and add a short README explaining how to run the program.

Recap & quick check

Key takeaways

Delivery can repeat
Idempotency protects effects
Retries are bounded
Dead letters need ownership
Queue age is a key signal

Quick check

1. What does at-least-once imply?

2. Which errors should retry?

3. What often matters more than depth?

Next: Architecture, Dependency Injection & Patterns