Lesson 2 — Retries, Backoff & Dead Letters

Tied to mission: a worker that crashes on the first transient error is useless. This lesson turns "fire and hope" into "fire, retry smart, and surface the unfixable." Builds directly on Lesson 1.

When is a job "failed"?

BullMQ marks a job failed in exactly two cases:

1. The processor threw an Error (not a string, not a literal — an actual Error object).
2. The job went stalled (the worker process died or blocked the event loop) and exhausted its maxStalledCount.

The two options that do 90% of the work

Built-in backoff strategies

await queue.add('charge-card', { orderId: 99 }, {
  attempts: 5,
  backoff: { type: 'exponential', delay: 1000 },
});

Timeline for delay: 1000, all attempts failing:

attempt 1  → fail  → wait 1000ms
attempt 2  → fail  → wait 2000ms
attempt 3  → fail  → wait 4000ms
attempt 4  → fail  → wait 8000ms
attempt 5  → fail  → FAILED (permanent), enters failed set

Jitter — stop the thundering herd

Imagine 100 jobs fail at the same instant because a downstream API blipped. Without jitter, all 100 retry on the same exponential schedule — they hit the API in a synchronized wave and fail again. Jitter randomises each delay to spread the load.

backoff: { type: 'exponential', delay: 1000, jitter: 0.5 }

jitter is 0–1. 0.5 means "anywhere between 50% and 100% of the computed delay." Use 0.5 as a sane default whenever many jobs can fail together.

Set defaults on the queue, override per job

const queue = new Queue<EmailPayload>('email', {
  connection,
  defaultJobOptions: {
    attempts: 5,
    backoff: { type: 'exponential', delay: 2000, jitter: 0.5 },
    removeOnComplete: 1000,   // keep last 1000 completed
    removeOnFail: 5000,       // keep last 5000 failed for debugging
  },
});

// override for the one job that needs more tries
await queue.add('send-critical', data, { attempts: 20 });

Inspecting failure inside the processor

The processor can behave differently based on how many times it has already tried. job.attemptsMade starts at 1 on the first run.

async (job: Job) => {
  try {
    await callFlakyApi(job.data.payload);
  } catch (e) {
    if (job.attemptsMade < job.opts.attempts) {
      console.log(`attempt ${job.attemptsMade} failed, will retry:`, (e as Error).message);
    }
    throw e;  // always re-throw so BullMQ handles the retry
  }
}

Custom backoff — when fixed/exponential aren't enough

Define the strategy on the worker (not the queue), then reference it by name when adding jobs.

new Worker<EmailPayload>('email', processor, {
  connection,
  settings: {
    backoffStrategy: (attemptsMade: number) => {
      // cap at 60s, with mild jitter
      const base = Math.min(60_000, 2 ** (attemptsMade - 1) * 1000);
      return base + Math.random() * 1000;
    },
  },
});

await queue.add('x', data, {
  attempts: 8,
  backoff: { type: 'custom' },  // uses backoffStrategy above
});

Return values are special: 0 → push to back of waiting; -1 → fail immediately, no more retries.

Dead-letter queue (the production pattern)

BullMQ has no built-in DLQ. The standard pattern: when a job exhausts its retries, listen for the final failure and re-enqueue it onto a separate queue for human inspection, alerting, or replay.

import { Queue, QueueEvents } from 'bullmq';

const dlq = new Queue('email-dead', { connection });
const qe = new QueueEvents('email', { connection });

qe.on('failed', async ({ jobId, failedReason, prev }) => {
  // fires on EVERY failure, so only act on the final one
  if (prev !== 'active') return;

  const job = await emailQueue.getJob(jobId);
  if (!job) return;

  // final attempt? attemptsMade === attempts means no more retries
  if (job.attemptsMade >= (job.opts.attempts ?? 1)) {
    await dlq.add(job.name, {
      originalId: job.id,
      data: job.data,
      reason: failedReason,
      attempts: job.attemptsMade,
      failedAt: new Date().toISOString(),
    });
    // optional: await job.remove();  // keep main queue clean
  }
});

Stalled jobs — when the worker itself dies

While a job is active, the worker renews a lock every stalledInterval ms. If the process crashes, or blocks the event loop (CPU-bound work, infinite loop), the lock isn't renewed. After stalledInterval BullMQ notices, moves the job back to waiting, and another worker picks it up — unless it has stalled maxStalledCount times, in which case it goes to failed.

new Worker('email', processor, {
  connection,
  stalledInterval: 30_000,   // default 30s — check locks this often
  maxStalledCount: 1,        // default 1 — after this many stalls, fail it
});

Lesson: keep processors async and I/O-bound. If you need to do CPU-heavy work (image resize, crypto), use sandboxed processors (separate threads) — covered in a later lesson.

Manual cleanup tools (housekeeping)

await queue.drain();              // remove waiting + delayed (not active/completed/failed)
await queue.drain(true);          // also delayed
await queue.clean(60_000, 1000, 'failed');  // remove failed older than 60s, max 1000
await queue.obliterate();         // wipe everything — no undo, queue gone

What you just learned

• attempts + backoff are the retry engine; exponential + jitter is the default.
• Always throw Error objects, never strings.
• Set removeOnComplete/removeOnFail or Redis grows unbounded.
• The failed event fires per-attempt; check attemptsMade before alerting.
• DLQ is a pattern, not a feature: re-enqueue terminal failures to a separate queue.
• Stalled jobs = worker didn't renew its lock. Keep processors async.

🧠 Quiz — 10 questions

📚 Primary source

📎 Reference

💬 Ask me next

• "How do I retry a job that's already permanently failed?"
• "Show me a real DLQ with alerting wired in."
• "What's a sandboxed processor and when do I need one?"
• "My worker keeps stalling — how do I debug it?"

Or say "next" for Lesson 3: delayed & scheduled (cron) jobs.