Lesson 7 — Going to Production

Tied to mission: "deploy and operate in production" — and the whole point: confidence running async workloads on Redis without paging yourself at 3am. Synthesizes L1–L6.

1. Redis must be configured for a queue, not a cache

This is the single most important prod setting, and it's not in your code — it's in Redis.

2. Connections — the maxRetriesPerRequest: null rule

BullMQ uses blocking Redis commands. ioredis's default (maxRetriesPerRequest) blocks BullMQ from working at all. You must null it:

import IORedis from 'ioredis';
const connection = new IORedis({ maxRetriesPerRequest: null });

So you can pass one ioredis instance to a Queue + Worker — but the Worker/QueueEvents internally duplicate() it for their blocking connection. The instance must support duplicate() (ioredis does). Redis connections are cheap; don't over-optimize sharing unless your provider caps you.

Isolate environments with prefix

new Queue('email', { prefix: 'myapp-prod', connection });
// keys become bull:myapp-prod:... instead of bull:email:...

Lets staging + prod share one Redis without colliding — though separate Redis instances is cleaner.

3. Graceful shutdown — worker.close() + your own timeout

On SIGTERM (deploy, k8s rolling update, scale-down), you want in-flight jobs to finish — not become stalled.

let shuttingDown = false;

async function shutdown() {
  if (shuttingDown) return;
  shuttingDown = true;
  console.log('draining worker...');
  await worker.close();   // stops picking new jobs; waits for active ones
  await queue.close();
  await connection.quit();
  process.exit(0);
}

process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown);

4. Concurrency vs parallelism — don't conflate them

CPU-bound work → sandboxed processors

If a job blocks the event loop (image resize, crypto, big parses), it starves the heartbeat → stalls (Lesson 6). Move it to a sandboxed processor — runs in a separate process or thread:

// worker.ts — the host
import { Worker } from 'bullmq';
new Worker('images', undefined, {
  connection,
  useWorkerThreads: true,        // v3.13+: threads (lighter) instead of spawn
  authorURL: import.meta.url,    // points to the processor file
});

// processor.ts — separate file, runs in the sandbox
import { parentPort } from 'worker_threads';
module.exports = async (job) => { /* heavy CPU here, isolated */ };

Crash in the sandbox doesn't kill the worker; the event loop of the host stays free to renew locks.

5. Make jobs idempotent

Retries (L2), rate-limit re-queues (L4), stalled re-runs (L6), and flow resumption (L5) all mean the same job can execute more than once. Design for it:

• Check for prior work before acting (a "processed" flag, an upsert on a unique key).
• Make side-effects commutative / dedupable (idempotency keys on payment APIs).
• Fetch → mutate → write as a single transaction, or use optimistic locking.

6. The production checklist

• ☑ maxmemory-policy noeviction + AOF persistence in Redis.
• ☑ Every ioredis instance created with maxRetriesPerRequest: null.
• ☑ removeOnComplete + removeOnFail on every queue (L6).
• ☑ SIGTERM/SIGINT handlers calling worker.close() + connection.quit().
• ☑ Idempotent processors.
• ☑ concurrency tuned for I/O; extra worker processes for CPU (+ sandboxed if blocking).
• ☑ Metrics enabled + stalled event wired to an alert (L6).
• ☑ upsertJobScheduler called at boot — idempotent, safe on every deploy (L3).
• ☑ A dead-letter queue for terminal failures (L2).
• ☑ Top-level process.on('uncaughtException') / unhandledRejection handlers — log and exit, don't leave a zombie.
• ☑ Redis unreachable at startup should fail fast and let the orchestrator restart you, not hang silently.

7. Full picture — a production-shaped service

import { Queue, Worker, QueueEvents, MetricsTime } from 'bullmq';
import IORedis from 'ioredis';

const connection = new IORedis({ maxRetriesPerRequest: null });

const email = new Queue('email', {
  connection,
  defaultJobOptions: {
    attempts: 5,
    backoff: { type: 'exponential', delay: 2000, jitter: 0.5 },
    removeOnComplete: { age: 86_400, count: 2000 },
    removeOnFail:    { age: 604_800, count: 5000 },
  },
});

// recurring digest — safe to call every boot
await email.upsertJobScheduler('daily-digest',
  { pattern: '0 0 9 * * 1-5' },
  { name: 'digest', data: {} });

const worker = new Worker('email', async (job) => {
  if (await alreadySent(job.id)) return;        // idempotent
  await sendMail(job.data);
}, {
  connection,
  concurrency: 20,                              // I/O-bound → high
  stalledInterval: 30_000,
  maxStalledCount: 1,
  metrics: { maxDataPoints: MetricsTime.ONE_WEEK * 2 },
});
worker.on('stalled', (id) => alert('stalled', id));

let shuttingDown = false;
async function shutdown() {
  if (shuttingDown) return; shuttingDown = true;
  await worker.close(); await connection.quit(); process.exit(0);
}
process.on('SIGTERM', shutdown).on('SIGINT', shutdown);

What you just learned

• Redis must be noeviction + AOF, or it'll lose/evict jobs.
• Every ioredis instance needs maxRetriesPerRequest: null; blocking cmds are internal duplicates.
• worker.close() on SIGTERM — but it doesn't self-timeout; race it against your grace period.
• concurrency is for I/O; scale worker processes for CPU; sandbox blocking work.
• Make jobs idempotent — retries, stalls, and rate-limits all re-run.
• The 12-item prod checklist is your pre-launch gate.

🧠 Quiz — 10 questions

📚 Primary sources

📎 Reference

💬 Where to go from here

• Wisdom: take a real question to the BullMQ Discord or GitHub Discussions — operators trade war stories there.
• Practice: instrument a real queue with metrics + a stalled alert before you need it.
• Deeper: BullMQ architecture & internals — the Redis data structures behind everything you just learned.
• Review: re-run a past quiz to lock in storage strength. Say "quiz me on flows" or "quiz me on retries".

This is the end of the core curriculum. Ask me anything — or tell me a new direction and I'll plan the next arc.