Interactive Explainer

Relevant for:JuniorMid-levelSenior

Why this matters at your level

Junior

Add a simple email send using Resend SDK. Understand why sending email synchronously in the request handler is wrong.

Mid-level

Implement idempotency keys + SSE + Outbox Pattern. Know the difference between at-least-once and exactly-once delivery. Add DLQ monitoring.

Senior

Design the full async architecture. Add DLQ monitoring with alerts, retry policies, back-pressure for SSE reconnect storms. Explain the Redis fallback strategy.

Build Challenge: Add Real-Time + Email to the Job Board

The job board from Level 1 is live. Now the CEO wants applicants to know about new job postings instantly and receive an email when they apply. You have one week. No double emails allowed.

~5 min read

Be the first to complete!

LEVEL 2 BUILD CHALLENGE

The most common upgrade to a working product: make it real-time and add notifications. This is where most junior engineers introduce two production bugs: duplicate email sends (because the API is called twice on retry) and race conditions between the SSE connection and the initial data load. Both bugs are invisible in development. Both appear immediately in production. This challenge is that upgrade — made safe.

The question this raises

How do you guarantee exactly-once email delivery when users retry failed requests?

Test your assumption first

A user submits a job application. The network drops after the server saves to the DB but before the browser gets the response. The browser shows "Network Error." The user clicks Apply again. There is no idempotency key and no unique constraint on (user_id, job_id). What happens?

Lesson outline

Before the upgrade: two desks, one bug

How this concept changes your thinking

Situation

Before

After

User applies for a job — network drops mid-request

“Frontend sends POST /api/apply. Network hiccup. Response lost. Frontend shows "Network Error." User clicks Apply again. Two applications submitted. Two confirmation emails sent. Two entries in the DB. Recruiter sees the same candidate applied twice.”

“Idempotency key: frontend generates a UUID on component mount. Includes it as Idempotency-Key header on every request. Server checks Redis: has this key been processed? Yes -> return cached response. No -> process and cache result for 24 hours. One email regardless of how many retries.”

New job posted — applicants should see it instantly

“Frontend polls GET /api/jobs every 10 seconds. 1,000 users = 6,000 requests/minute at idle. Postgres CPU at 30% just from polling. Every poll hits the DB even when nothing changed.”

“Server-Sent Events: client opens GET /api/jobs/stream. Server pushes new job events as they happen. 1,000 users = 1,000 persistent connections, zero DB polling. Redis pub/sub: when admin posts a job, PUBLISH jobs-channel newJob. SSE workers subscribed to the channel push to all connected clients.”

Email confirmation added to the apply flow

“POST /api/apply: save to DB, send email via Resend (synchronous). Resend API times out (5s). Request fails. DB has the application. Email was not sent. User gets error but application IS saved. User retries. Duplicate application.”

“POST /api/apply: save application + save outbox event in ONE transaction. Return 201 immediately (50ms). Background BullMQ worker reads outbox, sends email, marks sent. If email fails: worker retries 3 times. User always gets 201. Email always arrives eventually.”

The 3 bugs every recruiter finds

3 bugs every recruiter finds in a "real-time + email" feature

1. Double email: retry with no idempotency key sends the same email twice. Fix: Idempotency-Key header + Redis dedup cache. 2. Missing job update: SSE connection opens after the job was published — client misses the event. Fix: send current state on SSE connect, then stream deltas. 3. Email success but application not saved: email sent before DB transaction commits. Fix: Outbox Pattern — DB write first, email after commit.

What both sides tell you

The browser console shows a duplicate POST request on retry. The DB has two application rows. Redis has the idempotency key from the first request — but nobody checked it. The SSE stream shows a connection opened 3 seconds after the job was published. Reading all three reveals: no idempotency, no initial state on SSE connect.

sse-and-idempotency.ts

1// GET /api/jobs/stream — SSE endpoint with Redis pub/sub
2import { redis } from '@/lib/redis';
3import { db } from '@/lib/db';
4 
5export async function GET(req: Request) {
6  const encoder = new TextEncoder();
7 
8  const stream = new ReadableStream({
9    async start(controller) {
10      const send = (data: object) => {
11        controller.enqueue(encoder.encode(`data: ${JSON.stringify(data)}
12 
13`));
14      };
15 
16      // CRITICAL: Send current state on connect (prevents missed events)
17      // Client connecting 3s after a job was posted would miss it otherwise
18      const existingJobs = await db.job.findMany({
19        orderBy: { createdAt: 'desc' },
20        take: 20,
21      });
22      send({ type: 'initial', jobs: existingJobs });
23 
24      // Subscribe to Redis pub/sub for new job events
25      const subscriber = redis.duplicate();
26      await subscriber.subscribe('jobs-channel', (message) => {
27        const job = JSON.parse(message);
28        send({ type: 'new-job', job }); // push to all connected clients
29      });
30 
31      // Cleanup on disconnect
32      req.signal.addEventListener('abort', async () => {
33        await subscriber.unsubscribe('jobs-channel');
34        await subscriber.quit();
35        controller.close();
36      });
37    },
38  });
39 
40  return new Response(stream, {
41    headers: {
42      'Content-Type': 'text/event-stream',
43      'Cache-Control': 'no-cache',
44      'Connection': 'keep-alive',
45    },
46  });
47}
48 
49// Idempotency key middleware
50export async function checkIdempotency(req: Request) {
51  const idempotencyKey = req.headers.get('Idempotency-Key');
52  if (!idempotencyKey) return null; // key not provided — no dedup
53 
54  const cached = await redis.get(`idem:${idempotencyKey}`);
55  if (cached) {
56    // Already processed — return the cached response
57    return Response.json(JSON.parse(cached), { status: 200 });
58  }
59  return null; // not yet processed — proceed normally
60}
61 
62export async function storeIdempotencyResult(key: string, result: object) {
63  // Cache the result for 24 hours
64  await redis.setex(`idem:${key}`, 86400, JSON.stringify(result));
65}

outbox-email-worker.ts

1// Outbox Pattern + BullMQ email worker
2import { db } from '@/lib/db';
3import { Queue, Worker } from 'bullmq';
4import { Resend } from 'resend';
5import { redis } from '@/lib/redis';
6 
7const emailQueue = new Queue('emails', { connection: redis });
8const resend = new Resend(process.env.RESEND_API_KEY);
9 
10// POST /api/apply — atomic DB write + outbox event
11export async function submitApplication(data: ApplicationInput) {
12  const idempotencyKey = data.idempotencyKey;
13 
14  const application = await db.$transaction(async (tx) => {
15    // Save application
16    const app = await tx.application.create({
17      data: { jobId: data.jobId, email: data.email, resumeUrl: data.resumeUrl },
18    });
19 
20    // Outbox event in the SAME transaction (atomic)
21    await tx.outboxEvent.create({
22      data: {
23        id: idempotencyKey,       // use idempotency key as outbox event ID
24        topic: 'email.confirmation',
25        payload: JSON.stringify({ applicationId: app.id, email: data.email }),
26        processedAt: null,
27      },
28    });
29 
30    return app;
31  });
32 
33  return { applicationId: application.id };
34}
35 
36// Outbox relay: reads committed events, enqueues to BullMQ
37async function runOutboxRelay() {
38  const events = await db.outboxEvent.findMany({
39    where: { processedAt: null },
40    take: 50,
41  });
42 
43  for (const event of events) {
44    await emailQueue.add(event.topic, JSON.parse(event.payload), {
45      jobId: event.id,          // BullMQ dedup: same jobId = same job (not re-added)
46      attempts: 3,
47      backoff: { type: 'exponential', delay: 2000 },
48    });
49    await db.outboxEvent.update({
50      where: { id: event.id },
51      data: { processedAt: new Date() },
52    });
53  }
54}
55 
56// Email worker: idempotent send via Resend
57const emailWorker = new Worker('emails', async (job) => {
58  const { applicationId, email } = job.data;
59 
60  await resend.emails.send({
61    from: 'jobs@company.com',
62    to: email,
63    subject: 'Application received',
64    html: `<p>Application ${applicationId} received.</p>`,
65    headers: {
66      'X-Idempotency-Key': applicationId, // Resend deduplicates on their side too
67    },
68  });
69}, {
70  connection: redis,
71  concurrency: 5,
72});
73 
74emailWorker.on('failed', (job, err) => {
75  console.error(`[DLQ] Email job ${job?.id} failed after all retries: ${err.message}`);
76  // Alert on-call if DLQ has > 0 jobs
77});

4 Features of the Upgrade

1. Server-Sent Events for real-time job feed — No polling. Redis pub/sub fan-out to all connected clients. Initial state sent on connect so no events are missed.
2. Idempotency key on application submission — UUID generated on component mount, sent as Idempotency-Key header. Redis caches result for 24h. No double-processing on retry.
3. Outbox pattern for email confirmation — Application save + outbox event in one Prisma transaction. Relay publishes to BullMQ only after commit. No lost emails on server crash.
4. Resend email with idempotency key — Application UUID sent as X-Idempotency-Key to Resend. Even if BullMQ worker runs twice, Resend deduplicates on their side. Defense in depth.

How Stripe handles this

Stripe requires an idempotency key on every payment API call. Their documentation explicitly states: "Generate the idempotency key on the client before the request. Never generate it on the server." This is because if the request fails, the client must retry with the same key — which is only possible if the client generated and stored it before the first attempt.

The interview question: "What if Redis goes down?"

Prepared answer: if Redis is down, the idempotency cache is unavailable. Fallback: check the DB for an existing application (same user_id + job_id = duplicate). Slower (DB query vs Redis get) but safe. The Redis cache is a performance optimization, not the only protection against duplicates. This shows defense in depth: Redis first, DB constraint as fallback.

production-apply-flow.ts

1// Complete production apply flow with idempotency
2'use client';
3import { useRef } from 'react';
4 
5// Generate idempotency key once on component mount
6// NOT on button click — retries must use the same key
7export function ApplyButton({ jobId }: { jobId: string }) {
8  const idempotencyKey = useRef(crypto.randomUUID());
9 
10  async function handleApply() {
11    const res = await fetch('/api/apply', {
12      method: 'POST',
13      headers: {
14        'Content-Type': 'application/json',
15        'Idempotency-Key': idempotencyKey.current, // same key on every retry
16      },
17      body: JSON.stringify({ jobId }),
18    });
19 
20    if (!res.ok && res.status !== 200) {
21      // User can retry — same idempotency key prevents duplicate
22      console.error('Apply failed, safe to retry');
23    }
24  }
25 
26  return <button onClick={handleApply}>Apply</button>;
27}
28 
29// Server: POST /api/apply
30export async function POST(req: Request) {
31  const idempotencyKey = req.headers.get('Idempotency-Key');
32 
33  // Check idempotency cache first
34  if (idempotencyKey) {
35    const cached = await redis.get(`idem:${idempotencyKey}`);
36    if (cached) return Response.json(JSON.parse(cached)); // return cached result
37  }
38 
39  const { jobId } = await req.json();
40  const session = await auth();
41  if (!session) return Response.json({ error: 'Unauthorized' }, { status: 401 });
42 
43  // Atomic: application + outbox event in one transaction
44  const result = await db.$transaction(async (tx) => {
45    const app = await tx.application.create({
46      data: { jobId, userId: session.user.id },
47    });
48    await tx.outboxEvent.create({
49      data: {
50        id: idempotencyKey ?? crypto.randomUUID(),
51        topic: 'email.confirmation',
52        payload: JSON.stringify({ applicationId: app.id, email: session.user.email }),
53        processedAt: null,
54      },
55    });
56    return { applicationId: app.id };
57  });
58 
59  // Cache idempotency result for 24h
60  if (idempotencyKey) {
61    await redis.setex(`idem:${idempotencyKey}`, 86400, JSON.stringify(result));
62  }
63 
64  return Response.json(result, { status: 201 });
65}

Exam Answer vs. Production Reality

1 / 3

SSE vs WebSockets

📖 What the exam expects

SSE is one-way (server to client). WebSockets are bidirectional.

Toggle between what certifications teach and what production actually requires

How this might come up in interviews

Build challenges at the mid level test whether you understand async reliability patterns — not just whether the feature works in the happy path.

Common questions:

How do you prevent duplicate emails when a user retries an application?
What happens if the SSE connection opens after a job was posted?
How does the Outbox Pattern guarantee email delivery even if the server crashes?
What if Redis goes down — does your idempotency guarantee break?

Strong answers include:

Generates idempotency key on client mount (not on button click)
Sends initial state on SSE connect to prevent missed events
Uses DB transaction for application + outbox event (atomic)
Has a fallback for Redis failure (DB unique constraint)

Red flags:

Sends email synchronously in the POST handler
Generates idempotency key on the server
No initial state on SSE connect
No DLQ or retry policy for failed email sends

Ready to see how this works in the cloud?

Switch to Career Paths for structured paths (e.g. Developer, DevOps) and provider-specific lessons.

View role-based paths

Discussion

Questions? Discuss in the community or start a thread below.

Join Discord

In-app Q&A

Before the upgrade: two desks, one bug

How this concept changes your thinking

Situation

Before

After

User applies for a job — network drops mid-request

New job posted — applicants should see it instantly

“Frontend polls GET /api/jobs every 10 seconds. 1,000 users = 6,000 requests/minute at idle. Postgres CPU at 30% just from polling. Every poll hits the DB even when nothing changed.”

Email confirmation added to the apply flow

The 3 bugs every recruiter finds

3 bugs every recruiter finds in a "real-time + email" feature

What both sides tell you

sse-and-idempotency.ts

1// GET /api/jobs/stream — SSE endpoint with Redis pub/sub
2import { redis } from '@/lib/redis';
3import { db } from '@/lib/db';
4 
5export async function GET(req: Request) {
6  const encoder = new TextEncoder();
7 
8  const stream = new ReadableStream({
9    async start(controller) {
10      const send = (data: object) => {
11        controller.enqueue(encoder.encode(`data: ${JSON.stringify(data)}
12 
13`));
14      };
15 
16      // CRITICAL: Send current state on connect (prevents missed events)
17      // Client connecting 3s after a job was posted would miss it otherwise
18      const existingJobs = await db.job.findMany({
19        orderBy: { createdAt: 'desc' },
20        take: 20,
21      });
22      send({ type: 'initial', jobs: existingJobs });
23 
24      // Subscribe to Redis pub/sub for new job events
25      const subscriber = redis.duplicate();
26      await subscriber.subscribe('jobs-channel', (message) => {
27        const job = JSON.parse(message);
28        send({ type: 'new-job', job }); // push to all connected clients
29      });
30 
31      // Cleanup on disconnect
32      req.signal.addEventListener('abort', async () => {
33        await subscriber.unsubscribe('jobs-channel');
34        await subscriber.quit();
35        controller.close();
36      });
37    },
38  });
39 
40  return new Response(stream, {
41    headers: {
42      'Content-Type': 'text/event-stream',
43      'Cache-Control': 'no-cache',
44      'Connection': 'keep-alive',
45    },
46  });
47}
48 
49// Idempotency key middleware
50export async function checkIdempotency(req: Request) {
51  const idempotencyKey = req.headers.get('Idempotency-Key');
52  if (!idempotencyKey) return null; // key not provided — no dedup
53 
54  const cached = await redis.get(`idem:${idempotencyKey}`);
55  if (cached) {
56    // Already processed — return the cached response
57    return Response.json(JSON.parse(cached), { status: 200 });
58  }
59  return null; // not yet processed — proceed normally
60}
61 
62export async function storeIdempotencyResult(key: string, result: object) {
63  // Cache the result for 24 hours
64  await redis.setex(`idem:${key}`, 86400, JSON.stringify(result));
65}

outbox-email-worker.ts

1// Outbox Pattern + BullMQ email worker
2import { db } from '@/lib/db';
3import { Queue, Worker } from 'bullmq';
4import { Resend } from 'resend';
5import { redis } from '@/lib/redis';
6 
7const emailQueue = new Queue('emails', { connection: redis });
8const resend = new Resend(process.env.RESEND_API_KEY);
9 
10// POST /api/apply — atomic DB write + outbox event
11export async function submitApplication(data: ApplicationInput) {
12  const idempotencyKey = data.idempotencyKey;
13 
14  const application = await db.$transaction(async (tx) => {
15    // Save application
16    const app = await tx.application.create({
17      data: { jobId: data.jobId, email: data.email, resumeUrl: data.resumeUrl },
18    });
19 
20    // Outbox event in the SAME transaction (atomic)
21    await tx.outboxEvent.create({
22      data: {
23        id: idempotencyKey,       // use idempotency key as outbox event ID
24        topic: 'email.confirmation',
25        payload: JSON.stringify({ applicationId: app.id, email: data.email }),
26        processedAt: null,
27      },
28    });
29 
30    return app;
31  });
32 
33  return { applicationId: application.id };
34}
35 
36// Outbox relay: reads committed events, enqueues to BullMQ
37async function runOutboxRelay() {
38  const events = await db.outboxEvent.findMany({
39    where: { processedAt: null },
40    take: 50,
41  });
42 
43  for (const event of events) {
44    await emailQueue.add(event.topic, JSON.parse(event.payload), {
45      jobId: event.id,          // BullMQ dedup: same jobId = same job (not re-added)
46      attempts: 3,
47      backoff: { type: 'exponential', delay: 2000 },
48    });
49    await db.outboxEvent.update({
50      where: { id: event.id },
51      data: { processedAt: new Date() },
52    });
53  }
54}
55 
56// Email worker: idempotent send via Resend
57const emailWorker = new Worker('emails', async (job) => {
58  const { applicationId, email } = job.data;
59 
60  await resend.emails.send({
61    from: 'jobs@company.com',
62    to: email,
63    subject: 'Application received',
64    html: `<p>Application ${applicationId} received.</p>`,
65    headers: {
66      'X-Idempotency-Key': applicationId, // Resend deduplicates on their side too
67    },
68  });
69}, {
70  connection: redis,
71  concurrency: 5,
72});
73 
74emailWorker.on('failed', (job, err) => {
75  console.error(`[DLQ] Email job ${job?.id} failed after all retries: ${err.message}`);
76  // Alert on-call if DLQ has > 0 jobs
77});

4 Features of the Upgrade

1. Server-Sent Events for real-time job feed — No polling. Redis pub/sub fan-out to all connected clients. Initial state sent on connect so no events are missed.
2. Idempotency key on application submission — UUID generated on component mount, sent as Idempotency-Key header. Redis caches result for 24h. No double-processing on retry.
3. Outbox pattern for email confirmation — Application save + outbox event in one Prisma transaction. Relay publishes to BullMQ only after commit. No lost emails on server crash.
4. Resend email with idempotency key — Application UUID sent as X-Idempotency-Key to Resend. Even if BullMQ worker runs twice, Resend deduplicates on their side. Defense in depth.

How Stripe handles this

The interview question: "What if Redis goes down?"

production-apply-flow.ts

1// Complete production apply flow with idempotency
2'use client';
3import { useRef } from 'react';
4 
5// Generate idempotency key once on component mount
6// NOT on button click — retries must use the same key
7export function ApplyButton({ jobId }: { jobId: string }) {
8  const idempotencyKey = useRef(crypto.randomUUID());
9 
10  async function handleApply() {
11    const res = await fetch('/api/apply', {
12      method: 'POST',
13      headers: {
14        'Content-Type': 'application/json',
15        'Idempotency-Key': idempotencyKey.current, // same key on every retry
16      },
17      body: JSON.stringify({ jobId }),
18    });
19 
20    if (!res.ok && res.status !== 200) {
21      // User can retry — same idempotency key prevents duplicate
22      console.error('Apply failed, safe to retry');
23    }
24  }
25 
26  return <button onClick={handleApply}>Apply</button>;
27}
28 
29// Server: POST /api/apply
30export async function POST(req: Request) {
31  const idempotencyKey = req.headers.get('Idempotency-Key');
32 
33  // Check idempotency cache first
34  if (idempotencyKey) {
35    const cached = await redis.get(`idem:${idempotencyKey}`);
36    if (cached) return Response.json(JSON.parse(cached)); // return cached result
37  }
38 
39  const { jobId } = await req.json();
40  const session = await auth();
41  if (!session) return Response.json({ error: 'Unauthorized' }, { status: 401 });
42 
43  // Atomic: application + outbox event in one transaction
44  const result = await db.$transaction(async (tx) => {
45    const app = await tx.application.create({
46      data: { jobId, userId: session.user.id },
47    });
48    await tx.outboxEvent.create({
49      data: {
50        id: idempotencyKey ?? crypto.randomUUID(),
51        topic: 'email.confirmation',
52        payload: JSON.stringify({ applicationId: app.id, email: session.user.email }),
53        processedAt: null,
54      },
55    });
56    return { applicationId: app.id };
57  });
58 
59  // Cache idempotency result for 24h
60  if (idempotencyKey) {
61    await redis.setex(`idem:${idempotencyKey}`, 86400, JSON.stringify(result));
62  }
63 
64  return Response.json(result, { status: 201 });
65}