Design a notification system

“Your order has shipped.” “Ana sent you a message.” “Your password was changed.” “20% off this weekend.” Every product team wants to notify users, through push, email, SMS or a badge in the app, and each team building its own sender ends up with the same bugs: notifications sent twice, at 3 a.m., to users who turned them off, to phones that uninstalled the app months ago.

“Design a notification system” asks for one shared service that does this well for everyone. It’s a favorite interview question because it combines queues, retries, idempotency, third-party providers and a lot of product judgment.

New to some of these? Open what you need, or skip ahead.

Basics:What is a message queue?

A message queue sits between two programs. The producer puts a message in and moves on; a consumer takes it out and does the work later. If the consumer is slow or down, messages wait instead of getting lost. The Message queues lesson covers them in depth.

Basics:How do push notifications reach a phone?

Your server can’t contact a phone directly. Apple and Google run push services (APNs for iOS, FCM for Android) that keep a connection to every device. When your app starts, it registers and receives a device token, a long string identifying that app on that device, and sends it to your server. To notify the user, your server sends the message and the token to APNs or FCM, which delivers it.

Part Not started

Step 1: Requirements and estimates

Start by asking what kinds of notifications exist, because they behave very differently:

Transactional: caused by something the user did or that concerns them directly: “order shipped”, “new message”, “password changed”. Expected, time-sensitive, usually can’t be turned off entirely (security alerts).
Marketing: campaigns sent to many users at once: “sale this weekend”. Optional, must respect opt-outs and laws, and can wait a little.

Functional requirements:

Channels: push (iOS, Android), email, SMS, in-app inbox.
Other services call one API: “notify user 42 with template order_shipped and these values”.
Users choose per category and channel (no marketing push, yes order emails).
Quiet hours in the user’s time zone, except for urgent messages.
Scheduling (“send at 9 a.m. local time”) and campaigns to millions of users.

Non-functional: no duplicates, no lost transactional notifications, urgent ones within seconds even during a campaign, and an audit trail of what was sent.

Probe for: priority classes (security and OTP vs social vs marketing), legal constraints (consent and unsubscribe for email, SMS regulations, GDPR deletion), per-user frequency caps, localization, multi-device fan-out, delivery and engagement tracking, and who owns templates. The architecture follows from three properties: requests are bursty (campaigns), providers are external and rate-limited, and delivery is at-least-once internally, so idempotency and isolation by priority and channel matter more than raw throughput.

Back-of-the-envelope estimates

Users:                           50 million, ~1.5 devices each
Transactional notifications:     ~200 million / day  → ~2,000 / s average, ~10,000 / s peak
Marketing campaign:              20 million users in one campaign
  sent within 15 minutes:        20e6 / 900 s ≈ 22,000 / s on top of normal traffic

Push to providers:               each notification × devices (1.5) ≈ 50,000 device pushes / s peak
Email:                           a few million per day, mostly transactional receipts
SMS:                             only OTPs and critical alerts: expensive (cents per message)

Storage (log of what was sent):  ~250 million records / day × ~300 B ≈ 75 GB / day, kept ~90 days

Bursty: the busiest moments are campaigns, which can be ten times the normal rate.
Urgent must not wait behind bulk: a password-reset code stuck behind 20 million sale announcements is a broken login.
SMS is expensive, so it’s reserved for things like one-time codes.

Transactional vs marketing; channels push/email/SMS/in-app; preferences per category × channel; quiet hours in local time; scheduling and campaigns.
No duplicates, no lost transactional messages, urgent latency in seconds during campaigns, audit trail.
Campaign bursts (~20M in 15 min ≈ 22k/s) dominate; providers are rate-limited; SMS costly; legal consent and unsubscribe.

Check this part

Why must transactional and marketing notifications be handled separately?
Isolate them by priority.
What drives peak load in a notification system?
Campaign bursts can be ten times the normal rate.

One internal service for all teams: transactional and marketing notifications over push, email, SMS and in-app, respecting user preferences, quiet hours and rate limits, never sending duplicates. Peaks come from marketing campaigns: tens of millions of messages that must go out within minutes without starving urgent ones.

Part Not started

Step 2: The pipeline

Notification pipeline

Tap a box to see what it does.

The API other services call

POST /v1/notifications
Idempotency-Key: order-1042-shipped
Content-Type: application/json

{
  "userId": "u_42",
  "template": "order_shipped",
  "category": "orders",
  "priority": "normal",
  "data": { "orderId": "1042", "carrier": "DHL", "eta": "2026-11-12" }
}

→ 202 Accepted   { "notificationId": "n_9f3a…" }

202 Accepted: the API only records the request and returns. Rendering and sending happen later, in workers, so a slow email provider never slows down the order service.
Templates, not text: the caller sends a template name and values; the notification service renders it in the user’s language for each channel (a short push text, a full HTML email).
The idempotency key is chosen by the caller from what happened (order-1042-shipped), so if the order service retries the call, the user still gets one notification.
The router applies the user’s choices: if they turned off order pushes but allow emails, only an email is queued.

Spot the bugOrders that fail because of notifications

Tap the line with the bug.

@Transactional
fun shipOrder(orderId: String) {
    val order = orders.markShipped(orderId)
    for (device in devices.forUser(order.customerId)) {
        apns.send(device.token, "Your order ${order.id} has shipped")   // HTTP call, ~200 ms
    }
    email.send(order.customerEmail, render("order_shipped", order))     // HTTP call, ~500 ms
}

@Transactional
public void shipOrder(String orderId) {
    Order order = orders.markShipped(orderId);
    for (Device device : devices.forUser(order.customerId())) {
        apns.send(device.token(), "Your order " + order.id() + " has shipped");  // HTTP call
    }
    email.send(order.customerEmail(), render("order_shipped", order));         // HTTP call
}

async function shipOrder(orderId: string): Promise<void> {
  await db.transaction(async (tx) => {
    const order = await orders.markShipped(tx, orderId);
    for (const device of await devices.forUser(order.customerId)) {
      await apns.send(device.token, "Your order " + order.id + " has shipped");  // HTTP call
    }
    await email.send(order.customerEmail, render("order_shipped", order));     // HTTP call
  });
}

Sending happens inside the business transaction, synchronously:

If the email provider times out, the exception rolls back the shipment: the warehouse has shipped the package, but the order says it hasn’t. Retrying the operation re-sends the pushes that already went out.
The transaction holds a database connection for every provider round trip (seconds during an outage), so a provider hiccup exhausts the connection pool and takes the order service down.
Preferences, quiet hours, language and invalid tokens are ignored, and every service that sends notifications has to reimplement them.

Record the intent in the same transaction (an outbox row or event), and let the notification service do the sending asynchronously:

@Transactional
fun shipOrder(orderId: String) {
    val order = orders.markShipped(orderId)
    outbox.save(NotifyRequest(
        idempotencyKey = "order-${order.id}-shipped",
        userId = order.customerId,
        template = "order_shipped",
        data = mapOf("orderId" to order.id),
    ))   // published to the notification service after commit
}

Predict the output

A campaign is scheduled for 'Saturday 10:00'. The system stores it as 10:00 UTC and sends to all 20 million users at once. Users are in Europe, the Americas and Asia. What do many of them experience?

Exerciseeasy

Design the preference model

Users should be able to say: no marketing at all; order updates by push and email; chat messages by push only, but not between 22:00 and 07:00; security alerts always, on every channel. Design the data model and the routing rule.

›

Preferences are a small table: category × channel → on or off, plus per-user settings like time zone and quiet hours. Some categories can’t be turned off.

SolutionTry it yourself first

user_settings:           user_id, time_zone, quiet_start (22:00), quiet_end (07:00), locale
channel_preferences:     user_id, category, channel, enabled        (absent row = category default)
category_defaults:       category, channel, enabled, can_opt_out, bypasses_quiet_hours

Defaults:
  security   push/email/sms  enabled, can_opt_out = false, bypasses_quiet_hours = true
  orders     push/email      enabled
  chat       push            enabled
  marketing  push/email      enabled only with explicit consent (legal requirement in many countries)

Routing for one request:

For each channel, take the user’s row if present, else the category default; drop channels that are off (unless can_opt_out = false).
Drop channels the user can’t receive (no device token, no verified email).
If now is within quiet hours in the user’s time zone and the category doesn’t bypass them, delay non-urgent channels until quiet_end (or drop marketing).
Apply frequency caps (for example, at most 3 marketing pushes per week).

Isolation, throttling and deduplication

Queues per channel and priority (and per tenant for multi-product setups), with worker pools sized per queue, so a stuck SMS provider or a 20-million campaign can’t starve urgent pushes.
Provider rate limits: token buckets per provider credential; on 429 or 503, back off with jitter and slow the whole queue rather than hammering.
Deduplication at two levels: the API rejects repeated idempotency keys (a unique index, kept for days); workers record (notification, device, channel) attempts so a redelivered queue message doesn’t send again after a success.
Aggregation: collapse bursts (“Ana and 12 others liked your photo”) by delaying low-priority social notifications a few minutes and merging per user and key; push collapse-id/collapse_key replaces earlier ones on the device.
Campaigns are a separate producer: segment users offline, enqueue gradually by local-time bucket, cap the send rate, and support cancel mid-flight.

API: validate, idempotency key (unique index), persist, 202; never send inside callers’ transactions (outbox → notification service).
Router: preferences (category × channel with non-opt-out categories), reachability, quiet hours in local time, frequency caps.
Queues per channel × priority (× tenant); provider token buckets; backoff on 429/503.
Dedup at API (key) and worker (per device attempt); aggregation and collapse keys; campaigns as a throttled, local-time-bucketed, cancellable producer.

Check this part

Why does the notification API return 202 instead of sending right away?
The caller only needs to know the request was recorded.
How does the system avoid sending "order shipped" twice when the order service retries?
The key describes the event, so retries carry the same one.
How do you keep a login code from waiting behind a 20-million-user campaign?
Isolation, not just capacity.

Services call a notification API that validates the request and records it with an idempotency key. A preference step decides channels and timing; per-channel, per-priority queues feed workers that render templates and call providers. Urgent and bulk traffic never share a queue.

Part Not started

Step 3: Push notifications in detail

Push is the channel with the most moving parts, because the app on the phone is part of the system:

On every launch, the app asks the OS for a device token and sends it to your server with the user id, platform and app version. Tokens change (reinstall, restore from backup, OS updates), so registering once isn’t enough.
The server stores tokens per user; one user can have several (phone, tablet).
To notify, a worker sends the payload and token to APNs or FCM.
The provider answers: delivered to its system, try again later, or this token is no longer valid. Invalid tokens must be deleted.

Device registration is idempotent upsert by token (a token can move between users on shared devices or after logout, so re-assign it), with last_seen for pruning. APNs (HTTP/2, token-based auth with a .p8 key) returns 410 Unregistered with a timestamp and 400 BadDeviceToken; FCM HTTP v1 returns UNREGISTERED or INVALID_ARGUMENT. Delete on those, retry on 429/5xx with backoff, and never retry 4xx payload errors. Payloads should be small (4 KB limit), localized at render time, and free of sensitive data, since they appear on lock screens and pass through Apple and Google; send an id and let the app fetch details, or use a Notification Service Extension to decrypt or enrich.

The app registers its token on every launch

// AppDelegate
func application(_ application: UIApplication,
                 didRegisterForRemoteNotificationsWithDeviceToken deviceToken: Data) {
    let token = deviceToken.map { String(format: "%02x", $0) }.joined()
    Task {
        try? await api.registerDevice(DeviceRegistration(
            token: token, platform: "ios", appVersion: Bundle.main.appVersion,
            locale: Locale.current.identifier, timeZone: TimeZone.current.identifier))
    }
}

class PushService : FirebaseMessagingService() {
    override fun onNewToken(token: String) {                       // called on first launch and on rotation
        DeviceRegistrationWorker.enqueue(applicationContext, token) // WorkManager: retried until it succeeds
    }
}

// Also on every app start, in case the server lost it:
FirebaseMessaging.getInstance().token.addOnSuccessListener { token ->
    DeviceRegistrationWorker.enqueue(context, token)
}

// Server side: an idempotent upsert keyed by token
@PutMapping("/v1/devices/{token}")
public void register(@PathVariable String token, @RequestBody DeviceRegistration body,
                     @AuthenticationPrincipal Jwt jwt) {
    devices.upsert(new Device(token, jwt.getSubject(), body.platform(), body.appVersion(),
                              body.locale(), body.timeZone(), Instant.now()));
    // If the token belonged to another user (shared device, logout/login), it now belongs to this one.
}

// Server side: handling provider responses
async function sendPush(device: Device, payload: PushPayload): Promise<void> {
  const result = await provider.send(device.token, payload);
  if (result.ok) return log.sent(device, payload);
  if (result.reason === "Unregistered" || result.reason === "BadDeviceToken") {
    await devices.remove(device.token);                    // permanent: never retry
  } else if (result.status === 429 || result.status >= 500) {
    throw new RetryableError(result.retryAfterMs);         // transient: back off and retry
  } else {
    log.failed(device, payload, result.reason);            // payload error: fix the code, don't retry
  }
}

Register on every launch and whenever the token changes; the server’s upsert makes repeats harmless.
The token can move between users: if Ana logs out and Marko logs in on the same phone, the token must now point to Marko, or Ana’s messages show up on Marko’s lock screen.
Three kinds of failure: invalid token (delete it), temporary problem (retry later), bad request (fix the code).

Production puzzles

Spot the bugPush costs climbing, delivery falling

Tap the line with the bug.

suspend fun sendToUser(userId: String, payload: PushPayload) {
    for (device in devices.forUser(userId)) {
        val result = fcm.send(device.token, payload)
        if (!result.ok) {
            retryQueue.enqueue(device.token, payload, attempt = 1)
        }
    }
}

void sendToUser(String userId, PushPayload payload) {
    for (Device device : devices.forUser(userId)) {
        SendResult result = fcm.send(device.token(), payload);
        if (!result.ok()) {
            retryQueue.enqueue(device.token(), payload, 1);
        }
    }
}

async function sendToUser(userId: string, payload: PushPayload): Promise<void> {
  for (const device of await devices.forUser(userId)) {
    const result = await fcm.send(device.token, payload);
    if (!result.ok) {
      await retryQueue.enqueue(device.token, payload, 1);
    }
  }
}

Predict the output

A security notification payload contains 'Your verification code is 482913'. The user's phone is locked on a café table. Who can read the code?

Delivery tracking and the in-app inbox

Sent ≠ delivered ≠ seen: providers confirm acceptance, not display. Measure opens (the app reports when a notification is tapped) and keep the funnel per template.
In-app inbox: store each notification per user with read state; it’s the reliable copy when push is disabled or missed, and the badge count comes from it.
Feedback loops: email bounces and complaints (SES/SendGrid webhooks) disable addresses; SMS delivery receipts flag invalid numbers.

Token registration: idempotent upsert on every launch and rotation; re-assign tokens across users on shared devices; prune by last_seen.
Response classes: permanent (APNs 410/BadDeviceToken, FCM UNREGISTERED) → delete; transient (429/5xx) → backoff with jitter and Retry-After; request errors → alert.
Payloads ≤ 4 KB, localized, no secrets (lock screens, provider transit); Notification Service Extension for enrichment.
Track sent/opened per template; in-app inbox as the durable copy; email/SMS feedback loops.

Check this part

APNs answers 410 Unregistered for a token. What should the server do?
The app was uninstalled or the token replaced; it will never work again.
Why should the app send its device token on every launch?
The server's upsert makes repeats harmless.
Where should a one-time code shown in a notification come from?
Lock screens and provider transit expose payload text.

Apps register device tokens on every launch, and the server keeps them per user, app and platform. Remove tokens the provider reports as invalid (APNs 410, FCM UNREGISTERED). Retry transient failures with backoff, never permanent ones, and keep personal data out of the push payload.

Key takeaways

One shared service sends push, email, SMS and in-app notifications for every team, through one API with templates.
The API records the request with an idempotency key and returns; workers send later, through queues per channel and priority.
Respect preferences, quiet hours in the user’s time zone, and frequency caps; schedule campaigns in local time.
Register device tokens on every launch, delete invalid ones, and keep secrets out of notification text.

Never send inside business transactions; outbox to the notification service.
Isolate by channel and priority; throttle per provider; dedup at the API and per device attempt.
Classify provider errors: delete, retry with backoff, or alert.
Measure the funnel and keep an in-app inbox as the durable record.