Message queues · Commitwise

Placing an order takes 4 seconds: save the order, charge the card, send the confirmation email, update the warehouse, notify analytics, generate the invoice PDF. If the email provider is slow, checkout is slow; if it’s down, checkout fails. Only two of those steps need to happen before you tell the user “thank you”. A message queue lets the rest happen right after, reliably, without making anyone wait.

New to some of these? Open what you need, or skip ahead.

Basics:What are clients and servers?

A server is a computer (usually many of them) that runs your backend: it stores data and answers requests. A client is whatever sends the requests: your mobile app, a website, another server.

Big systems put many identical servers behind one address, with a load balancer in front that spreads requests between them. Data lives in a database, and frequently used data is often kept in a fast cache (in memory) so the database isn’t asked every time.

Basics:What does idempotent mean?

An operation is idempotent if doing it twice has the same effect as doing it once. “Set the status to shipped” is idempotent; “add 10 EUR to the balance” isn’t: run it twice and the customer gets 20.

It matters whenever something can be retried, which in networked systems is always: a timeout doesn’t tell you whether the first attempt worked.

Part Not started

Why put a queue in between

In a busy restaurant, the waiter doesn’t stand in the kitchen until your food is ready. They pin your order ticket on a rail and go back to the tables. Cooks take tickets from the rail in order, at their own speed. If ten orders arrive at once, the rail just gets longer for a while; nobody is turned away, and the waiter stays fast.

A message queue is that ticket rail. A producer (the checkout service) puts a message on the queue (“order 1001 was placed”) and moves on. One or more consumers (the email service, the warehouse service) take messages off and do the work.

Queues decouple time (work happens later), rate (producers and consumers run at different speeds; the queue absorbs the difference), availability (consumers can be down; messages wait) and knowledge (producers publish facts; any number of consumers subscribe). Two families: work queues (SQS, RabbitMQ: each message handled by one consumer, deleted after ack) and logs (Kafka, Kinesis, Pub/Sub: an append-only, partitioned, retained stream that many consumer groups read independently, replayable). The costs are eventual consistency, duplicate delivery and operational complexity.

Checkout with a queue

Tap a box to see what it does.

What the queue gives you:

Fast responses: the user waits only for the steps that must happen now.
Spike absorption: if orders arrive faster than emails can be sent, the queue grows for a while and shrinks later; nothing is lost.
Isolation: the email provider being down delays emails, not orders.
Independent teams: adding “send a push notification on every order” means a new consumer; checkout doesn’t change.

The price: the work isn’t done when the user sees “thank you”. The email arrives a few seconds later, and your code must handle that “not yet” state.

Predict the output

During a sale, orders arrive at 1,000 per second for 60 seconds. The email consumers can send 400 emails per second. How many emails are waiting when the spike ends, and how long until the queue is empty if orders then drop to 100 per second?

Exerciseeasy

Now or later?

For a ride-hailing app’s “end trip” action, decide which steps must finish before the driver’s app shows “Trip complete”, and which can go through a queue:

Mark the trip as completed
Calculate the fare
Charge the rider’s card
Email the receipt
Update the driver’s weekly earnings dashboard
Ask the rider for a rating (push notification)
Update the city heat map

›

For each step ask: does the user need its result on the next screen, and does anything else depend on it happening first?

SolutionTry it yourself first

Synchronous (before responding): 1 (the trip must be finished, or the driver could be assigned another ride with this one open) and 2 (the next screen shows the fare). 3, charging, is often done asynchronously too, with the trip in a “payment pending” state and retries, because a slow payment provider shouldn’t block the driver; but the fare and the obligation to pay are recorded synchronously.

Through a queue: 4, 5, 6 and 7. None changes what the driver sees next, each can fail and retry on its own, and the heat map and dashboards are happy with data a few seconds old.

The general rule: do synchronously what the user must see or what later steps depend on; publish an event for everything else.

Queue, pub/sub, or log?

	Work queue (SQS, RabbitMQ)	Log (Kafka, Kinesis)
Delivery	Each message to one consumer, deleted on ack	Retained for days; each consumer group tracks its own offset
Fan-out	Via exchanges/topics to separate queues	Any number of groups read the same data
Ordering	Best effort (FIFO queues: per group, limited throughput)	Per partition, strict
Replay	No (once acked, gone)	Yes, by resetting offsets
Scaling consumers	Add consumers freely	Parallelism capped by partition count
Typical use	Jobs, tasks, commands	Event streams, CDC, analytics, event sourcing

Queues decouple time, rate, availability and ownership; the costs are eventual consistency, duplicates and operational load.
Work queues (SQS, RabbitMQ): per-message ack/delete, competing consumers, no replay. Logs (Kafka): partitioned, retained, replayable, per-group offsets, parallelism bounded by partitions.
Size by arrival vs service rate; monitor backlog/lag as the scaling signal.

Check this part

Why send the confirmation email through a queue instead of during checkout?
The email happens a few seconds later, independently.
Orders arrive faster than consumers can process them for a minute. What happens?
The queue absorbs the spike; the backlog drains when traffic drops.
You need several independent consumers to read the same events and replay them after a bug fix. Which fits?
Logs retain events and track an offset per group.

A producer puts a message on a queue and moves on; consumers take messages and do the work later, at their own pace. Queues make slow work asynchronous, absorb traffic spikes, survive consumer outages, and let teams react to events without calling each other.

Part Not started

At least once, so make it safe twice

A consumer takes a message, does the work, then acknowledges it (“done, delete it”). If it crashes before acknowledging, the queue gives the message to another consumer. That’s what makes queues reliable, and it has a consequence: a message can be processed twice, for example when the work finished but the crash happened just before the ack.

Guarantee	How	Risk
At most once	Ack first, then work	A crash loses the message
At least once	Work first, then ack	A crash after the work means it runs again
Exactly once	At least once + duplicates ignored	Must be built: idempotent processing

Work, then acknowledge

// A generic queue client: receive, process, ack (SQS, RabbitMQ and Kafka all follow this shape).
suspend fun consume(queue: Queue) {
    while (currentCoroutineContext().isActive) {
        val message = queue.receive()                 // invisible to others while we work on it
        try {
            sendConfirmationEmail(message.orderId)    // 1. do the work
            queue.ack(message)                        // 2. only then: done, delete it
        } catch (e: Exception) {
            queue.nack(message, retryAfter = 30.seconds)   // try again later
        }
    }
}

// A generic queue client: receive, process, ack (SQS, RabbitMQ and Kafka all follow this shape).
void consume(Queue queue) {
    while (!Thread.currentThread().isInterrupted()) {
        Message message = queue.receive();             // invisible to others while we work on it
        try {
            sendConfirmationEmail(message.orderId());  // 1. do the work
            queue.ack(message);                        // 2. only then: done, delete it
        } catch (Exception e) {
            queue.nack(message, Duration.ofSeconds(30));   // try again later
        }
    }
}

// A generic queue client: receive, process, ack (SQS, RabbitMQ and Kafka all follow this shape).
async function consume(queue: Queue, signal: AbortSignal): Promise<void> {
  while (!signal.aborted) {
    const message = await queue.receive();          // invisible to others while we work on it
    try {
      await sendConfirmationEmail(message.orderId); // 1. do the work
      await queue.ack(message);                     // 2. only then: done, delete it
    } catch {
      await queue.nack(message, { retryAfterSeconds: 30 });   // try again later
    }
  }
}

Spot the bugEmails that are never sent

Tap the line with the bug.

suspend fun consume(queue: Queue) {
    while (currentCoroutineContext().isActive) {
        val message = queue.receive()
        queue.ack(message)
        sendConfirmationEmail(message.orderId)
    }
}

void consume(Queue queue) {
    while (!Thread.currentThread().isInterrupted()) {
        Message message = queue.receive();
        queue.ack(message);
        sendConfirmationEmail(message.orderId());
    }
}

async function consume(queue: Queue, signal: AbortSignal): Promise<void> {
  while (!signal.aborted) {
    const message = await queue.receive();
    await queue.ack(message);
    await sendConfirmationEmail(message.orderId);
  }
}

The message is acknowledged before the work. If sending fails, or the process is killed by a deploy, the queue has already deleted the message: that customer never gets an email, and nothing will retry it. This is at-most-once delivery, by accident.

Acknowledge after the work succeeds, and let failures be retried:

sendConfirmationEmail(message.orderId)
queue.ack(message)

sendConfirmationEmail(message.orderId());
queue.ack(message);

await sendConfirmationEmail(message.orderId);
await queue.ack(message);

Which brings the opposite problem: if the email is sent and the process dies before ack, the customer gets two emails. For an email that’s tolerable; for money, it isn’t, which is what the next section is about.

Idempotent consumers

Since a message can arrive twice, the consumer must make the second time harmless. The usual way: every message has a unique id, and the consumer remembers which ids it has already processed, in the same database transaction as the work itself:

Ignore messages you've already handled

@Transactional
fun handle(event: PaymentReceived) {
    val firstTime = processed.insertIfAbsent(event.messageId)      // unique index on message_id
    if (!firstTime) return                                         // seen it: do nothing
    accounts.credit(event.accountId, event.amountCents)            // same transaction as the marker
}

@Transactional
void handle(PaymentReceived event) {
    boolean firstTime = processed.insertIfAbsent(event.messageId());   // unique index on message_id
    if (!firstTime) return;                                             // seen it: do nothing
    accounts.credit(event.accountId(), event.amountCents());            // same transaction as the marker
}

async function handle(event: PaymentReceived): Promise<void> {
  await db.transaction(async (tx) => {
    const inserted = await tx.query(
      "INSERT INTO processed_messages (message_id) VALUES ($1) ON CONFLICT DO NOTHING",
      [event.messageId],
    );
    if (inserted.rowCount === 0) return;                         // seen it: do nothing
    await tx.query("UPDATE accounts SET balance = balance + $1 WHERE id = $2",
      [event.amountCents, event.accountId]);                     // same transaction as the marker
  });
}

If the marker and the credit are written in one transaction, either both happen or neither does. A duplicate message finds the marker and does nothing. Some operations are naturally idempotent and need no marker: “set status to shipped” can safely run twice; “add 10 EUR” can’t.

Production puzzles

Predict the output

Checkout saves the order in PostgreSQL, commits, then publishes OrderPlaced to Kafka. The broker is briefly unreachable and the pod is killed mid-retry. What state is the system in?

Spot the bugDouble credits after a rebalance

Tap the line with the bug.

@KafkaListener(topics = ["payments"])
fun onPayment(event: PaymentReceived) {
    accounts.credit(event.accountId, event.amountCents)
    log.info("credited {} to {}", event.amountCents, event.accountId)
}

@KafkaListener(topics = "payments")
void onPayment(PaymentReceived event) {
    accounts.credit(event.accountId(), event.amountCents());
    log.info("credited {} to {}", event.amountCents(), event.accountId());
}

await consumer.run({
  eachMessage: async ({ message }) => {
    const event = JSON.parse(message.value!.toString()) as PaymentReceived;
    await accounts.credit(event.accountId, event.amountCents);
  },
});

Delivery is at least once in practice: ack/commit after the work; crashes, timeouts, rebalances and redrives produce duplicates.
Idempotent consumers: a processed-ids table with a unique constraint updated in the same transaction as the side effect, or naturally idempotent operations (set, upsert by business key).
Dual write (DB commit + publish) loses or invents events; use a transactional outbox with a relay or CDC (Debezium).
Kafka EOS covers Kafka→Kafka only; external side effects still need idempotency.

Check this part

When should a consumer acknowledge a message?
Acking first loses the message if the work fails or the process dies.
Why must a consumer that adds money to an account be idempotent?
Remember processed message ids in the same transaction as the credit.
Save the order, then publish an event. What can go wrong, and what fixes it?
Write the event in the same transaction; a relay publishes it at least once.
Why do Kafka consumers see duplicates around deploys?
Make handlers idempotent; commit offsets after processing.

Queues deliver at least once: a message can arrive twice after a crash or timeout. Acknowledge only after the work is done, make consumers idempotent (remember processed message ids), and publish events through an outbox so a saved order always gets its event.

Part Not started

Order, scale and failures

With one consumer, messages are handled in order, but slowly. With ten consumers, they’re handled ten times faster, but no longer strictly in order: consumer 3 may finish “order 1001 cancelled” before consumer 7 finishes “order 1001 paid”. For most messages that’s fine. For messages about the same thing, it often isn’t.

Ordering is per partition (or per FIFO message group), never global at scale. Route by a key (order id, account id) so all events for one entity land in one partition and are processed sequentially, while different keys proceed in parallel. Consumer parallelism is then bounded by the number of partitions, and a hot key bounds a partition. Retries must not violate the ordering you paid for: an in-place retry blocks the partition; moving a message to a retry topic lets later events for the same key overtake it.

Keep one order's events in order

// Kafka: the record key decides the partition; same key → same partition → in order.
producer.send(ProducerRecord("orders", order.id.toString(), json.encode(event)))

// Kafka: the record key decides the partition; same key → same partition → in order.
producer.send(new ProducerRecord<>("orders", order.id().toString(), json.encode(event)));

// Kafka: the record key decides the partition; same key → same partition → in order.
await producer.send({
  topic: "orders",
  messages: [{ key: String(order.id), value: JSON.stringify(event) }],
});

By giving every message about order 1001 the key 1001, all of them go to the same partition, and one consumer handles them one after another. Messages about other orders go to other partitions and are handled in parallel. You get order where it matters, and speed everywhere else.

Messages that keep failing

Some messages can never succeed: malformed data, a deleted customer, a bug. If the consumer retries them forever, they block everything behind them (a poison message). After a few attempts, move the message to a dead-letter queue (DLQ): a side queue where failed messages wait for a person (or a fixed version of the code) to look at them. Then alert on it; a DLQ nobody watches is a silent data-loss queue.

Practice: what to watch

Backlog (lag): how many messages are waiting. Growing steadily means consumers are too slow: add consumers or make them faster.
Age of the oldest message: how late the slowest work is. “Emails are 20 minutes late” is what users notice.
Error and retry rate: a spike usually means a dependency is down or a bad deploy.
Dead-letter queue size: should be zero most days; anything in it needs a person.

Predict the output

A Kafka topic has 6 partitions. The consumer group runs 10 instances after autoscaling. How many instances actually consume?

Exercisehard

An order-events pipeline

Design the consumer side for OrderEvents (Created, Paid, Cancelled, Shipped) feeding a fulfillment service. Requirements: events for one order are applied in order; a temporary failure (warehouse API down) must not lose or reorder events; a malformed event must not block other orders; and processing must be correct with duplicates. Describe partitioning, retries, dead-lettering, idempotency and what you’d monitor.

›

Retrying in place preserves order but blocks the partition; a retry topic unblocks the partition but lets later events for the same order overtake. Can you block only the affected order?

SolutionTry it yourself first

Partitioning: key by orderId; partitions sized for peak parallelism (e.g. 48), consumer instances ≤ partitions.
Idempotency: each event carries eventId and a per-order sequence. The consumer stores, per order, the last applied sequence in the same transaction as the state change; duplicates (sequence ≤ last) are skipped, and the state machine rejects impossible transitions (Shipped after Cancelled).
Transient failures: retry in place with exponential backoff and jitter for a short budget (seconds). If it’s still failing, park the order: write the event to a per-order “blocked” store (or a retry topic) and record that order as blocked; subsequent events for a blocked order are parked behind it instead of applied. Other orders keep flowing. A retry worker replays parked events in sequence when the dependency recovers.
Poison messages: events that fail deserialization or validation go straight to a DLQ with the error and original payload; alert on DLQ > 0, and provide a tool to redrive after a fix.
Out-of-order arrival (from upstream retries): use the sequence number to buffer a gap briefly or reject and request a resync, instead of trusting arrival order.
Monitoring: lag per partition (a single stuck partition hides in averages), age of oldest message, blocked-order count, DLQ size, processing error rate by type.

Order per partition/message group: key by entity id; parallelism ≤ partitions; hot keys bound throughput; repartitioning remaps keys.
Retries: in-place backoff preserves order but blocks the partition; retry topics unblock but reorder. Park per key (block only the affected entity) when both matter; sequence numbers make duplicates and gaps detectable.
Poison messages → DLQ with alerting and redrive tooling.
Scale and alert on lag per partition and oldest-message age, not CPU.

Check this part

How do you keep all events for one order in order while processing other orders in parallel?
Same key, same partition, handled one after another.
What is a dead-letter queue for?
Someone must watch it and fix or replay those messages.
A topic has 6 partitions and the group has 10 consumers. How many consume?
Each partition goes to one consumer in a group; extras are idle.
Moving failed messages to a retry topic has which ordering risk?
Park per key, or use sequence numbers to detect and hold gaps.

To process faster, add consumers; to keep order, route all messages for one key (one order, one account) to the same partition. Messages that keep failing go to a dead-letter queue instead of blocking everything. Watch the backlog to know when to scale.

Key takeaways

A queue lets slow or unreliable work happen after the response, absorbs spikes, and isolates failures.
Acknowledge a message only after the work succeeds; then expect occasional duplicates.
Make consumers idempotent: remember processed message ids in the same transaction as the work.
Use a key to keep one entity’s messages in order, and a dead-letter queue for messages that keep failing.

Pick work queues for tasks and logs for replayable event streams; size for arrival vs service rate.
At-least-once is the reality: idempotency keys and transactional markers make it effectively exactly once.
Publish via a transactional outbox; never dual-write a database and a broker.
Partition by entity for order, cap parallelism by partitions, park per key on failures, and alert on lag and DLQ size.