Consumer lag
How far a consumer group is behind the newest messages in a topic: the gap between the latest offset and the group's committed offset.
Definition
Lag is measured per partition and summed per group. Steady low lag is healthy; lag that keeps growing means consumers can't keep up or are stuck on a message. Lag, and the age of the oldest unprocessed message, are the main signals for alerting on and autoscaling message consumers.
In practice
After a deploy, the email group's lag climbs to 400,000 because a deserialization error stopped one partition.
Related terms
- Partition (Kafka)One ordered, append-only log inside a Kafka topic. Messages with the same key go to the same partition, and each partition is read by one consumer per group.
- Dead-letter queueA separate queue where messages that keep failing are moved, so they stop blocking the rest and can be inspected later.