Replication and consistency

A user edits their profile, taps Save, and the next screen shows the old version. Another user refreshes a comment thread twice: the first refresh shows a new reply, the second doesn’t. A third sees an answer in a chat before the question it answers. Nothing crashed, no data was lost, and every one of these is “working as designed”: the app read from copies of the data that weren’t all up to date at the same moment.

Copies are not optional at scale: they keep the service up when a machine dies and spread the reads. This lesson covers how databases keep copies in sync, what users see while they’re not, and how to choose which guarantees each feature needs.

New to some of these? Open what you need, or skip ahead.

Basics:What are clients and servers?

A server is a computer (usually many of them) that runs your backend: it stores data and answers requests. A client is whatever sends the requests: your mobile app, a website, another server.

Big systems put many identical servers behind one address, with a load balancer in front that spreads requests between them. Data lives in a database, and frequently used data is often kept in a fast cache (in memory) so the database isn’t asked every time.

Basics:What is a network partition?

A network partition is when machines that are all running can’t reach each other: a broken switch, a misconfigured firewall, a cut cable between data centers. Each side still works and still gets requests, but can’t tell whether the other side crashed or is just unreachable.

Part Not started

One leader, many followers

A newspaper has one editorial office where stories are written, and printing plants around the country that print copies. Every story goes through the office first; the plants receive it a few minutes later and print what they’ve received. Readers in any city get the paper from the nearest plant, a little behind the office.

Most databases work like this. One server, the leader (or primary), accepts all writes. It records every change in a log and sends the log to followers (replicas), which apply the same changes in the same order. Reads can go to any follower, so adding followers adds read capacity, and if the leader dies, a follower can take over.

Single-leader replication (PostgreSQL streaming replication, MySQL, MongoDB replica sets, most managed databases) ships the leader’s write-ahead log or a logical change stream to followers, which apply it in order. The key choices: synchronous acknowledgement (a write commits only after N followers confirm: no data loss on leader failure, higher latency, and writes block if the synchronous follower is down), asynchronous (fast, but followers lag and an unreplicated tail is lost on failover), or semi-synchronous (one synchronous follower, the rest async). Failover detects the dead leader (timeouts, so always a guess), promotes the most up-to-date follower, and redirects writes; the classic failure modes are lost writes and split brain, two nodes both accepting writes.

Single-leader replication

Tap a box to see what it does.

Synchronous follower: the leader waits until it has the change. Safe, but slower.
Asynchronous followers: the leader doesn’t wait. Fast, but they’re slightly behind: this delay is the replication lag.
Failover: when the leader dies, a follower is promoted to leader. If it was asynchronous, the last few writes the old leader accepted but hadn’t sent yet are lost.

Predict the output

The leader acknowledges a payment write and crashes 30 ms later. Its only followers are asynchronous and were 80 ms behind. A follower is promoted. What happened to the payment?

Failover and split brain

Failure detection is a timeout, so it can’t distinguish a dead leader from a slow or partitioned one. If the old leader is alive on the other side of a partition and keeps accepting writes after a follower was promoted, two leaders diverge: split brain. Defenses:

Consensus for leadership (Raft or Paxos in etcd, ZooKeeper, Patroni, managed services): a new leader needs a majority, and a minority side can’t elect one.
Fencing tokens: every leader term gets an increasing number; storage rejects writes carrying an older term, so a stale leader’s writes fail even if it doesn’t know it’s been replaced.
Lease-based leadership: a leader stops accepting writes when its lease expires, before a new leader may be elected (requires bounded clock drift).

Timeouts are a trade-off: too short and load spikes trigger needless failovers (which add load); too long and outages last longer.

Single leader ships its log to followers. Sync: no loss, higher latency, writes block on follower failure. Async: fast, lag, loses the unreplicated tail on failover. Semi-sync as the usual compromise.
Failure detection is a timeout guess; failover risks lost writes and split brain.
Prevent split brain with consensus-based leadership, fencing tokens on writes, or leases with bounded clock drift.

Check this part

In leader-follower replication, where do writes go?
Followers copy the leader's log in order.
What is replication lag?
Reads from a lagging follower can return old data.
What prevents a deposed leader from accepting writes after a partition?
Storage rejects writes from an older leader term.

Most databases replicate with one leader that takes writes and followers that copy its change log. Synchronous followers are always current but slow writes down and block when they fail; asynchronous followers are fast but lag, and lose recent writes if the leader dies. Failover promotes a follower, and must prevent two leaders.

Part Not started

What users see when copies lag

Lag is usually milliseconds, so most screens never notice. But it shows up in three recognizable ways, and each has a simple fix.

Eventual consistency only promises convergence “eventually”, which permits arbitrarily confusing intermediate states. Three session guarantees eliminate the anomalies users notice, without paying for strong consistency everywhere: read-your-writes, monotonic reads and consistent prefix reads. They’re cheaper because they’re scoped to one user’s session or one causal chain, not the whole system.

Anomaly	What the user sees	Fix
Reading stale own writes	Saves a new name; the next screen shows the old one	Read-your-writes: read the user’s own data from the leader for a few seconds after they write, or wait until the replica has reached the write’s log position
Non-monotonic reads	Refreshes and sees a new comment; refreshes again and it’s gone	Monotonic reads: route each user to the same replica (sticky by user id), or track the newest position they’ve seen
No consistent prefix	Sees “Yes, 7 pm works!” before “Dinner tomorrow?”	Consistent prefix: write causally related data to the same partition (one conversation, one shard), or attach causal ordering metadata

Read-your-writes with a log position

// After a write, return the leader's log position to the client (e.g. in a header or the session).
fun updateProfile(userId: Long, change: ProfileChange): WriteToken {
    leader.update(userId, change)
    return WriteToken(leader.currentLogPosition())       // PostgreSQL: pg_current_wal_lsn()
}

// On a later read, use a replica only if it has caught up to that position.
fun profile(userId: Long, token: WriteToken?): Profile {
    val replica = replicas.random()
    val db = if (token == null || replica.replayedPosition() >= token.position) replica else leader
    return db.loadProfile(userId)
}

WriteToken updateProfile(long userId, ProfileChange change) {
    leader.update(userId, change);
    return new WriteToken(leader.currentLogPosition());   // PostgreSQL: pg_current_wal_lsn()
}

Profile profile(long userId, WriteToken token) {
    Replica replica = replicas.random();
    boolean caughtUp = token == null || replica.replayedPosition().compareTo(token.position()) >= 0;
    return (caughtUp ? replica : leader).loadProfile(userId);
}

async function updateProfile(userId: string, change: ProfileChange): Promise<WriteToken> {
  await leader.update(userId, change);
  return { position: await leader.currentLogPosition() };    // PostgreSQL: pg_current_wal_lsn()
}

async function profile(userId: string, token?: WriteToken): Promise<Profile> {
  const replica = pickReplica();
  const caughtUp = !token || (await replica.replayedPosition()) >= token.position;
  return (caughtUp ? replica : leader).loadProfile(userId);
}

The write returns a token: the leader’s position in its log right after the change.
The next read checks whether the replica has applied the log up to that position. If yes, the replica already has the change; if not, the read goes to the leader.
Only this user’s next reads are affected, so the leader isn’t flooded.

Spot the bugComments that come and go

Tap the line with the bug.

fun comments(postId: Long): List<Comment> {
    val replica = replicas[Random.nextInt(replicas.size)]
    return replica.query("SELECT * FROM comments WHERE post_id = ? ORDER BY created_at", postId)
}

List<Comment> comments(long postId) {
    Replica replica = replicas.get(ThreadLocalRandom.current().nextInt(replicas.size()));
    return replica.query("SELECT * FROM comments WHERE post_id = ? ORDER BY created_at", postId);
}

async function comments(postId: number): Promise<Comment[]> {
  const replica = replicas[Math.floor(Math.random() * replicas.length)];
  return replica.query("SELECT * FROM comments WHERE post_id = $1 ORDER BY created_at", [postId]);
}

Each request picks a random replica, and replicas lag by different amounts. The first refresh hits the up-to-date replica and shows a new comment; the second hits the lagging one, and the comment disappears. Time seems to go backwards. Users report it as “comments randomly vanish”, and it’s nearly impossible to reproduce on a developer’s machine with one database.

Give each user monotonic reads: route a user to the same replica every time (hash of the user id), falling back to another only if that one is down, or remember the newest log position the user has seen and never read from a replica behind it.

val replica = replicas[Math.floorMod(userId.hashCode(), replicas.size)]   // same user, same replica

Exerciseeasy

Which guarantee does each screen need?

A social app reads from replicas with up to 2 seconds of lag. For each screen, say whether lag is acceptable and, if not, which fix applies: (1) the global trending list, (2) the user’s own profile right after editing it, (3) a chat conversation, (4) the follower count on someone else’s profile, (5) a bank-style wallet balance after a top-up.

›

Ask: whose write is the user expecting to see? Would seeing things out of order or going backwards confuse them? Is money involved?

SolutionTry it yourself first

Trending list: lag is fine. Nobody can tell whether it’s 2 seconds old.
Own profile after editing: needs read-your-writes (leader for this user for a few seconds, or a write token).
Chat: needs consistent prefix (all messages of a conversation in one partition, ordered by a per-conversation sequence) and read-your-writes for the sender’s own messages, usually also shown optimistically by the client.
Follower count on someone else’s profile: lag is fine; approximate counts are normal.
Wallet balance after a top-up: read from the leader (or with a write token). Showing the old balance makes users top up twice.

Most screens tolerate lag; only a few need stronger guarantees, and they need them for specific users at specific moments.

Consistency models, briefly

Linearizable (strong): every read sees the latest committed write, as if there were one copy. Needed for locks, leader election, uniqueness (usernames), and balance checks. Costs coordination and latency; unavailable on the minority side of a partition.
Sequential / causal: everyone sees causally related operations in the same order (a reply after its question); concurrent unrelated operations may be seen in different orders. Achievable while staying available.
Session guarantees: read-your-writes, monotonic reads and writes, writes-follow-reads; scoped to a client session.
Eventual: replicas converge if writes stop; no ordering promises in the meantime.

Interviews reward naming the guarantee a feature needs and the cheapest mechanism that provides it, rather than “use strong consistency” everywhere.

Session guarantees fix visible lag anomalies cheaply: read-your-writes (leader reads or LSN/write tokens), monotonic reads (sticky replica per user or max-seen position), consistent prefix (causally related data in one partition, sequence numbers).
Models: linearizable (locks, uniqueness, balances) > causal > session > eventual. Pick per feature, not per system.

Check this part

A user refreshes and a new comment disappears. Which guarantee is missing?
Their second read went to a replica further behind than the first.
How can a user see their own edit right away while most reads use replicas?
Read-your-writes, scoped to one user.
Which feature needs linearizable reads?
Uniqueness requires seeing the latest committed state.

Replication lag causes three visible anomalies: reading stale data right after your own write, data that disappears on refresh (non-monotonic reads), and answers that appear before their questions (no consistent prefix). Each has a targeted fix: read-your-writes, sticky reads, and keeping causally related data on one partition.

Part Not started

Writes in more than one place

With one leader, users far away wait for a round trip to the leader’s region on every write, and the leader’s region going down stops all writes. Some systems therefore accept writes in several places: a leader in each region (multi-leader), or any replica at all (leaderless, as in Cassandra and DynamoDB).

The catch: two people can change the same thing in two places at the same time. When the copies meet, there’s a conflict, and something has to decide what the final value is.

Multi-leader (one leader per region, or every offline-capable client as a “leader”) and leaderless (Dynamo-style) replication trade conflicts for write availability and locality. Conflict handling options: avoid them (route all writes for a record to its home region), last-write-wins by timestamp (simple, silently drops concurrent writes, vulnerable to clock skew), application merges (custom or user-resolved), or CRDTs (data types whose concurrent updates merge deterministically: counters, sets, sequences for collaborative text). Leaderless systems use quorums with read repair and anti-entropy to converge.

Quorum arithmetic

N = 3 replicas of each key
W = 2  → a write succeeds after 2 replicas confirm
R = 2  → a read asks 2 replicas and takes the newest version

R + W = 4 > N = 3  → every read set overlaps every write set in at least one replica,
                     so a successful read sees the latest successful write
                     (barring concurrent writes, failures mid-write and sloppy quorums)

W = 1, R = 1        → fastest, R + W = 2 ≤ 3: reads can miss recent writes
W = 3, R = 1        → fast reads, but a write fails if any replica is down

With N copies, a write waits for W of them and a read asks R of them.
If R + W > N, the replicas a read asks and the replicas that got the latest write always share at least one, so the read sees it.
Lower W or R means faster, more available operations, and a higher chance of stale reads.

Predict the output

Two users edit the same note title at the same moment, one in Europe ('Shopping') and one in the US ('Groceries'). The database resolves conflicts with last-write-wins, and the US server's clock is 300 ms fast. What's the final title?

Production puzzles

Spot the bugLikes that go missing

Tap the line with the bug.

// Multi-region, last-write-wins store; each region handles its own users
fun like(postId: String) {
    val current = store.get("likes:$postId")?.toLong() ?: 0
    store.put("likes:$postId", (current + 1).toString())
}

// Multi-region, last-write-wins store; each region handles its own users
void like(String postId) {
    long current = Optional.ofNullable(store.get("likes:" + postId)).map(Long::parseLong).orElse(0L);
    store.put("likes:" + postId, Long.toString(current + 1));
}

// Multi-region, last-write-wins store; each region handles its own users
async function like(postId: string): Promise<void> {
  const current = Number((await store.get("likes:" + postId)) ?? 0);
  await store.put("likes:" + postId, String(current + 1));
}

Multi-leader/leaderless: write locality and availability, at the cost of conflicts. Avoid (home region per record), LWW (drops concurrent writes, clock skew), merge, or CRDTs.
Quorums: R + W > N for overlap; sloppy quorums and hinted handoff weaken it; read repair and anti-entropy converge replicas.
Never read-modify-write counters on LWW stores; use CRDT counters, idempotent membership rows, or a single home for the key.

Check this part

With N = 3, which W and R guarantee a read overlaps the latest write?
R + W = 4 > 3.
What does last-write-wins do with two concurrent edits?
The other edit is lost silently.
How should a multi-region like counter be stored?
Merging data types don't lose concurrent increments.

Multi-leader and leaderless replication accept writes in several places, for multi-region latency and availability, at the price of conflicts. Resolve them with last-write-wins (loses data), merges or CRDTs. Leaderless systems use quorums: with N copies, W write acks and R read replies, R + W greater than N makes reads overlap the latest write.

Key takeaways

A leader takes writes and followers copy them; followers lag a little behind.
Asynchronous followers can lose the last writes if the leader dies; synchronous ones can’t, but are slower.
Lag shows up as stale own edits, data that disappears on refresh, or answers before questions; each has a targeted fix.
With writes in several places, conflicts happen; last-write-wins silently drops one side.

Sync vs async vs semi-sync is a durability/latency trade-off; failover needs split-brain protection.
Session guarantees fix most user-visible anomalies without global strong consistency.
Choose consistency per feature: linearizable for uniqueness, locks and money; eventual for feeds and counts.
Multi-leader and leaderless need conflict strategies (CRDTs, merges, home regions) and quorum reasoning.