Design a news feed · Commitwise

Open almost any social app and the first screen is a feed: posts from the people and pages you follow, newest or most relevant first, loading more as you scroll. It looks simple. But with 300 million users, some following thousands of accounts and some followed by tens of millions, “show me the latest posts from everyone I follow” is one of the most read-heavy problems in system design.

“Design a news feed” is a classic interview question because it forces the central trade-off of large systems: do the work when data is written, or when it is read?

New to some of these? Open what you need, or skip ahead.

Basics:What is a cache?

A cache keeps a copy of data somewhere faster to read, usually in memory, so that repeated reads don’t hit the slower database. Redis is a popular in-memory store used for caches. The Caching lesson covers it.

Basics:What is a message queue?

A message queue lets one service hand work to another without waiting: the producer puts a message in, a worker takes it out later. The Message queues lesson covers them.

Part Not started

Step 1: Requirements and estimates

Functional requirements:

A user can publish a post (text, images).
A user can follow and unfollow other accounts.
A user sees a home feed: posts from accounts they follow, newest first (ranking comes later), loaded page by page.

Non-functional requirements:

The feed opens fast, ideally under 200 ms on the server.
A new post doesn’t have to appear in every follower’s feed instantly; a few seconds is fine. This is eventual consistency, and it’s what makes the design possible.
The system must handle celebrities: accounts with tens of millions of followers.

Out of scope unless asked: comments, likes, ads, notifications, search.

Clarify: chronological or ranked feed (ranked changes storage and pagination), feed composition (followed accounts only, or also groups, recommendations, ads), media handling (out of scope: object storage and CDN), freshness target, and whether the feed must be consistent across devices. The key numbers are the read/write ratio and the follower distribution: a power law, where the median account has a few hundred followers and the top accounts have tens of millions. Those two facts decide the fan-out strategy.

Back-of-the-envelope estimates

Daily active users:              300 million
Feed opens:                      ~10 per user per day → 3 billion / day ≈ 35,000 / s (peak ~100,000 / s)
Posts:                           ~0.3 per user per day → 100 million / day ≈ 1,200 / s
Average followers per poster:    ~200   (median lower; a few accounts above 10 million)

Fan-out on write:                100 million posts × 200 followers = 20 billion feed inserts / day
                                 ≈ 230,000 / s average
Feed cache:                      keep the latest 500 post ids per active user
                                 300 million × 500 × 8 bytes ≈ 1.2 TB (sharded Redis cluster)
Posts:                           100 million / day × ~1 KB ≈ 100 GB / day (media stored separately)

Reads (35,000 per second) far outnumber writes (1,200 per second). We can afford extra work when posting to make reading cheap.
20 billion feed inserts a day sounds huge, but each one is just “add post id to a list”, and only ids go into feeds: the posts themselves are stored once.
1.2 TB of feed cache fits in a cluster of in-memory servers.

Post, follow, home feed (chronological then ranked), paginated; posts appear within seconds (eventual consistency).
~35k feed reads/s (peak 100k) vs ~1.2k posts/s; read-heavy by ~30–100×.
Follower counts follow a power law: median hundreds, top accounts tens of millions.
Feed caches of post ids only: ~500 ids × 300M users ≈ 1.2 TB in sharded Redis.

Check this part

Why is it acceptable for a post to take a few seconds to reach followers' feeds?
Eventual consistency makes fan-out practical.
Which fact most shapes the fan-out strategy?
Celebrities break pure fan-out on write.

Users publish posts and see a feed of posts from accounts they follow, newest or ranked first, paginated. Reads outnumber writes about 100 to 1, and follower counts are extremely uneven: most users have hundreds, a few have tens of millions. The feed must load in under a few hundred milliseconds; a post can take seconds to appear.

Part Not started

Step 2: Fan-out on write, on read, or both

There are two ways to build a feed:

Fan-out on read (pull): when you open the app, the server looks up everyone you follow, fetches their latest posts, and merges them. Posting is cheap; opening the feed is expensive, and it happens 30 times more often.
Fan-out on write (push): when someone posts, a background worker adds the post’s id to the cached feed of each follower. Opening the feed is just reading one ready list. Posting is expensive, but it happens in the background.

Push is the better default, because reads are so much more common. Its weak spot is celebrities: one post from an account with 50 million followers means 50 million inserts. So big systems use a hybrid: push for normal accounts, and for the few celebrity accounts, pull their posts at read time and merge them in.

Push (precomputed timelines) gives O(1) reads at the cost of write amplification proportional to follower count, wasted work for inactive followers, and slow propagation for huge accounts. Pull gives cheap writes and always-fresh data at the cost of read-time fan-in over all followees (hundreds of queries or a scatter-gather per feed open). The hybrid: fan out on write for accounts below a follower threshold (say 10,000–100,000), and for accounts above it, store posts only in the author’s own timeline and merge them into the reader’s feed at read time. Readers follow few celebrities, so the read-time merge is small. Optimizations: skip fan-out to users inactive for 30 days (rebuild their feed on next login), bound feed length, and store only ids plus minimal metadata.

Hybrid feed architecture

Tap a box to see what it does.

The fan-out worker

class FanOutWorker(
    private val graph: FollowGraph,
    private val feeds: FeedCache,
    private val activity: UserActivity,
) {
    suspend fun onPostCreated(event: PostCreated) {
        if (graph.followerCount(event.authorId) > CELEBRITY_THRESHOLD) return   // pulled at read time

        graph.followers(event.authorId, pageSize = 1_000).collect { page ->     // paged, never all at once
            val active = page.filter { activity.isActiveWithinDays(it, 30) }
            feeds.pushAll(active, event.postId, event.createdAt)                 // one pipelined Redis call
        }
    }

    companion object { const val CELEBRITY_THRESHOLD = 50_000 }
}

class RedisFeedCache(private val redis: RedisClient) : FeedCache {
    override suspend fun pushAll(userIds: List<UserId>, postId: PostId, at: Instant) {
        redis.pipelined {
            userIds.forEach { user ->
                zadd("feed:$user", at.toEpochMilli().toDouble(), postId.value)
                zremrangebyrank("feed:$user", 0, -501)        // keep only the newest 500
            }
        }
    }
}

@Component
public class FanOutWorker {
    private static final long CELEBRITY_THRESHOLD = 50_000;
    private final FollowGraph graph;
    private final StringRedisTemplate redis;
    private final UserActivity activity;

    // constructor omitted

    @KafkaListener(topics = "post-created")
    public void onPostCreated(PostCreated event) {
        if (graph.followerCount(event.authorId()) > CELEBRITY_THRESHOLD) return;   // pulled at read time

        graph.forEachFollowerPage(event.authorId(), 1_000, page -> {
            List<String> active = page.stream().filter(u -> activity.isActiveWithinDays(u, 30)).toList();
            redis.executePipelined((RedisCallback<Object>) conn -> {
                for (String user : active) {
                    byte[] key = ("feed:" + user).getBytes(UTF_8);
                    conn.zSetCommands().zAdd(key, event.createdAt().toEpochMilli(), event.postId().getBytes(UTF_8));
                    conn.zSetCommands().zRemRange(key, 0, -501);    // keep only the newest 500
                }
                return null;
            });
        });
    }
}

const CELEBRITY_THRESHOLD = 50_000;

export async function onPostCreated(event: PostCreated, deps: Deps) {
  if ((await deps.graph.followerCount(event.authorId)) > CELEBRITY_THRESHOLD) return;   // pulled at read time

  for await (const page of deps.graph.followerPages(event.authorId, 1_000)) {
    const active = page.filter((u) => deps.activity.isActiveWithinDays(u, 30));
    const pipeline = deps.redis.pipeline();
    for (const user of active) {
      pipeline.zadd(`feed:${user}`, event.createdAt.getTime(), event.postId);
      pipeline.zremrangebyrank(`feed:${user}`, 0, -501);   // keep only the newest 500
    }
    await pipeline.exec();
  }
}

Only the post id goes into each follower’s feed. The post itself is stored once; the feed service looks posts up by id when it builds a page.
Followers are processed in pages of 1,000, so a popular author doesn’t load millions of ids into memory.
Inactive users are skipped. If someone hasn’t opened the app in 30 days, there’s no point updating their feed; it’s rebuilt when they come back.
Celebrities are skipped entirely here. Their posts are merged in when the feed is read.

Predict the output

A celebrity with 40 million followers posts, and the system uses pure fan-out on write with workers that together insert 500,000 feed entries per second. How long until the last follower sees the post, ignoring all other traffic?

Push: O(1) reads, write amplification ∝ followers; pull: cheap writes, read-time fan-in over all followees.
Hybrid: push below a follower threshold, pull celebrities’ author timelines at read time and merge.
Store ids only in feeds (sorted sets scored by time, capped ~500); hydrate from a post cache.
Skip inactive followers (rebuild on return); page through followers; pipeline Redis writes.
Isolate fan-out from publishing via a queue; one huge fan-out shouldn’t starve others.

Check this part

What does fan-out on write do when someone posts?
Reading a feed then becomes one list lookup.
Why are celebrity posts pulled at read time instead?
Readers follow few celebrities, so merging is cheap.
Why store only post ids in the feed cache?
Hydrate ids into posts at read time from a post cache.

Fan-out on write pushes each new post id into every follower's cached feed, making reads a single list lookup but making celebrity posts very expensive. Fan-out on read merges followees' recent posts at read time, which is cheap to write but slow to read. The usual answer is a hybrid: push for normal accounts, pull for celebrities, merged at read time.

Part Not started

Step 3: Reading and paginating the feed

The app never loads the whole feed. It asks for a page (say 20 posts), and when the user scrolls near the end, it asks for the next one.

The tricky part is saying which page. “Give me posts 21 to 40” (an offset) breaks on a feed, because new posts arrive at the top while you scroll: when 3 new posts have arrived, “posts 21 to 40” now starts with 3 posts you’ve already seen. Instead, the server returns a cursor with each page, meaning “the last post you got was this one”, and the next request asks for “posts older than the cursor”. New posts at the top don’t affect that.

Building a page: read up to N ids older than the cursor from the user’s feed cache, fetch the celebrity authors the user follows and their posts older than the cursor (from author timelines, cached), merge by score, filter (deleted posts, blocked or muted authors, privacy changes), hydrate in one batched multi-get from a post cache, and return the page plus a cursor encoding the last (score, postId). The tie-breaker matters: many posts share a millisecond at scale. For a ranked feed, the order isn’t time, so cursors become opaque tokens referring to a server-side snapshot of the ranked candidate list (cached for the session), since re-ranking between pages would duplicate or drop posts. Cache misses (new or returning users) fall back to pull and rebuild the cache asynchronously.

Spot the bugThe same posts appear twice while scrolling

Tap the line with the bug.

@GetMapping("/feed")
fun feed(
    @AuthenticationPrincipal user: UserId,
    @RequestParam(defaultValue = "0") page: Int,
): FeedPage {
    val ids = feedCache.range(user, start = page * 20L, end = page * 20L + 19)
    return FeedPage(posts = posts.hydrate(ids), nextPage = page + 1)
}

@GetMapping("/feed")
public FeedPage feed(
        @AuthenticationPrincipal UserId user,
        @RequestParam(defaultValue = "0") int page) {
    List<String> ids = feedCache.range(user, page * 20L, page * 20L + 19);
    return new FeedPage(posts.hydrate(ids), page + 1);
}

app.get("/feed", async (req, res) => {
  const page = Number(req.query.page ?? 0);
  const ids = await feedCache.range(req.user.id, page * 20, page * 20 + 19);
  res.json({ posts: await posts.hydrate(ids), nextPage: page + 1 });
});

Pages are defined by position, and positions move. After 5 new posts arrive, every old post shifts down by 5, so page 1 (positions 20 to 39) begins with the last 5 posts of the page the user already saw: duplicates. Deletions shift posts up and cause skips instead, which nobody reports because nobody notices a missing post.

Use a cursor: the position in the feed’s order, not in the list. Return the score and id of the last post, and ask for posts strictly older than that:

@GetMapping("/feed")
fun feed(@AuthenticationPrincipal user: UserId, @RequestParam cursor: String?): FeedPage {
    val after = cursor?.let(FeedCursor::decode)                    // (timestampMs, postId) or null
    val entries = feedCache.olderThan(user, after, limit = 20)     // ZREVRANGEBYSCORE with a tie-break on id
    return FeedPage(
        posts = posts.hydrate(entries.map { it.postId }),
        nextCursor = entries.lastOrNull()?.let { FeedCursor(it.score, it.postId).encode() },
    )
}

@GetMapping("/feed")
public FeedPage feed(@AuthenticationPrincipal UserId user, @RequestParam(required = false) String cursor) {
    FeedCursor after = cursor == null ? null : FeedCursor.decode(cursor);
    List<FeedEntry> entries = feedCache.olderThan(user, after, 20);
    String next = entries.isEmpty() ? null : new FeedCursor(entries.getLast().score(), entries.getLast().postId()).encode();
    return new FeedPage(posts.hydrate(entries.stream().map(FeedEntry::postId).toList()), next);
}

app.get("/feed", async (req, res) => {
  const after = req.query.cursor ? FeedCursor.decode(String(req.query.cursor)) : null;
  const entries = await feedCache.olderThan(req.user.id, after, 20);
  const last = entries.at(-1);
  res.json({
    posts: await posts.hydrate(entries.map((e) => e.postId)),
    nextCursor: last ? FeedCursor.encode(last.score, last.postId) : null,
  });
});

Make the cursor opaque (base64 of the values, optionally signed) so clients don’t build their own and you can change its contents later.

The client: load the next page before the end

@Observable
final class FeedViewModel {
    private(set) var posts: [Post] = []
    private var nextCursor: String?
    private var isLoading = false
    private var reachedEnd = false
    private let api: FeedApi
    init(api: FeedApi) { self.api = api }

    func onAppear(of post: Post) async {
        // Start loading when the user is 5 posts from the end
        guard let index = posts.firstIndex(where: { $0.id == post.id }),
              index >= posts.count - 5 else { return }
        await loadMore()
    }

    func loadMore() async {
        guard !isLoading, !reachedEnd else { return }       // one request at a time
        isLoading = true
        defer { isLoading = false }
        do {
            let page = try await api.feed(cursor: nextCursor)
            let known = Set(posts.map(\.id))
            posts += page.posts.filter { !known.contains($0.id) }   // defensive dedup
            nextCursor = page.nextCursor
            reachedEnd = page.nextCursor == nil
        } catch {
            // keep the cursor; the next scroll retries the same page
        }
    }

    func refresh() async {
        nextCursor = nil; reachedEnd = false
        let page = try? await api.feed(cursor: nil)
        if let page { posts = page.posts; nextCursor = page.nextCursor }
    }
}

class FeedViewModel(private val api: FeedApi) : ViewModel() {
    private val _posts = MutableStateFlow<List<Post>>(emptyList())
    val posts: StateFlow<List<Post>> = _posts.asStateFlow()
    private var nextCursor: String? = null
    private var reachedEnd = false
    private var loading: Job? = null

    fun onItemVisible(index: Int) {
        if (index >= _posts.value.size - 5) loadMore()   // prefetch 5 posts before the end
    }

    fun loadMore() {
        if (loading?.isActive == true || reachedEnd) return   // one request at a time
        loading = viewModelScope.launch {
            runCatching { api.feed(nextCursor) }.onSuccess { page ->
                _posts.update { old -> (old + page.posts).distinctBy { it.id } }
                nextCursor = page.nextCursor
                reachedEnd = page.nextCursor == null
            }
        }
    }
}

export function useFeed(api: FeedApi) {
  const [posts, setPosts] = useState<Post[]>([]);
  const cursor = useRef<string | null>(null);
  const loading = useRef(false);
  const reachedEnd = useRef(false);

  const loadMore = useCallback(async () => {
    if (loading.current || reachedEnd.current) return;          // one request at a time
    loading.current = true;
    try {
      const page = await api.feed(cursor.current);
      setPosts((old) => {
        const known = new Set(old.map((p) => p.id));
        return [...old, ...page.posts.filter((p) => !known.has(p.id))];
      });
      cursor.current = page.nextCursor;
      reachedEnd.current = page.nextCursor === null;
    } finally {
      loading.current = false;
    }
  }, [api]);

  return { posts, loadMore };
}

Production puzzles

Predict the output

The feed is ranked by a model, and the server re-ranks the candidates on every page request. A user scrolls three pages. What do they experience?

Predict the output

A user blocks an account. Their feed cache still contains 40 post ids from that account. What's the cheapest correct behavior?

Other things interviewers probe

New user or returning after 30 days: no feed cache; build it with a pull over followed accounts (bounded), return the first page, and populate the cache asynchronously.
Unfollow: filter at read time immediately; remove ids from the cache lazily.
Hot posts: a viral post is hydrated millions of times per minute; a post cache with request coalescing (one fetch per key per node) and a short local in-process cache in front of Redis protect the database.
Multi-region: feed caches are per region; fan-out events are replicated, and each region fans out locally.

Exercisemedium

Celebrity merge

User Ana follows 300 normal accounts and 3 celebrities. Her feed cache has normal accounts’ post ids with timestamps. Describe step by step how the feed service builds her first page of 20 posts, and what the cursor should contain.

›

The celebrities’ posts aren’t in her cache. Where are they, and how many of them do you need at most?

SolutionTry it yourself first

Read up to 20 newest entries from Ana’s feed cache: (timestamp, postId) pairs.
For each of the 3 celebrities, read up to 20 newest posts from their author timeline (cached, since millions of readers need it). At most 60 candidates.
Merge all candidates by timestamp (ties broken by post id) and take the newest 20.
Filter deleted posts and blocked or muted authors (fetch a few extra in steps 1 and 2 so the page stays full).
Hydrate the 20 ids into posts with one batched lookup.
The cursor is the (timestamp, postId) of the 20th post. The next page repeats the same steps with “older than the cursor” in every source.

Cursor pagination on (score, id), opaque and optionally signed; offsets duplicate and skip as the feed shifts.
Page build: cached ids older than cursor + celebrity author timelines, merge, filter (deleted, blocked, muted), batched hydration, next cursor.
Ranked feeds: rank once per session and paginate a cached snapshot; re-ranking per page duplicates and drops.
Read-time filtering is the correctness net; clean caches lazily; over-fetch to keep pages full.
Client: prefetch near the end, one request at a time, dedupe by id, keep the cursor on failure.

Check this part

Why does offset pagination show duplicates in a feed?
Use a cursor that says "older than this post".
What should a feed cursor contain?
Many posts can share a timestamp, so add the id.
How do you paginate a ranked feed without duplicates?
Scores change between requests.

Paginate with a cursor (the score and id of the last item seen), never with an offset, or new posts shift pages and cause duplicates. The feed service merges cached ids with celebrity posts, filters deleted or blocked content, hydrates posts in batches and returns the next cursor. The client loads the next page before the user reaches the end.

Key takeaways

A feed is read far more often than posts are written, so do the work when posting.
Push post ids into followers’ feeds, except for celebrities, whose posts are merged at read time.
Store only ids in feeds; posts are stored once.
Paginate with a cursor, never an offset.

Hybrid fan-out driven by the read/write ratio and the follower power law.
Capped id lists in sharded Redis, hydrated from a post cache with coalescing.
Read-time filtering for blocks, deletes and privacy; lazy cache cleanup.
Cursor pagination on (score, id); snapshots for ranked feeds.