← All articles

News feeds, fan-out and the celebrity problem

Updated Nov 10, 2026 · 4 min read

A news feed shows you recent posts from the accounts you follow, newest first. It looks like one query, but at scale it’s the hardest read in a social app. You follow hundreds of accounts, you open the feed many times a day, and it has to load in a blink. The big design question is when to do the work of gathering those posts.

Fan-out on read

The simple way is to build the feed when someone opens it. Look up everyone they follow, fetch each account’s recent posts, merge them by time and return the top 20. Posting is cheap, because a post is written once. Reading is expensive, because every open touches hundreds of accounts.

That trade is backwards for most social apps. People read far more than they post, so the expensive side is the one that happens all day.

Fan-out on write

The usual fix is to do the work when someone posts. Every user gets a ready-made feed, a list of post ids kept in a cache such as Redis. When you post, a background worker looks up your followers and adds the new post’s id to each of their lists. Opening the feed becomes a single read of a list that’s already built.

postadd idone readPosterQueueFan-out workerslook up followersFeed cacheids per userReader
With fan-out on write, a post is copied into each follower’s feed when it’s made. Reading is one lookup.

The cost moves to writes, and it grows with followers. A post from an account with 200 followers becomes 200 inserts. Fifty million posts a day at that average is 10 billion inserts a day, or about 115,000 a second. That sounds like a lot, but each insert is a tiny append to a list in memory, and it happens in the background where nobody waits for it.

Fan-out on readFan-out on write
Work happensEvery time a feed opensEvery time someone posts
Cost of one postOne writeOne insert per follower
Cost of one feed openHundreds of lookups and a mergeOne read
New post shows upRight awayAfter the workers reach you, usually seconds
Best whenFew readers, or people follow few accountsReads far outnumber posts

The celebrity problem

Fan-out on write breaks down for the few accounts with tens of millions of followers. One post from a singer with 40 million followers is 40 million inserts. Even with a big pool of workers that takes minutes, her fans see the post late, and everyone else’s posts queue up behind hers.

The standard answer is a hybrid. Ordinary accounts fan out on write. Accounts above a follower threshold are skipped, and their posts are stored once. When you open your feed, the service reads your ready-made list, fetches recent posts from the handful of big accounts you follow, and merges the two. Most people follow only a few such accounts, so the merge is cheap, and nobody pays for 40 million inserts.

What the feed cache holds

The feed cache stores ids, not posts. An entry is a post id and an author id, 8 bytes each, so 16 bytes. The post text, images and like counts live in a separate post cache, and the feed service looks them up in one batch for the page it’s about to show. That keeps each feed small, and an edited post only has to change in one place.

Feeds are also capped. Nobody scrolls back through thousands of posts, so keeping the newest few hundred covers almost everyone. Older entries fall off the end, and a rare deep scroll can fall back to building the rest on read.

Sizing it

Only keep feeds for people who come back. Many registered accounts haven’t opened the app in months, and fanning out to them wastes both memory and inserts. Cache feeds for users active in the last few weeks, and build a feed on read the first time someone returns.

100,000,000 active users × 500 entries × 16 bytes
= 800,000,000,000 bytes ≈ 800 GB

That’s too much for one machine, but it splits cleanly by user id across a cluster of cache nodes, because each feed belongs to exactly one user.

Paging with cursors

A feed is read a page at a time. Asking for page two with ?offset=20 means “skip the first 20”, but the top of a feed keeps moving. If 3 posts arrive while you read page one, skipping 20 lands 3 posts too early, and you see the end of page one again. A deleted post shifts things the other way, and you silently miss one.

A cursor fixes this. Each page comes back with the id of its last post, and the next request asks for posts older than that one, like ?before=p981. The cursor pins the page to a post rather than a position, so nothing above it can shift what comes next. It’s also faster, because the store can jump straight to that point instead of counting past everything before it.

OffsetCursor
Next page asks forSkip the first NOlder than this post
New posts arrive while you scrollRepeats postsUnaffected
A post is deletedSkips oneUnaffected
Jump to page 50Possible, and slowNot possible, and nobody does it in a feed