News feeds, fan-out and the celebrity problem
Updated Nov 10, 2026 · 4 min read
A news feed shows you recent posts from the accounts you follow, newest first. It looks like one query, but at scale it’s the hardest read in a social app. You follow hundreds of accounts, you open the feed many times a day, and it has to load in a blink. The big design question is when to do the work of gathering those posts.
Fan-out on read
The simple way is to build the feed when someone opens it. Look up everyone they follow, fetch each account’s recent posts, merge them by time and return the top 20. Posting is cheap, because a post is written once. Reading is expensive, because every open touches hundreds of accounts.
That trade is backwards for most social apps. People read far more than they post, so the expensive side is the one that happens all day.
Fan-out on write
The usual fix is to do the work when someone posts. Every user gets a ready-made feed, a list of post ids kept in a cache such as Redis. When you post, a background worker looks up your followers and adds the new post’s id to each of their lists. Opening the feed becomes a single read of a list that’s already built.
The cost moves to writes, and it grows with followers. A post from an account with 200 followers becomes 200 inserts. Fifty million posts a day at that average is 10 billion inserts a day, or about 115,000 a second. That sounds like a lot, but each insert is a tiny append to a list in memory, and it happens in the background where nobody waits for it.
| Fan-out on read | Fan-out on write | |
|---|---|---|
| Work happens | Every time a feed opens | Every time someone posts |
| Cost of one post | One write | One insert per follower |
| Cost of one feed open | Hundreds of lookups and a merge | One read |
| New post shows up | Right away | After the workers reach you, usually seconds |
| Best when | Few readers, or people follow few accounts | Reads far outnumber posts |
The celebrity problem
Fan-out on write breaks down for the few accounts with tens of millions of followers. One post from a singer with 40 million followers is 40 million inserts. Even with a big pool of workers that takes minutes, her fans see the post late, and everyone else’s posts queue up behind hers.
The standard answer is a hybrid. Ordinary accounts fan out on write. Accounts above a follower threshold are skipped, and their posts are stored once. When you open your feed, the service reads your ready-made list, fetches recent posts from the handful of big accounts you follow, and merges the two. Most people follow only a few such accounts, so the merge is cheap, and nobody pays for 40 million inserts.
What the feed cache holds
The feed cache stores ids, not posts. An entry is a post id and an author id, 8 bytes each, so 16 bytes. The post text, images and like counts live in a separate post cache, and the feed service looks them up in one batch for the page it’s about to show. That keeps each feed small, and an edited post only has to change in one place.
Feeds are also capped. Nobody scrolls back through thousands of posts, so keeping the newest few hundred covers almost everyone. Older entries fall off the end, and a rare deep scroll can fall back to building the rest on read.
Sizing it
Only keep feeds for people who come back. Many registered accounts haven’t opened the app in months, and fanning out to them wastes both memory and inserts. Cache feeds for users active in the last few weeks, and build a feed on read the first time someone returns.
100,000,000 active users × 500 entries × 16 bytes
= 800,000,000,000 bytes ≈ 800 GBThat’s too much for one machine, but it splits cleanly by user id across a cluster of cache nodes, because each feed belongs to exactly one user.
Paging with cursors
A feed is read a page at a time. Asking for page two with ?offset=20 means “skip the first 20”, but the top of a feed keeps moving. If 3 posts arrive while you read page one, skipping 20 lands 3 posts too early, and you see the end of page one again. A deleted post shifts things the other way, and you silently miss one.
A cursor fixes this. Each page comes back with the id of its last post, and the next request asks for posts older than that one, like ?before=p981. The cursor pins the page to a post rather than a position, so nothing above it can shift what comes next. It’s also faster, because the store can jump straight to that point instead of counting past everything before it.
| Offset | Cursor | |
|---|---|---|
| Next page asks for | Skip the first N | Older than this post |
| New posts arrive while you scroll | Repeats posts | Unaffected |
| A post is deleted | Skips one | Unaffected |
| Jump to page 50 | Possible, and slow | Not possible, and nobody does it in a feed |