A design review checklist for the flaws that hurt first
Updated Nov 10, 2026 · 4 min read
A design review is cheap insurance. A diagram and a page of text can hide a flaw that would take a week of incidents to find in production. The same few flaws turn up again and again, and each one has a question that exposes it. Ask what happens when a part dies, when a request repeats, when data goes stale, and when two requests race.
One of anything
Walk the diagram and count. Any box with no second copy is a single point of failure, and the dangerous ones are usually small and easy to overlook, like one Redis node holding sessions, one cron server, or one DNS provider. Managed services that already run across zones are fine. A database with a standby that takes over on its own is a short blip. A node with no copy at all is an outage.
Requests that repeat
Networks drop answers. A client that times out doesn’t know whether its request failed or only its reply did, so it retries. For reads that’s harmless. For anything that moves money or creates something, the retry repeats the action. The fix is an idempotency key. The client makes one random key per action and sends it with every try, the server stores the result under that key, and a repeat gets the first result back.
Retries that multiply
When services call each other in a chain and each one retries, the retries multiply. If each of 3 services tries 4 times, one user request can become 4 × 4 × 4 = 64 attempts on the database, right when it is struggling to recover.
- Retry in one layer only, usually the one closest to the user, and let the others fail fast.
- Wait longer after each failure, and add randomness so clients don’t retry in lockstep.
- Cap retries with a budget, for example no more than 10% extra load.
- Set a timeout on every call, so a hung dependency can’t hold your threads forever.
Caches in front of money
A cache serves a copy, and copies go stale. That is fine for a product page, where a price an hour old is a small problem. It is not fine at checkout, where it means charging a price that no longer exists. Keep caches for display, and read anything you charge for from the source of truth, or check that what the user saw is still current.
Keys everyone writes to
Sharding spreads load only if the key spreads it. A key that everyone writes at the same moment, like today’s date or a single popular account, sends all writes to one shard while the rest sit idle. That’s a hot shard. Pick a key with many values in use at once, such as a hash of the user or device id, and check how your busiest moment spreads, not your average one.
Check, then act
Code that reads a value, decides, then writes leaves a gap where another request slips in. A single-use coupon gets used twice, the last seat gets sold twice. Do the check and the change in one statement, such as an update that only matches when the coupon is unused, and act only if it changed a row. A unique constraint does the same job for inserts.
Two writes that should be one
Committing to a database and then publishing an event are two separate steps, and a crash between them leaves the database and the rest of the world disagreeing. A transactional outbox fixes it. Write the event to an outbox table in the same transaction as the change, and let a separate relay publish it. The relay may publish twice, so consumers ignore repeats.
Waiting on someone else
Anything a request waits for adds its slowness and its outages to yours. If checkout waits for the email provider, checkout is down when the email provider is. Ask of every call whether the user needs its answer right now. If not, put it on a queue and do it after you reply.
The checklist
| Ask | What goes wrong | Usual fix |
|---|---|---|
| Is there one of anything? | It dies and takes everything with it | A second copy that takes over on its own |
| What if this request repeats? | Money moves twice | Idempotency key |
| How do retries add up? | A blip becomes a storm | Retry in one layer, with backoff, jitter and a budget |
| Can a cached value be charged? | Stale prices on real payments | Read from the source of truth at checkout |
| Where does the busiest moment land? | One hot shard | A key with many values in use at once |
| Can two requests race here? | Used twice, sold twice | One conditional statement, or a unique constraint |
| Are two systems written separately? | They disagree after a crash | Transactional outbox |
| Does the user need this answer now? | Someone else’s outage is yours | A queue and a timeout |