Load balancers and how they spread traffic across servers
Updated Nov 8, 2026 · 3 min read
One server eventually runs out of CPU, memory or connections, and when it fails, everything fails with it. A load balancer sits in front of a group of identical servers and spreads requests across them. Clients talk to one address; behind it you can add servers, remove them, or lose one, and the clients never notice.
Layer 4 vs layer 7
| Layer 4 (transport) | Layer 7 (application) | |
|---|---|---|
| Looks at | IP addresses and ports | The HTTP request, with path, headers and cookies |
| Can route by | Connection | URL path, host name, header, user |
| TLS | Usually passed through | Usually terminated here |
| Cost | Very cheap per request | More CPU per request |
| Good for | Raw throughput, non-HTTP protocols | Web apps, APIs, routing /api and /static to different pools |
Choosing a server
- Round robin sends to each server in turn. Simple and fair when requests cost about the same.
- Least connections picks the server with the fewest requests in flight. Better when some requests are slow.
- Weighted gives bigger machines a bigger share.
- Hashing on a key (client IP, user id or the URL) means the same key lands on the same server, which helps when servers keep a local cache. Consistent hashing keeps most keys in place when a server is added or removed.
- Power of two choices picks two servers at random and sends the request to the less busy one. Nearly as good as least connections, with far less shared state.
Health checks and draining
A load balancer is only as good as its idea of which servers are alive. It probes each one, typically an HTTP request to a /health endpoint every few seconds, and stops sending traffic to any server that fails a few in a row. When a server comes back it is added again.
For planned changes such as deploys, servers are drained instead. The load balancer stops sending new requests and lets the ones in flight finish before the server shuts down. Nobody sees an error.
Keep servers stateless
Spreading traffic freely only works if any server can answer any request. Keep session data in a shared store or a signed cookie rather than in one server’s memory. Sticky sessions, which pin a user to one server, are a workaround that brings back the problems you were solving, like uneven load and lost sessions when that server dies.
Do not make the balancer the weak point
A single load balancer is a new single point of failure. Production setups run at least two, either as an active/standby pair sharing one address that moves on failure, or as several active balancers behind DNS or anycast. Managed cloud load balancers do this for you.
Sizing a fleet that survives a failure
Plan from the peak, not the average, keep each server below full load so latency stays flat, and add a spare so losing one server does not overload the rest.
| Step | Working | Result |
|---|---|---|
| Capacity per server at target | 2,000 × 0.7 | 1,400 req/s |
| Servers for peak | 12,000 ÷ 1,400 | 8.6, round up to 9 |
| Survive one failure (N + 1) | 9 + 1 | 10 servers |