← All articles

Load balancers and how they spread traffic across servers

Updated Nov 8, 2026 · 3 min read

One server eventually runs out of CPU, memory or connections, and when it fails, everything fails with it. A load balancer sits in front of a group of identical servers and spreads requests across them. Clients talk to one address; behind it you can add servers, remove them, or lose one, and the clients never notice.

Layer 4 vs layer 7

Layer 4 (transport)Layer 7 (application)
Looks atIP addresses and portsThe HTTP request, with path, headers and cookies
Can route byConnectionURL path, host name, header, user
TLSUsually passed throughUsually terminated here
CostVery cheap per requestMore CPU per request
Good forRaw throughput, non-HTTP protocolsWeb apps, APIs, routing /api and /static to different pools

Choosing a server

  • Round robin sends to each server in turn. Simple and fair when requests cost about the same.
  • Least connections picks the server with the fewest requests in flight. Better when some requests are slow.
  • Weighted gives bigger machines a bigger share.
  • Hashing on a key (client IP, user id or the URL) means the same key lands on the same server, which helps when servers keep a local cache. Consistent hashing keeps most keys in place when a server is added or removed.
  • Power of two choices picks two servers at random and sends the request to the less busy one. Nearly as good as least connections, with far less shared state.

Health checks and draining

A load balancer is only as good as its idea of which servers are alive. It probes each one, typically an HTTP request to a /health endpoint every few seconds, and stops sending traffic to any server that fails a few in a row. When a server comes back it is added again.

For planned changes such as deploys, servers are drained instead. The load balancer stops sending new requests and lets the ones in flight finish before the server shuts down. Nobody sees an error.

UsersLoad balancerprobes /healthApp 1healthyApp 2healthyApp 3failed 3 checks
A failed health check takes a server out of rotation. The others share its traffic until it recovers.

Keep servers stateless

Spreading traffic freely only works if any server can answer any request. Keep session data in a shared store or a signed cookie rather than in one server’s memory. Sticky sessions, which pin a user to one server, are a workaround that brings back the problems you were solving, like uneven load and lost sessions when that server dies.

Do not make the balancer the weak point

A single load balancer is a new single point of failure. Production setups run at least two, either as an active/standby pair sharing one address that moves on failure, or as several active balancers behind DNS or anycast. Managed cloud load balancers do this for you.

Sizing a fleet that survives a failure

Plan from the peak, not the average, keep each server below full load so latency stays flat, and add a spare so losing one server does not overload the rest.

StepWorkingResult
Capacity per server at target2,000 × 0.71,400 req/s
Servers for peak12,000 ÷ 1,4008.6, round up to 9
Survive one failure (N + 1)9 + 110 servers
Say you get 12,000 requests/s at peak, each server handles 2,000/s, target 70% utilization
Target per server1,400 req/s
10 servers, one down~1,330 req/s
9 servers, one down1,500 req/s
Load on each surviving server at peak after one fails, against the 1,400 req/s target.