Rate limiting with token buckets, windows and one place to count
Updated Nov 8, 2026 · 3 min read
A rate limiter caps how many requests one client may make in a period of time. It protects a service from scrapers, runaway scripts and honest bugs, and it keeps one noisy customer from slowing everyone else down. It sits in front of the real work, usually in an API gateway, and answers one question per request. Does this one get through?
Who counts as one client
Limits are kept per key. For a public API that is usually the API key, because it names a customer and their plan. Without one, you fall back to the user id or the IP address. IPs are a weak key, because a whole office can share one and an attacker can rotate through thousands.
Saying no politely
A request over the limit gets 429 Too Many Requests. Add a Retry-After header with the seconds until the client may try again, and well-behaved clients will back off on their own. Many APIs also send RateLimit-Remaining on every response so clients can slow down before they hit the wall.
The four common algorithms
| Algorithm | How it works | Good at | Watch out for |
|---|---|---|---|
| Fixed window | One counter per key per minute, reset on the minute | Cheapest, with one number per key | Double bursts at the edge of each window |
| Sliding window log | Keep a timestamp for every request in the last minute | Exact | Memory grows with traffic |
| Token bucket | Tokens refill at a steady rate up to a cap; each request takes one | Allows short bursts, holds the average | Two numbers to tune |
| Leaky bucket | Requests queue and leave at a steady rate | Perfectly smooth output | Bursts wait instead of failing fast |
The edge problem with fixed windows
With a limit of 100 a minute and a counter that resets on the minute, a client can send 100 requests at 12:00:59 and another 100 at 12:01:00. That is 200 requests in two seconds, twice what the limit was meant to allow. Sliding windows fix this by always looking back a full minute from now.
Why token buckets are the default
Real clients are bursty. A web app loads a page and fires 30 requests at once, then goes quiet. A token bucket with room for 30 tokens, refilled at 10 a second, lets the whole burst through immediately and still holds the client to 10 a second over time. It only needs two numbers per key, the tokens left and when it last refilled.
tokens = min(capacity, tokens + (now - last) × rate)
last = now
if tokens >= 1: tokens -= 1, allow
else: reject with 429Where to count
Most services run several gateway servers behind a load balancer. If each one counts in its own memory, a client whose requests are spread across 8 gateways gets 8 times its limit. The count has to live in one shared place, usually Redis, where an atomic script reads and updates the bucket in one step so two gateways cannot both spend the last token.
Failing open or closed
If the limiter’s store goes down, you must choose. Failing open lets every request through, so customers keep working but you are unprotected. Failing closed rejects everything. It’s safe, but it turns a limiter outage into a full outage. Most public APIs fail open and alert loudly.