Serving language models on GPUs
Updated Nov 10, 2026 · 4 min read
Serving a language model is unlike serving a web page. Each answer is built one token at a time, every token needs the whole model, and the hardware is expensive enough that idle time is the biggest cost you have. Almost every serving trick comes from one fact. Generating a token is limited by how fast the GPU can read memory, not by how fast it can do maths.
Prefill and decode
A request runs in two phases. Prefill reads the whole prompt in one pass, works through every token of it in parallel, and produces the first output token. Decode then adds one token per step, each step feeding the newest token back in. Prefill is heavy on maths and fast per token. Decode is light on maths and runs once for every token of the answer.
- The request waits in a queue for room in the batch.
- Prefill reads the prompt and fills the KV cache.
- The first token streams to the user.
- Decode adds a token per step, and each one streams as soon as it exists.
- The answer ends and its KV cache is freed.
Streaming matters because users judge speed by the first words. An answer that takes 15 seconds to finish feels fast if words appear after half a second.
Why batching pays
An 8-billion-parameter model in 16-bit is 16 GB of weights. Every decode step reads all 16 GB from GPU memory, and that read takes far longer than the maths. With one request, the GPU’s maths units mostly wait. With 32 requests in a batch, one read of the weights serves all 32, so a step costs only a little more than it did for one. Total tokens a second climb many times over, and each user’s speed dips a little.
Bigger is not always better. As the batch grows, steps get slower, so each user’s answer arrives more slowly. Every serving setup picks a point on that curve, trading tokens per user for tokens per GPU.
Continuous batching
The simple way to batch waits for a group of requests, runs them together and starts the next group only when the longest answer is done. Short answers sit idle waiting for long ones. Modern servers such as vLLM and TGI use continuous batching instead. After every step, finished requests leave the batch and waiting ones join, so the GPU stays full and nobody waits for a stranger’s long answer.
The KV cache
Each decode step needs to look back at every earlier token. Rather than recompute them, the server keeps each token’s keys and values, the KV cache, on the GPU for as long as the request runs. Its size per token is 2 × layers × KV heads × head size × bytes per number. A typical 8B model with 32 layers, 8 KV heads and heads of 128 numbers, in 16-bit, needs about 128 KB per token.
| Tokens per request | KV cache per request | Requests at once |
|---|---|---|
| 2,000 | 256 MB | about 230 |
| 8,000 | 1 GB | about 58 |
| 32,000 | 4 GB | about 14 |
Prefix caching
A token’s KV cache depends only on the tokens before it. If every request starts with the same 3,000-token system prompt, every request computes the same KV cache for those tokens. Prefix caching keeps it and reuses it, so prefill only runs on the new part. That cuts the time to first token and saves memory, since requests share one copy. Caching whole answers is different, and it only helps when the entire prompt repeats exactly.
What to measure
| Measure | What it tells you | Moves with |
|---|---|---|
| Time to first token | How long until words appear | Queueing and prefill |
| Tokens a second per user | How fast the words appear | Drops as the batch grows |
| Tokens a second per GPU | What you get for the money | Rises as the batch grows |
| KV cache in use | How many more requests fit | Prompt length and batch size |
Sizing and scaling a fleet
Start from tokens. 50 requests a second with 400-token answers is 20,000 tokens a second. If one GPU makes about 2,000 a second with batching, that keeps 10 GPUs fully busy, and you add spare room for spikes and for prefill.
Scaling is slow. A new GPU server has to start, pull a large image and load gigabytes of weights, which takes minutes. GPU utilization is a poor signal too, because with continuous batching it reads near 100% at almost any load, and the engine reserves most GPU memory for the KV cache up front. Scale on the queue instead, meaning requests waiting and time to first token, and keep a warm spare ahead of known busy hours.