← All articles

Serving language models on GPUs

Updated Nov 10, 2026 · 4 min read

Serving a language model is unlike serving a web page. Each answer is built one token at a time, every token needs the whole model, and the hardware is expensive enough that idle time is the biggest cost you have. Almost every serving trick comes from one fact. Generating a token is limited by how fast the GPU can read memory, not by how fast it can do maths.

Prefill and decode

A request runs in two phases. Prefill reads the whole prompt in one pass, works through every token of it in parallel, and produces the first output token. Decode then adds one token per step, each step feeding the newest token back in. Prefill is heavy on maths and fast per token. Decode is light on maths and runs once for every token of the answer.

  1. The request waits in a queue for room in the batch.
  2. Prefill reads the prompt and fills the KV cache.
  3. The first token streams to the user.
  4. Decode adds a token per step, and each one streams as soon as it exists.
  5. The answer ends and its KV cache is freed.

Streaming matters because users judge speed by the first words. An answer that takes 15 seconds to finish feels fast if words appear after half a second.

Why batching pays

An 8-billion-parameter model in 16-bit is 16 GB of weights. Every decode step reads all 16 GB from GPU memory, and that read takes far longer than the maths. With one request, the GPU’s maths units mostly wait. With 32 requests in a batch, one read of the weights serves all 32, so a step costs only a little more than it did for one. Total tokens a second climb many times over, and each user’s speed dips a little.

Bigger is not always better. As the batch grows, steps get slower, so each user’s answer arrives more slowly. Every serving setup picks a point on that curve, trading tokens per user for tokens per GPU.

Continuous batching

The simple way to batch waits for a group of requests, runs them together and starts the next group only when the longest answer is done. Short answers sit idle waiting for long ones. Modern servers such as vLLM and TGI use continuous batching instead. After every step, finished requests leave the batch and waiting ones join, so the GPU stays full and nobody waits for a stranger’s long answer.

The KV cache

Each decode step needs to look back at every earlier token. Rather than recompute them, the server keeps each token’s keys and values, the KV cache, on the GPU for as long as the request runs. Its size per token is 2 × layers × KV heads × head size × bytes per number. A typical 8B model with 32 layers, 8 KV heads and heads of 128 numbers, in 16-bit, needs about 128 KB per token.

Tokens per requestKV cache per requestRequests at once
2,000256 MBabout 230
8,0001 GBabout 58
32,0004 GBabout 14
Requests that fit in 60 GB of KV cache at 128 KB per token.

Prefix caching

A token’s KV cache depends only on the tokens before it. If every request starts with the same 3,000-token system prompt, every request computes the same KV cache for those tokens. Prefix caching keeps it and reuses it, so prefill only runs on the new part. That cuts the time to first token and saves memory, since requests share one copy. Caching whole answers is different, and it only helps when the entire prompt repeats exactly.

What to measure

MeasureWhat it tells youMoves with
Time to first tokenHow long until words appearQueueing and prefill
Tokens a second per userHow fast the words appearDrops as the batch grows
Tokens a second per GPUWhat you get for the moneyRises as the batch grows
KV cache in useHow many more requests fitPrompt length and batch size

Sizing and scaling a fleet

Start from tokens. 50 requests a second with 400-token answers is 20,000 tokens a second. If one GPU makes about 2,000 a second with batching, that keeps 10 GPUs fully busy, and you add spare room for spikes and for prefill.

Scaling is slow. A new GPU server has to start, pull a large image and load gigabytes of weights, which takes minutes. GPU utilization is a poor signal too, because with continuous batching it reads near 100% at almost any load, and the engine reserves most GPU memory for the KV cache up front. Scale on the queue instead, meaning requests waiting and time to first token, and keep a warm spare ahead of known busy hours.