Serving LLMs in Production: An Inference Infrastructure Primer

Inference is where the bill lands

Training is a one-time cost; inference runs forever. The difference between a viable product and a money pit is usually a handful of serving decisions. Three levers dominate: batching, the KV cache, and quantization.

Continuous batching

GPUs are throughput machines that hate idle time. Naive request-per-forward-pass wastes them. Continuous batching interleaves many requests at the token level, admitting new ones as others finish, keeping the device saturated.

# Sketch of a token-level scheduler
while queue or active:
    admit_new_requests(active, queue, max_batch)
    logits = model.forward(batch_of(active))
    for req in active:
        req.append(sample(logits[req.id]))
    retire_finished(active)

The KV cache

Autoregressive decoding recomputes attention over the whole prefix at every step — unless you cache the keys and values already computed. The KV cache turns quadratic reuse into linear, and it is often the largest consumer of GPU memory, which is why paged attention exists to manage it like virtual memory.

Quantization

Serving weights in 8-bit or 4-bit instead of 16-bit shrinks memory and boosts throughput, usually with minimal quality loss when done with a calibration set. Measure on your own eval before and after — a benchmark you trust keeps quantization honest.

Putting it together

Start by measuring tokens per second and cost per million tokens. Then turn each lever, one at a time, and watch those two numbers. Optimization you cannot measure is just superstition.