model deployment optimization

Model deployment is the gap between a model that works in a training notebook and a model that answers real requests under a real latency budget, and almost everything in that gap is about removing work the training-time version did for convenience but production can't afford. A model trained in PyTorch with Python's dynamic graph and eager execution is flexible to iterate on; a model serving thousands of requests per second needs a static, compiled, heavily optimized version of the exact same computation — and the tooling that bridges those two versions is most of what "deployment" actually means in practice.

Operator fusion collapses several small, separately-launched operations into one, and the savings come from eliminating the overhead between them, not from doing less math. A naive execution graph might run a convolution, then a batch normalization, then a ReLU activation as three separate kernel launches, each one writing its full output to memory before the next operation reads it back — fusion compiles the three into a single kernel that keeps intermediate values in fast on-chip memory the whole time, never materializing them to slower memory at all. The arithmetic performed is identical; what disappears is the memory traffic and kernel-launch overhead between steps, which is frequently the larger cost for small operations, especially on GPUs where launching a kernel has fixed overhead regardless of how little work it does.

Quantization reduces a model's numeric precision, and the deployment-time version differs from training-time quantization in what it's optimizing for. INT8 post-training quantization converts a model's weights and activations from 32-bit floating point to 8-bit integers after training is already complete, typically delivering a 2-to-4x inference speedup on hardware with INT8-optimized execution paths, at the cost of needing a calibration pass on representative data to choose good quantization ranges. Mixed-precision inference instead keeps numerically sensitive operations — softmax, layer normalization — in higher precision while running the bulk of the computation in FP16 or BF16, which on GPUs with dedicated Tensor Core hardware for those lower-precision formats can roughly double throughput with minimal measured accuracy loss. These are deployment-stage decisions made after a model is trained and validated, distinct from quantization-aware training, which bakes simulated quantization into the training process itself specifically to recover accuracy that post-training quantization alone would lose at more aggressive precision levels.

Continuous batching solved a specific inefficiency static batching couldn't: requests of very different lengths finishing at very different times. Traditional dynamic batching groups several requests together and runs them as one batch, but if one request in the batch needs far more generation steps than the others — common for LLM text generation, where output length varies enormously — the whole batch is stuck waiting for the longest one to finish before any slot frees up. Continuous batching instead adds a new incoming request to an already-running batch the moment any slot in it frees up, rather than waiting for the full batch to complete together, which is why it became the standard approach for LLM serving specifically, where request length variance is large enough that static batching leaves significant throughput on the table.

KV caching and paged attention solved two different problems in autoregressive generation, one about redundant computation and one about memory fragmentation. Caching the key and value tensors from previous generation steps avoids recomputing attention over the entire sequence from scratch at every new token, cutting per-token computation from quadratic to roughly linear in sequence length — a foundational optimization nearly every LLM serving system uses. vLLM's paged attention then addressed a separate problem this caching introduces: naively allocated KV cache memory fragments badly as requests of varying length come and go, wasting a large fraction of GPU memory on unusable gaps. Paged attention manages the KV cache in fixed-size blocks the way an operating system manages virtual memory pages, letting requests share and reclaim memory far more efficiently and substantially increasing how many concurrent requests a given amount of GPU memory can actually serve.

Read model deployment as a translation problem, not a packaging problem: every optimization — fusion, quantization, continuous batching, paged attention — exists because the representation that makes a model easy to train and iterate on is deliberately not the representation that makes it cheap to run at scale, and the entire deployment toolchain is the systematic work of converting from one to the other without changing what the model actually computes.