The physics and engineering of high-throughput, low-latency machine learning serving — from prefill compute saturation to token decode memory bandwidth bottlenecks.
ML inference is the serving side: weights are fixed, and the job is turning requests into tokens (or predictions) under latency, throughput, and cost constraints. For LLMs specifically, the shape of the problem is set by two very different phases.
Prefill processes the whole prompt at once. It's a big matmul over many tokens, so it's compute-bound — you're chasing FLOPs utilization on the tensor cores. Time-to-first-token lives here.
Decode generates one token at a time. Each step reads the entire weight set and the whole KV cache to produce a single token, so arithmetic intensity is terrible and you're memory-bandwidth-bound. Inter-token latency lives here. This is why an H100's 3.35 TB/s of HBM matters more than its FLOPs for decode, and why batching is the central trick: more sequences per weight-read amortizes the bandwidth cost until you run out of KV cache memory.
Swap finished sequences out of the batch mid-flight instead of waiting for the slowest one. Biggest single throughput win in practice.
KV cache in fixed blocks instead of contiguous reservations, killing fragmentation and letting you run much larger batches. vLLM's original contribution.
FP8 or INT4 weights cut bandwidth proportionally. Weight-only (AWQ, GPTQ) helps decode a lot; activation quantization additionally helps prefill compute. FP8 on Hopper/Blackwell is close to free quality-wise; INT4 needs care.
A small draft model proposes k tokens, the big model verifies them in one forward pass. Trades spare compute for bandwidth, so it's a direct answer to the decode bottleneck. Medusa and EAGLE are the self-drafting variants.
Run the two phases on separate hardware pools since they want different things. Increasingly standard at scale.
Reuse KV for shared system prompts across requests.
Simulate roofline memory-vs-compute bottlenecks, decode throughput, KV-cache footprint, and batch serving curves on real silicon architectures.
curl -X POST https://www.amemories.com/edge/infer \
-H "Content-Type: application/json" \
-d '{"peak_tflops":990,"hbm_bw_gbs":3350,"hbm_cap_gb":80,"params_b":70,
"hidden_dim":8192,"layers":80,"n_heads":64,"n_kv_heads":8,
"seq_len":4096,"batch":1,"weight_bytes":2}'
Returns JSON with outputs (decode throughput, prefill throughput, TTFT, latency, bottleneck classification, arithmetic intensity, utilization, memory footprint), the full batch serving curve, and compute node timings.