speculative decoding

Speculative Decoding is an LLM inference acceleration technique that uses a small draft model to propose multiple tokens simultaneously, verified in parallel by the target model — achieving 2-4x speedup without changing model quality.

The Core Problem

• Autoregressive LLM generation is sequential: one token at a time.
• Each forward pass through a 70B+ model takes ~100ms on a GPU.
• The GPU is severely underutilized — most computation is memory-bandwidth bound.
• Solution: Generate multiple tokens per target model forward pass.

How Speculative Decoding Works

1. Draft Phase: A small model (3B, 7B) generates K candidate tokens autoregressively.
2. Verify Phase: The large target model processes all K tokens in ONE forward pass (parallel).
3. Accept/Reject: Accept tokens where target model agrees with draft; reject the first disagreement.
4. Correction: Sample from the corrected distribution at the first rejection point.
5. Result: On average, 3-4 tokens accepted per target model forward pass.

Why It Works

• The verify step is nearly free — a forward pass processing K tokens costs only slightly more than 1 token for memory-bound models.
• The small draft model produces correct tokens most of the time for easy/predictable parts of the text.

Variants

• Self-Speculation / MEDUSA: Train additional "heads" on the target model itself as draft.
• SpecTr: Use multiple draft models; choose the best candidates.
• Prompt Lookup Decoding: Draft from the input prompt itself (fast, no extra model).

Typical Speedups

Speculative decoding is a near-free inference speedup — widely adopted in production LLM serving systems including vLLM, TGI, and Google's production inference.