top-p sampling

Top-p sampling, also called nucleus sampling, picks the next token from the smallest set of candidates whose cumulative probability reaches a threshold p — typically 0.9 to 0.95 — rather than from a fixed number of top candidates the way top-k sampling does. Ari Holtzman and colleagues introduced the method in their 2019 paper "The Curious Case of Neural Text Degeneration," which diagnosed a specific failure that greedy and beam search decoding kept producing: text that loops into repetitive phrases, because always picking the single highest-probability token (or the highest-probability full sequence) systematically favors safe, low-information continuations that compound into dull, repeating loops rather than natural language. Nucleus sampling's fix was to truncate the unreliable long tail of unlikely tokens — the part of the distribution responsible for incoherent output when sampled directly — while still sampling randomly enough within the trustworthy remainder to avoid the repetition that deterministic decoding produces.

The entire point of nucleus sampling, versus top-k's fixed cutoff, is that the size of the trustworthy set changes from token to token, and a fixed number can never track that. At a position where the model is extremely confident — say, completing "The capital of France is" — the single correct next token might already account for 95% of the probability mass on its own, so the nucleus at p of 0.9 contains just one or two tokens; at a far more open-ended position, like the start of a creative sentence, dozens of genuinely plausible next tokens might each carry only a percent or two of probability, so reaching the same 0.9 cumulative threshold could require keeping 40 or 50 tokens in the nucleus. A fixed top-k of, say, 10 would badly under-restrict the confident case (letting through 8 tokens that shouldn't have been considered) and badly over-restrict the open-ended case (cutting off dozens of genuinely reasonable options) — the exact mismatch top-p was built to avoid.

Holtzman and colleagues demonstrated the problem nucleus sampling fixes with a direct side-by-side comparison, not just a theoretical argument. Their paper showed GPT-2 continuations under beam search degenerating into visible repetition loops — the same phrase or near-phrase recurring within a handful of sentences — while naive unrestricted random sampling at temperature 1 produced noticeably less coherent text, occasionally veering into outright nonsense by drawing from the unreliable tail of low-probability tokens; nucleus sampling at p around 0.95 sat between those two failure modes, retaining enough randomness to avoid the beam-search loops while truncating away the tail responsible for incoherent sampling.

In production, nucleus sampling almost never runs alone — it's paired with temperature, and the combination is the actual default across most chat APIs. Temperature reshapes the probability distribution itself before any truncation happens, sharpening or flattening it the way covered under temperature, and top-p then truncates whatever shape temperature produced down to the trustworthy nucleus; OpenAI's and Anthropic's APIs both expose a top-p parameter defaulting in the 0.9-to-1.0 range precisely because this stacked combination — moderate temperature, then nucleus truncation — is what most teams converge on after experimentation, rather than either technique alone.

Nucleus sampling's real contribution was recognizing that the right amount of randomness to inject isn't a fixed count or a fixed probability cutoff applied blindly — it's whatever changes with how confident the model actually is at that specific position, and letting the cumulative-probability threshold do that adjusting automatically is simpler and more robust than trying to hand-tune a fixed k for every situation a model might encounter.