Top-k Sampling vs Top-p Sampling
Top-k sampling always restricts each next token to the k most probable candidates. Top-p (nucleus) sampling instead keeps the smallest set of candidates whose probabilities add up to p, so the number of candidates grows when many tokens are plausible and shrinks when one is clearly best.
Definitions
- Top-k Sampling PTL-0108
- Top-k sampling is a decoding method that, at each step, restricts sampling to the k most probable next tokens and renormalizes their probabilities.
- Top-p Sampling PTL-0007
- Top-p sampling, also called nucleus sampling, draws each next token only from the smallest set of candidates whose cumulative probability exceeds a threshold p.
Differences
| Top-k Sampling | Top-p Sampling | |
|---|---|---|
| Cutoff | A fixed number of tokens | A probability mass |
| Candidate set size | Constant | Varies with the model's confidence |
| Weakness | Too narrow or too wide depending on the context | Can still include unlikely tokens when the distribution is flat |
Which to use
- Top-k is a simple guard against very unlikely tokens.
- Top-p adapts to the model's confidence; either is usually combined with temperature.
Sources
- Fan et al. (2018). Hierarchical Neural Story Generation.
- Holtzman et al. (2019). The Curious Case of Neural Text Degeneration.
- Holtzman et al. (2019). The Curious Case of Neural Text Degeneration.
Cite this page
Protologue. (2026). Top-k Sampling vs Top-p Sampling. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0). https://protologue.com/compare/top-k-vs-top-p-sampling/