Top-p sampling

Top-p sampling, also called nucleus sampling, is a stochastic decoding strategy used to generate sequences from autoregressive probabilistic models such as large language models. At each step the model gives a probability to every possible next token. The method sorts these tokens by probability, keeps the smallest set whose cumulative probability exceeds a chosen threshold p, rescales the probabilities inside that set so they sum to one, and then draws the next token at random from it. Because the size of this set shrinks when the model is confident and grows when it is uncertain, the technique adapts to the situation and is more flexible than top-k sampling, which always draws from a fixed number of candidates. It was originally proposed by Ari Holtzman, Yejin Choi and colleagues in 2019 to reduce the repetitive and nonsensical text that deterministic methods such as beam search tend to produce, and it has since spread to protein engineering, drug design and geophysical denoising.