Top-k sampling

Top-k sampling is a decoding strategy for autoregressive language models that restricts each generation step to the k most probable next tokens. At every position the model emits logits over the entire vocabulary, a softmax converts them into a probability distribution, the k highest-probability tokens are kept, their probabilities are rescaled to sum to one, and the next token is drawn at random from that truncated distribution ^[1][2]. Every token ranked below k receives probability zero for that step.

The method sits between two extremes of next-token prediction. Setting k to 1 collapses it into greedy decoding, which always emits the single most likely token. Setting k to the full vocabulary size recovers pure sampling, which draws directly from the model's distribution and therefore keeps the long tail of tens of thousands of low-probability candidates in play ^[2]. Top-k established the truncation approach in neural text generation, and although nucleus sampling displaced it as the usual recommendation after 2019, the top_k parameter remains present in nearly every inference stack and still appears in the sampling settings shipped with 2025-era open-weight models ^[3][4][5][6].

Definition and mechanics

Holtzman et al. give the standard formalization. Given a distribution P(x | x_1:i-1) over the vocabulary V, the top-k vocabulary is the subset of size k that maximizes the total probability mass of its members. The mass of that subset is then used as the denominator for a rescaled distribution, which assigns each kept token its original probability divided by that sum and assigns zero to everything else. Sampling proceeds from the rescaled distribution ^[2].

Two properties follow directly from that construction. First, the size of the candidate pool is fixed by the user and does not respond to the shape of the distribution. Second, the amount of probability mass being renormalized varies from step to step, sometimes drastically: at a confident step the top k tokens can account for nearly all of the mass, while at an uncertain step they may account for only part of it. Holtzman et al. singled out this varying scaling factor as the structural difference between top-k and nucleus sampling, which fixes the mass and lets the pool size float instead ^[2].

The computational cost is a partial sort over the vocabulary at each step, which is small next to the forward pass itself. Because the filter operates on logits alone, it drops into any inference loop and chains with the other logit-level filters described below ^[4][5].

Origins in story generation

Top-k random sampling was introduced by Angela Fan, Mike Lewis, and Yann Dauphin in "Hierarchical Neural Story Generation," presented at ACL 2018 ^[1]. The paper's subject was a two-stage story generator built on a dataset of 303,358 human-written stories paired with writing prompts scraped from Reddit's WritingPrompts forum, and the sampler was a practical fix rather than the headline contribution. Their description is one sentence long: "We randomly sample from the k = 10 most likely candidates from this distribution." The justification was empirical and comparative: "We find this sampling strategy substantially more effective than beam search, which tends to produce common phrases and repetitive text from the training set" ^[1]. The same k = 10 scheme was used for generating the prompts themselves.

GPT-2 adopted the method a year later and gave it visibility far beyond story generation. The GPT-2 paper cites Fan et al. directly and states that its published sample completions used "Top-k random sampling with k = 40" ^[3]. For the zero-shot summarization experiment the same paper used a much tighter setting, k = 2, on the grounds that it "reduces repetition and encourages more abstractive summaries than greedy decoding" ^[3]. That spread, k = 2 for a constrained task and k = 40 for open-ended continuation, is an early illustration of how task-dependent the parameter is.

Why truncation is needed at all

The case for discarding part of the distribution rests on a pair of failures at opposite ends. Maximization-based decoding produces "output text that is bland, incoherent, or gets stuck in repetitive loops," a behavior Holtzman et al. named neural text degeneration ^[2]. Pure sampling fails in the other direction. Holtzman et al. blame what they call the unreliable tail: tens of thousands of individually low-probability candidates that are over-represented in aggregate, which is why sampling from the untruncated distribution drifts into text unrelated to its context ^[2].

The paper's main comparison table, computed on GPT-2 Large continuations of WebText, quantifies both failures. Perplexity here is the perplexity of the generated text under the model, so values far below the human figure indicate text that is too predictable rather than text that is better.

Scroll sideways for more →

Source: Holtzman et al., Table 1 ^[2].

The repetition column counts how often a method gets stuck in a loop within the first 200 tokens, where a phrase of at least two words repeating three or more times at the end of a generation counts as a repetition. Greedy decoding hit that condition in 73.66 percent of generations and beam search with 16 beams in 28.94 percent, against 0.28 percent for human text ^[2]. The paper also documented the mechanism behind those loops: once a phrase has been emitted, the probability the model assigns to repeating it rises with each further repetition, which creates a positive feedback loop that maximization-based search walks straight into ^[2].

The fixed-k problem and nucleus sampling

Holtzman et al. accepted that top-k beats both beam search and full-distribution sampling, then argued that a constant k is the wrong knob. Their reasoning turns on how much the shape of the next-token distribution varies. In some contexts "the head of the next word distribution can be flat across tens or hundreds of reasonable options," typically where a generic noun or verb could follow; in others almost all the mass sits on one or two tokens, as in the completion of a fixed phrase ^[2]. A single k cannot serve both: "if k is small, in some contexts there is a risk of generating bland or generic text, while if k is large the top-k vocabulary will include inappropriate candidates which will have their probability of being sampled increased by the renormalization" ^[2].

Nucleus sampling, which the same paper introduced, inverts the control. Instead of fixing the number of candidates it fixes the probability mass: the smallest set of highest-probability tokens whose cumulative probability reaches p is kept, so the pool expands where the model is uncertain and contracts where it is confident ^[2].

The measured gap is visible in the table above. At k = 40 the generated text scored a perplexity of 6.88 against 12.38 for human text, meaning it was substantially too probable, and its HUSE score was 0.19. Pushing k to 640 brought the distributional statistics close to human values and lifted HUSE to 0.94, but Holtzman et al. noted that generations at high k show high variance in likelihood and observable incoherence. Nucleus sampling at p = 0.95 reached 0.97 while matching human perplexity closely ^[2]. The Hugging Face explainer that popularized both methods put the same criticism plainly: top-k "does not dynamically adapt the number of words that are filtered from the next word probability distribution" ^[7].

Interaction with temperature

Temperature rescales the logits before the softmax; Holtzman et al. note that setting it below 1 "skews the distribution towards high probability events," which implicitly lowers the mass in the tail ^[2]. Its relationship to top-k is unusual, and different from its relationship to the probability-based truncation methods.

Dividing logits by any positive temperature preserves their ordering, so the identity of the top k tokens does not change. The authors of top-n-sigma sampling make the point directly: "While top-k sampling does maintain temperature invariance, it uses a fixed k value, which merely shifts the problem" ^[8]. The probability-based filters have no such property. For top-p and min-p, the same paper observes, "the selected token set varies with temperature": raising the temperature flattens the distribution, so more tokens clear the threshold and the tail creeps back in. It characterizes existing methods including top-p and min-p as ones that "inadvertently include more noise tokens at higher temperatures" ^[8].

Because the composition order matters for those other filters, implementations document it. Transformers builds its logit processor list with temperature first and the truncation filters after it ^[9]. The llama.cpp server takes the opposite convention, with a default sampler chain of penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature that applies temperature last ^[5]. Since top-k's candidate set does not move with temperature, the two orders coincide for top-k alone; they diverge once top-p or min-p is in the chain ^[8].

Vendor guidance generally discourages tuning several of these at once. Google's generative AI SDK states of top-p that "It's recommended to adjust either temperature or top_p, but not both" ^[10], and Anthropic labels both top_k and top_p as "Recommended for advanced use cases only" ^[11].

Repetition and degeneration

Truncation controls which tokens are eligible but does nothing about the repetition feedback loop itself, so production stacks pair it with explicit repetition controls. The best known is the penalized sampling of the CTRL model, which divides the logits of already-generated tokens by a penalty factor before the softmax; Keskar et al. wrote that greedy sampling with a penalty of about 1.2 "yields a good balance between truthful generation and lack of repetition" ^[12]. That paper is the citation Transformers gives for its repetition_penalty argument, and the library also offers no_repeat_ngram_size, which forbids any n-gram of the given size from occurring twice ^[4].

Some 2025 model cards go further and treat sampling settings as a correctness requirement rather than a style preference. The Qwen3 card instructs users to "DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions," and recommends presence_penalty between 0 and 2 for repetitive outputs ^[6]. DeepSeek-R1 similarly asks users to "Set the temperature within the range of 0.5-0.7 (0.6 is recommended)" to avoid endless repetitions and incoherent output ^[13].

Later truncation methods

Research after 2019 largely kept the truncation idea and changed the criterion. The methods below are all supported in at least one mainstream inference stack.

Scroll sideways for more →

Min-p has had the broadest uptake of the newer ones. It keeps every token whose probability is at least a base value multiplied by the probability of the most likely token, so the cutoff tightens automatically when the model is confident and loosens when it is not ^[16]. Because the scaling is relative to the peak rather than to a cumulative sum, its authors argue it holds up better at high temperature than top-p. It is implemented in Transformers, vLLM, and llama.cpp, and llama.cpp's server enables it by default at 0.05 while leaving typical-p off ^[4][5][18].

Min-p is also the subject of an unusually public methodological dispute. The paper was selected for an oral presentation at ICLR 2025, and in June 2025 Rylan Schaeffer, Joshua Kazdan, and Yegor Denisov-Blanch published a re-examination concluding that "evidence presented in the original paper fails to support claims that min-p improves quality, diversity, or a trade-off between quality and diversity" ^[19]. Their critique covers the human evaluation, the benchmark sweeps once hyperparameter counts are controlled, the LLM-as-judge results, and adoption statistics that were subsequently removed from the paper ^[19]. The original authors ran a further human evaluation in response, which the critique reports as also failing to separate min-p from baselines ^[19]. Both papers remain on arXiv, and the sampler remains in the libraries either way.

Why truncation works

Two lines of theory explain what these methods buy. Meister et al. describe truncation methods collectively as sampling adapters and argue that the shift they impose "can be viewed as a trade-off between precision and recall": the model loses the ability to produce some valid strings, but its precision on desirable text increases. They note that this trade is invisible to perplexity and shows up only in precision-emphasizing measures ^[20].

Finlayson et al. supply a guarantee for the threshold family. They prove that truncation methods discarding tokens below a probability threshold "can guarantee that all sampled tokens have nonzero true probability," while conceding that a threshold is a coarse instrument that also discards some tokens the true distribution does allow. Their proposed alternative exploits the softmax bottleneck to identify legitimate tokens without relying on a threshold at all ^[21].

Support in libraries and APIs

Scroll sideways for more →

Hosted providers differ on whether to surface the parameter at all, and one of them has begun withdrawing it. OpenAI's published API specification exposes top_p and temperature but no rank-based cutoff ^[22]. Google's SDK accepts top_k, and Anthropic's Messages API reference still documents it as an advanced control ^[10][11], but Anthropic's model migration guide states: "Starting with Claude Opus 4.7, setting temperature, top_p, or top_k to any non-default value returns a 400 error." It directs callers to omit the parameters entirely and applies the same restriction to Claude Sonnet 5 ^[23]. Prompting, rather than sampler tuning, is the recommended replacement ^[23]. Open-weight model cards, in turn, increasingly ship a specific k. Qwen3 recommends TopK=20 together with Temperature=0.6, TopP=0.95, and MinP=0 for thinking mode, and Temperature=0.7 with TopP=0.8 for non-thinking mode ^[6]. That combination, a tight rank cutoff stacked underneath a mass cutoff, is close to what the Hugging Face explainer recommended in 2020: top-p used with top-k "can avoid very low ranked words while allowing for some dynamic selection" ^[7].

See also

• Decoding strategies
• Top-p sampling
• Temperature sampling
• Greedy decoding
• Beam search
• Speculative decoding

References

1. ^Fan, Angela; Lewis, Mike; Dauphin, Yann. "Hierarchical Neural Story Generation." Proceedings of ACL 2018. aclanthology.org/P18-1082.pdf (arXiv preprint: arxiv.org/...1805.04833)
2. ^Holtzman, Ari; Buys, Jan; Du, Li; Forbes, Maxwell; Choi, Yejin. "The Curious Case of Neural Text Degeneration." ICLR 2020 (arXiv v1 22 April 2019, v2 14 February 2020). arxiv.org/...1904.09751
3. ^Radford, Alec et al. "Language Models are Unsupervised Multitask Learners." OpenAI, 2019. cdn.openai.com/...upervised_multitask_learners.pdf
4. ^Hugging Face. "Generation" (GenerationConfig API reference), Transformers documentation. huggingface.co/...text_generation
5. ^llama.cpp. Server README (sampling parameters and default sampler order). github.com/...README.md
6. ^Qwen team. Qwen3-235B-A22B model card, "Best Practices" section. huggingface.co/...Qwen3-235B-A22B
7. ^von Platen, Patrick. "How to generate text: using different decoding methods for language generation with Transformers." Hugging Face blog. huggingface.co/...how-to-generate
8. ^Tang, Chenxia; Liu, Jianchun; Xu, Hongli; Huang, Liusheng. "Top-nσ: Not All Logits Are You Need." arXiv, 12 November 2024. arxiv.org/...2411.07641
9. ^Hugging Face Transformers source, `generation/utils.py` (order of logits processors). github.com/...utils.py
10. ^Google. google-genai Python SDK, `types.py` (GenerationConfig `top_k` and `top_p` field documentation). github.com/...types.py
11. ^Anthropic. Messages API reference (`temperature`, `top_k`, `top_p`). platform.claude.com/...messages
12. ^Keskar, Nitish Shirish; McCann, Bryan; Varshney, Lav R.; Xiong, Caiming; Socher, Richard. "CTRL: A Conditional Transformer Language Model for Controllable Generation." arXiv, 11 September 2019. arxiv.org/...1909.05858
13. ^DeepSeek. DeepSeek-R1 model card, "Usage Recommendations." huggingface.co/...DeepSeek-R1
14. ^Meister, Clara; Pimentel, Tiago; Wiher, Gian; Cotterell, Ryan. "Locally Typical Sampling." arXiv, 1 February 2022. arxiv.org/...2202.00666
15. ^Hewitt, John; Manning, Christopher D.; Liang, Percy. "Truncation Sampling as Language Model Desmoothing." arXiv, 27 October 2022. arxiv.org/...2210.15191
16. ^Nguyen, Minh Nhat; Baker, Andrew; Neo, Clement; Roush, Allen; Kirsch, Andreas; Shwartz-Ziv, Ravid. "Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs." arXiv, 1 July 2024. arxiv.org/...2407.01082
17. ^Baghaei Potraghloo, Erfan; Azizi, Seyedarmin; Kundu, Souvik; Pedram, Massoud. "Top-H Decoding: Adapting the Creativity and Coherence with Bounded Entropy in Text Generation." arXiv, 2 September 2025. arxiv.org/...2509.02510
18. ^vLLM. `sampling_params.py` source (SamplingParams defaults). github.com/...sampling_params.py
19. ^Schaeffer, Rylan; Kazdan, Joshua; Denisov-Blanch, Yegor. "Min-p, Max Exaggeration: A Critical Analysis of Min-p Sampling in Language Models." arXiv, 16 June 2025. arxiv.org/...2506.13681
20. ^Meister, Clara; Pimentel, Tiago; Malagutti, Luca; Wilcox, Ethan G.; Cotterell, Ryan. "On the Efficacy of Sampling Adapters." ACL 2023. arxiv.org/...2307.03749
21. ^Finlayson, Matthew; Hewitt, John; Koller, Alexander; Swayamdipta, Swabha; Sabharwal, Ashish. "Closing the Curious Case of Neural Text Degeneration." arXiv, 2 October 2023. arxiv.org/...2310.01693
22. ^OpenAI. Public OpenAPI specification for the OpenAI API. github.com/...openapi.yaml
23. ^Anthropic. "Model migration guide" (sampling parameter removal on Claude Opus 4.7 and later, and on Claude Sonnet 5). platform.claude.com/...migration-guide