# Next-token prediction

> Source: https://aiwiki.ai/wiki/next_token_prediction
> Updated: 2026-07-24
> Fact-checked: 2026-07-24
> Categories: Large Language Models, Machine Learning, Natural Language Processing, Training & Optimization
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Next-token prediction." aiwiki.ai, 24 Jul 2026. https://aiwiki.ai/wiki/next_token_prediction
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

Next-token prediction is the training objective used by most modern [language models](https://aiwiki.ai/wiki/language_model): the model reads a prefix of tokenized text, outputs a probability distribution over which token comes next, and training adjusts its parameters to raise the probability it assigned to the token that actually followed. Supervision comes from the text itself rather than from human labels, which makes the objective a form of [self-supervised learning](https://aiwiki.ai/wiki/self-supervised_learning) and lets it consume any corpus that can be tokenized. [GPT](https://aiwiki.ai/wiki/gpt), [Llama](https://aiwiki.ai/wiki/llama), and nearly every other [large language model](https://aiwiki.ai/wiki/large_language_model) in production use are pretrained this way [1][2][31].

The objective itself is old. It dates to statistical language modeling in the mid-twentieth century and was standard practice for [n-gram](https://aiwiki.ai/wiki/n-gram) models and early neural language models long before [transformers](https://aiwiki.ai/wiki/transformers) existed [3][4]. What changed after 2018 was the scale at which it was applied and what came out the other side: models trained only to continue text turned out to translate, summarize, and answer questions without being trained on any of those tasks explicitly [2][5].

The distance between the plainness of the objective and the breadth of the resulting behavior is why next-token prediction became a subject of argument rather than a technical footnote. One position treats accurate prediction as evidence that a model has built an internal account of whatever produced the text [6]; another treats the left-to-right, one-token-at-a-time format as a structural limit [7]. Both have supporting results, and the practical response since 2024 has been to modify the objective rather than replace it [8][31].

## Formulation

A language model assigns a probability to a token sequence by factorizing it with the chain rule of probability. The GPT-2 paper factorizes the joint probability over symbols as a product of conditional probabilities, `p(x) = prod_i p(s_i | s_1, ..., s_(i-1))`, attributing the approach to work by Jelinek and Mercer in 1980 and by Bengio and colleagues in 2003 [2][4]. Every conditional term in that product is one next-token prediction, and a model that generates by sampling those terms in order is an [autoregressive model](https://aiwiki.ai/wiki/autoregressive_model).

Training maximizes the log-likelihood of the observed corpus under this factorization. The original GPT paper writes the pretraining objective as maximizing the sum over positions of `log P(u_i | u_(i-k), ..., u_(i-1); Theta)`, where `k` is the size of the [context window](https://aiwiki.ai/wiki/context_window) and the conditional probability is produced by a multi-layer Transformer decoder whose final hidden state is mapped to vocabulary [logits](https://aiwiki.ai/wiki/logits) and passed through a [softmax](https://aiwiki.ai/wiki/softmax) [1][9]. Maximizing that log-likelihood is the same as minimizing [cross-entropy](https://aiwiki.ai/wiki/cross-entropy) between the model's predicted distribution and a one-hot target, which is why implementations describe the [loss function](https://aiwiki.ai/wiki/loss_function) as cross-entropy and why the two framings are used interchangeably. The DeepSeek-V3 technical report, for instance, writes its per-position loss explicitly as `CrossEntropy` over ground-truth tokens [15].

Because the target for every position is simply the token that follows it, a single sequence of length `n` supplies `n` supervised examples at no extra labeling cost. That density of signal is what makes the objective practical at trillion-token scale.

## Perplexity and bits per token

Model quality on the objective is usually reported as [perplexity](https://aiwiki.ai/wiki/perplexity), the inverse probability the model assigns to a held-out text, normalized by its length. A uniform model over a three-word vocabulary has perplexity 3 on any test set drawn from it; a model that concentrates 0.8 of its mass on the word that actually recurs scores 1.89 on the same text [10]. Perplexity is often described as a weighted branching factor for this reason, and it derives from cross-entropy: it is the exponential of the average per-token cross-entropy loss, which is why training curves in log-loss and evaluation tables in perplexity carry the same information [10]. Loss is also reported in bits per token or bits per byte, which sidesteps the fact that the perplexities of two language models are comparable only if they use identical vocabularies [10].

The information-theoretic framing predates neural models by decades. In 1951 [Claude Shannon](https://aiwiki.ai/wiki/claude_shannon) estimated the [entropy](https://aiwiki.ai/wiki/entropy) of printed English using a method that depended on experimental results in predicting the next letter of a text when the preceding text is known. Counting letter frequencies alone gave 4.14 bits per letter against a 4.7-bit uniform baseline over a 26-letter alphabet; Shannon concluded that in ordinary literary English, long-range statistical effects spanning up to 100 letters reduce the entropy to something of the order of one bit per letter, a redundancy of roughly 75 percent [3]. His experiment is next-token prediction with a human in the model's seat, and it established that reducing predictive loss on natural text requires exploiting progressively longer-range structure.

## Teacher forcing and autoregressive inference

Training and generation use the objective differently, and the mismatch has consequences.

During training, the model conditions on the ground-truth prefix at every position rather than on its own earlier outputs. This is teacher forcing, a technique carried over from the recurrent-network literature of the 1980s [11][8]. With a causal attention mask, all positions in a sequence can be scored in a single forward pass, so teacher forcing is also what makes transformer pretraining parallel across sequence length rather than sequential [9].

At generation time the model has no ground truth to lean on and must condition on tokens it produced itself. Work on [recurrent neural networks](https://aiwiki.ai/wiki/recurrent_neural_network) in 2015 identified the resulting train-test discrepancy as a source of brittleness, noting that generation errors "may accumulate along the way," and proposed sequence-level training and scheduled sampling as remedies [12][13]. The same concern reappears in criticism of large language models, where it is usually framed as compounding error over long outputs.

Bachmann and Nagarajan argued in 2024 that the two phases should be analyzed separately, and that the compounding-error criticism assumes teacher forcing has already produced an accurate next-token predictor. Their paper describes a failure mode in which teacher forcing does not produce one: on a lookahead task (path finding on a randomly generated graph), the ground-truth prefix leaks enough information that the model can fit the later positions with a shortcut they label Clever Hans cheating, which starves the gradient signal needed to learn the real procedure and leaves the first node of the path as what they call an indecipherable token, the one position the shortcut cannot fit. Both Transformer and Mamba models failed the task, and a teacherless objective that predicts multiple tokens in advance using dummy tokens gave preliminary evidence of a fix [8].

Decoding introduces a second set of choices independent of the objective. The distribution the model produces at each step can be consumed by [greedy decoding](https://aiwiki.ai/wiki/greedy_decoding), [beam search](https://aiwiki.ai/wiki/beam_search), [temperature](https://aiwiki.ai/wiki/temperature_sampling) scaling, [nucleus sampling](https://aiwiki.ai/wiki/top_p_sampling), or [speculative decoding](https://aiwiki.ai/wiki/speculative_decoding), and reuse of past activations through a [KV cache](https://aiwiki.ai/wiki/kv_cache) is what keeps per-token cost from growing quadratically during generation.

## The role of tokenization

Next-token prediction operates over tokens, not characters or words, so [tokenization](https://aiwiki.ai/wiki/tokenization) shapes what the objective can express. Subword vocabularies built with [byte pair encoding](https://aiwiki.ai/wiki/byte_pair_encoding), introduced for neural machine translation in 2015, solved the open-vocabulary problem by letting rare words decompose into frequent pieces [14]. Production models use byte-level variants: DeepSeek-V3 uses byte-level BPE with an extended vocabulary of 128K tokens [15].

The consequences are not purely cosmetic. Because numbers are segmented into multi-digit chunks that depend on the tokenizer, arithmetic accuracy varies with how a number happens to be split, though models can often recover performance when prompted to convert between tokenizations step by step [16]. DeepSeek's team documented a related artifact they call token boundary bias, arising when the tokenizer merges punctuation with line breaks, and mitigated it by randomly splitting such tokens during training [15].

Two 2024 lines of work attacked the token unit itself. The [Byte Latent Transformer](https://aiwiki.ai/wiki/byte_latent_transformer) encodes raw bytes into dynamically sized patches segmented by an entropy criterion, allocating compute according to local complexity instead of a fixed vocabulary [17]. The [Large Concept Model](https://aiwiki.ai/wiki/large_concept_model) moved the prediction step up a level, performing autoregressive prediction of sentence embeddings rather than tokens [18]. Neither has displaced subword tokenization in shipped frontier models.

## From n-grams to GPT

| Year | Development | Contribution to the objective |
| --- | --- | --- |
| 1951 | Shannon, "Prediction and Entropy of Printed English" | Framed next-symbol prediction as entropy estimation; about 1 bit per letter for English [3] |
| 2003 | Bengio, Ducharme, Vincent and Jauvin, "A Neural Probabilistic Language Model" | Learned distributed word representations jointly with the next-word probability function, beating n-gram baselines [4] |
| 2017 | Vaswani et al., "Attention Is All You Need" | Replaced recurrence with [attention](https://aiwiki.ai/wiki/attention), making parallel teacher-forced training practical [9] |
| 2018 | Radford et al., GPT | Generative [pre-training](https://aiwiki.ai/wiki/pre-training) on BooksCorpus (over 7,000 unpublished books), then task [fine-tuning](https://aiwiki.ai/wiki/fine_tuning) [1] |
| 2019 | Radford et al., GPT-2 | 1.5B parameters on WebText; state of the art on 7 of 8 language modeling datasets zero-shot, while still underfitting its own training set [2] |
| 2020 | Brown et al., GPT-3 | 175B parameters; task performance without gradient updates, establishing [in-context learning](https://aiwiki.ai/wiki/in_context_learning) [5] |

[Yoshua Bengio](https://aiwiki.ai/wiki/yoshua_bengio) and colleagues framed the 2003 model as a way to fight the curse of dimensionality: a sequence never seen in training can still receive high probability if it is built from words with nearby representations [4]. That argument is the direct ancestor of the embedding-based prediction used in every transformer language model today.

## Why the objective scales

The empirical case for scaling next-token prediction rests on the observation that its loss falls predictably. The 2020 [scaling laws](https://aiwiki.ai/wiki/scaling_laws) paper reported that cross-entropy loss follows a power law in model size, dataset size, and training compute, and concluded that larger models were markedly more sample-efficient, favoring very large models trained on relatively modest data [19]. The 2022 [Chinchilla](https://aiwiki.ai/wiki/chinchilla_scaling) analysis revised the prescription: for a fixed compute budget, parameters and training tokens should be scaled equally, so that every doubling of model size is matched by a doubling of training tokens. Chinchilla itself has 70B parameters and used the same compute budget as Gopher with four times the data; it outperformed Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) across a wide range of downstream tasks [20].

Both results describe the same underlying property. Predictive loss on natural text keeps improving with scale rather than saturating, and improvements in loss track improvements on downstream tasks closely enough that pretraining loss functions as a proxy for capability. GPT-2's authors noted in 2019 that their largest model still underfit WebText, which is roughly the empirical statement that the objective had not yet been exhausted [2].

## Is next-token prediction enough?

[Ilya Sutskever](https://aiwiki.ai/wiki/ilya_sutskever) gave the strongest version of the affirmative case in a March 2023 interview. His claim was that text is a projection of the thoughts and processes that produced it, so predicting it accurately requires modeling those processes: "Predicting the next token well means that you understand the underlying reality that led to the creation of that token. It's not statistics. Like it is statistics but what is statistics? In order to understand those statistics to compress them, you need to understand what is it about the world that creates this set of statistics?" [6]

There is theoretical and empirical support for that reading. A 2023 analysis showed that even linear next-token predictors trained on chain-of-thought data can approximate any function efficiently computable by a Turing machine, attributing much of current model capability to the autoregressive training scheme rather than to any particular architecture [21]. Interpretability work found a transformer trained only to predict legal Othello moves developed an internal representation of the board that could be edited to steer its output [22], and later work found linear representations of spatial and temporal coordinates inside the Llama-2 family [23]. Both are evidence that pure sequence prediction can induce something like a [world model](https://aiwiki.ai/wiki/world_model).

[Yann LeCun](https://aiwiki.ai/wiki/yann_lecun) has made the most prominent negative case. On a March 2024 podcast appearance he argued that every token produced carries some probability of taking the output out of the set of reasonable answers, so that "the probability that you stay within the set of correct answer decreases and it decreases exponentially," and that models trained this way "just produce one word after the other instinctively" without planning the answer. His alternative is a joint-embedding predictive architecture ([JEPA](https://aiwiki.ai/wiki/jepa)) that predicts in a learned representation space rather than over tokens [7].

Bachmann and Nagarajan's contribution cuts across both camps. They accept that the compounding-error argument is often stated too loosely, since it presumes an accurate predictor was learned, and then identify a separate failure that occurs during teacher-forced training itself [8].

Two empirical results show the objective leaving fingerprints on model behavior. The "embers of autoregression" study found that models are more accurate when the correct output is a high-probability string than when it is a low-probability one, even on deterministic tasks where output probability should be irrelevant [24]. The reversal curse describes models trained on statements of the form "A is B" failing to generalize to "B is A," a limitation consistent with a strictly left-to-right factorization [25].

## Masked and diffusion alternatives

[BERT](https://aiwiki.ai/wiki/bert) took the other branch in 2018, pretraining a [masked language model](https://aiwiki.ai/wiki/masked_language_model) that conditions on both left and right context by predicting tokens hidden at random positions [26]. Bidirectional conditioning produced representations that fine-tune well for classification and tagging, but the objective does not define a straightforward left-to-right generation procedure, and it was the autoregressive branch that scaled into general-purpose systems.

[Diffusion language models](https://aiwiki.ai/wiki/diffusion_language_models) returned the non-autoregressive idea to the frontier in 2025 by predicting many tokens in parallel and refining them over several steps [27][28].

| Model | Developer or authors | Reported result |
| --- | --- | --- |
| LLaDA 8B (February 2025) | Nie et al. | Diffusion model trained from scratch; competitive with LLaMA3 8B on in-context learning and reported to alleviate the reversal curse [27] |
| Mercury Coder (report June 2025) | [Inception Labs](https://aiwiki.ai/wiki/inception_labs) | 1,109 tokens/sec (Mini) and 737 tokens/sec (Small) on NVIDIA H100 GPUs in independent Artificial Analysis evaluations, outperforming speed-optimized frontier models by up to 10x on average at comparable quality [28] |
| [Gemini Diffusion](https://aiwiki.ai/wiki/gemini_diffusion) | [Google DeepMind](https://aiwiki.ai/wiki/google_deepmind) | Experimental text diffusion demo; the model page reports 1,479 tokens/sec sampling speed excluding overhead, plus error correction during generation [29] |

A 2025 scaling study found that masked [discrete diffusion](https://aiwiki.ai/wiki/discrete_diffusion) models outperform autoregressive models when compute is plentiful but data is scarce, because randomized masking implicitly trains over many token orderings and therefore extracts more from repeated passes over the same corpus [30]. As of mid-2026 the two families are being combined rather than ranked: NVIDIA's Nemotron-Labs-Diffusion unifies autoregressive, diffusion, and self-speculation decoding in one model and reports that letting diffusion draft while the autoregressive mode verifies outperforms multi-token prediction methods on both acceptance rate and real-device efficiency [33].

## Multi-token prediction

The most widely adopted modification keeps the autoregressive factorization but asks the model to predict several future tokens at each position. Gloeckle and colleagues set out an influential version in April 2024 using `n` independent output heads on a shared trunk. Their 13B models solved 12 percent more HumanEval problems and 17 percent more MBPP problems than comparable next-token models, the gains grew with model size, and 4-token models ran up to three times faster at inference [31].

[DeepSeek-V3](https://aiwiki.ai/wiki/deepseek_v3) adopted [multi-token prediction](https://aiwiki.ai/wiki/multi_token_prediction) in December 2024 with a different implementation: instead of parallel heads, it uses sequential modules that preserve the full causal chain at each prediction depth. Its published configuration sets depth `D` to 1, so each token predicts one additional token beyond the immediate next one, with the auxiliary loss weighted at 0.3 for the first 10 trillion training tokens and 0.1 for the remaining 4.8 trillion. The modules can be discarded at inference or reused as speculative-decoding drafters; DeepSeek reports an 85 to 90 percent acceptance rate for the second predicted token and 1.8 times the tokens per second [15].

Multi-token prediction has since spread to other frontier training recipes. NVIDIA's [Nemotron](https://aiwiki.ai/wiki/nemotron) 3 Ultra, a 550-billion-parameter mixture-of-experts model with 55 billion active parameters described in a June 2026 technical report, lists multi-token prediction among the techniques behind a claimed 6x inference throughput advantage over state of the art publicly available models at comparable accuracy [32].

## After pretraining

Next-token prediction remains the pretraining substrate, but it is no longer the last training stage for frontier systems. [Reasoning models](https://aiwiki.ai/wiki/reasoning_models) apply reinforcement learning on top of a pretrained base: DeepSeek-R1, released in January 2025, showed that reasoning behaviors including self-reflection and verification can be induced through reinforcement learning alone, without human-annotated reasoning traces [34]. This shifts part of the capability budget from predictive loss to [test-time compute](https://aiwiki.ai/wiki/test_time_compute) and reward signals.

Sutskever, whose 2023 framing of prediction as understanding remains the reference statement of the optimistic view, described the situation differently in November 2025. On the ceiling of the pretraining objective he said: "At some point though, pre-training will run out of data. The data is very clearly finite." He added that current models "generalize dramatically worse than people" and that further progress means either a different pretraining recipe, reinforcement learning, or something else again [35]. The 2026 research picture matches that description. The token-level predictive objective is still the base layer of nearly every system, and most active research directions are attempts to add something on top of it.

## See also

- [Language model](https://aiwiki.ai/wiki/language_model)
- [Autoregressive model](https://aiwiki.ai/wiki/autoregressive_model)
- [Perplexity](https://aiwiki.ai/wiki/perplexity)
- [Tokenization](https://aiwiki.ai/wiki/tokenization)
- [Multi-token prediction](https://aiwiki.ai/wiki/multi_token_prediction)
- [Masked language model](https://aiwiki.ai/wiki/masked_language_model)

## References

1. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I. "Improving Language Understanding by Generative Pre-Training." OpenAI, 2018. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
2. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I. "Language Models are Unsupervised Multitask Learners." OpenAI, 2019. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
3. Shannon, C. E. "Prediction and Entropy of Printed English." Bell System Technical Journal, vol. 30, pp. 50-64, 1951. https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf
4. Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C. "A Neural Probabilistic Language Model." Journal of Machine Learning Research, vol. 3, pp. 1137-1155, 2003. https://www.jmlr.org/papers/v3/bengio03a.html
5. Brown, T. B., et al. "Language Models are Few-Shot Learners." arXiv:2005.14165, May 28, 2020. https://arxiv.org/abs/2005.14165
6. Patel, D. "Ilya Sutskever (OpenAI Chief Scientist)." Dwarkesh Podcast, March 27, 2023. https://www.dwarkesh.com/p/ilya-sutskever
7. Fridman, L. "Yann LeCun: Meta AI, Open Source, Limits of LLMs, AGI and the Future of AI." Lex Fridman Podcast #416, March 7, 2024. Transcript: https://lexfridman.com/yann-lecun-3-transcript
8. Bachmann, G., Nagarajan, V. "The pitfalls of next-token prediction." arXiv:2403.06963, March 11, 2024 (v3 July 29, 2025). https://arxiv.org/abs/2403.06963
9. Vaswani, A., et al. "Attention Is All You Need." arXiv:1706.03762, June 12, 2017. https://arxiv.org/abs/1706.03762
10. Jurafsky, D., Martin, J. H. "N-gram Language Models," chapter 3 of Speech and Language Processing, 3rd edition draft, January 6, 2026. https://web.stanford.edu/~jurafsky/slp3/3.pdf
11. Williams, R. J., Zipser, D. "A Learning Algorithm for Continually Running Fully Recurrent Neural Networks." Neural Computation, vol. 1, no. 2, pp. 270-280, 1989. https://doi.org/10.1162/neco.1989.1.2.270
12. Ranzato, M., Chopra, S., Auli, M., Zaremba, W. "Sequence Level Training with Recurrent Neural Networks." arXiv:1511.06732, November 20, 2015. https://arxiv.org/abs/1511.06732
13. Bengio, S., Vinyals, O., Jaitly, N., Shazeer, N. "Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks." arXiv:1506.03099, June 9, 2015. https://arxiv.org/abs/1506.03099
14. Sennrich, R., Haddow, B., Birch, A. "Neural Machine Translation of Rare Words with Subword Units." arXiv:1508.07909, August 31, 2015. https://arxiv.org/abs/1508.07909
15. DeepSeek-AI. "DeepSeek-V3 Technical Report." arXiv:2412.19437, December 27, 2024. https://arxiv.org/abs/2412.19437
16. Singh, A. K., Strouse, D. J. "Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs." arXiv:2402.14903, February 22, 2024. https://arxiv.org/abs/2402.14903
17. Pagnoni, A., et al. "Byte Latent Transformer: Patches Scale Better Than Tokens." arXiv:2412.09871, December 13, 2024. https://arxiv.org/abs/2412.09871
18. LCM team, Barrault, L., Duquenne, P.-A., et al. "Large Concept Models: Language Modeling in a Sentence Representation Space." arXiv:2412.08821, December 11, 2024. https://arxiv.org/abs/2412.08821
19. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., et al. "Scaling Laws for Neural Language Models." arXiv:2001.08361, January 23, 2020. https://arxiv.org/abs/2001.08361
20. Hoffmann, J., Borgeaud, S., Mensch, A., et al. "Training Compute-Optimal Large Language Models." arXiv:2203.15556, March 29, 2022. https://arxiv.org/abs/2203.15556
21. Malach, E. "Auto-Regressive Next-Token Predictors are Universal Learners." arXiv:2309.06979, September 13, 2023. https://arxiv.org/abs/2309.06979
22. Li, K., Hopkins, A. K., Bau, D., Viegas, F., Pfister, H., Wattenberg, M. "Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task." arXiv:2210.13382, October 24, 2022. https://arxiv.org/abs/2210.13382
23. Gurnee, W., Tegmark, M. "Language Models Represent Space and Time." arXiv:2310.02207, October 3, 2023. https://arxiv.org/abs/2310.02207
24. McCoy, R. T., Yao, S., Friedman, D., Hardy, M., Griffiths, T. L. "Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve." arXiv:2309.13638, September 24, 2023. https://arxiv.org/abs/2309.13638
25. Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stickland, A. C., Korbak, T., Evans, O. "The Reversal Curse: LLMs trained on 'A is B' fail to learn 'B is A'." arXiv:2309.12288, September 21, 2023. https://arxiv.org/abs/2309.12288
26. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." arXiv:1810.04805, October 11, 2018. https://arxiv.org/abs/1810.04805
27. Nie, S., Zhu, F., You, Z., et al. "Large Language Diffusion Models." arXiv:2502.09992, February 14, 2025. https://arxiv.org/abs/2502.09992
28. Inception Labs, et al. "Mercury: Ultra-Fast Language Models Based on Diffusion." arXiv:2506.17298, June 17, 2025. https://arxiv.org/abs/2506.17298
29. Google DeepMind. "Gemini Diffusion." Model page. https://deepmind.google/models/gemini-diffusion/
30. Prabhudesai, M., Wu, M., Zadeh, A., Fragkiadaki, K., Pathak, D. "Diffusion Beats Autoregressive in Data-Constrained Settings." arXiv:2507.15857, July 21, 2025. https://arxiv.org/abs/2507.15857
31. Gloeckle, F., Youbi Idrissi, B., Roziere, B., Lopez-Paz, D., Synnaeve, G. "Better & Faster Large Language Models via Multi-token Prediction." arXiv:2404.19737, April 30, 2024. https://arxiv.org/abs/2404.19737
32. NVIDIA. "Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning." arXiv:2606.15007, June 12, 2026. https://arxiv.org/abs/2606.15007
33. Fu, Y., Whalen, L., Garg, A., et al. "Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding." arXiv:2607.05722, July 7, 2026. https://arxiv.org/abs/2607.05722
34. DeepSeek-AI. "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." arXiv:2501.12948, January 22, 2025. https://arxiv.org/abs/2501.12948
35. Patel, D. "Ilya Sutskever." Dwarkesh Podcast, November 25, 2025. https://www.dwarkesh.com/p/ilya-sutskever-2