Multi-token prediction

Multi-token prediction (often abbreviated MTP) is a language modeling training objective in which the model is trained to predict several future tokens at each context position rather than only the next token. The standard autoregressive objective, next-token prediction, asks the model to maximize the likelihood of the immediately following token given the prefix. MTP generalizes this by attaching n parallel prediction heads on top of a shared transformer trunk, one for each of the next n tokens, and adding the auxiliary cross-entropy losses to the main objective during training ^[1]. The technique was popularized as a deliberate training objective by a Meta AI paper in April 2024 ("Better and Faster Large Language Models via Multi-token Prediction" by Fabian Gloeckle and colleagues), and was adopted at scale in DeepSeek-V3 in December 2024, where the same MTP heads doubled as drafters for speculative decoding at inference time ^[1][2].

MTP is attractive for two distinct reasons. First, it provides denser supervision per training step, because each hidden state has to encode information useful for predicting several future positions rather than just the next one. This appears to improve sample efficiency and downstream performance on generative tasks, with the gains growing larger at bigger model scales ^[1]. Second, the auxiliary heads can be reused at inference time as a self-contained drafter for speculative decoding, delivering roughly 1.8 to 3 times faster generation without quality loss ^[1][2][3]. By 2026 MTP layers ship inside open models including DeepSeek-V3, DeepSeek-R1, GLM-4.5, MiMo-7B, Qwen3-Next, Nemotron 3 Super, and Ling-3.0-flash; Google released separate MTP drafter checkpoints for Gemma 4; and inference engines such as vLLM, SGLang, TensorRT-LLM, and llama.cpp can run a model's MTP layers as its built-in draft model ^{[3][4][6][14][17][18][19][20][21]}.

Background and motivation

For most of the modern history of large language models, the workhorse training objective has been maximum likelihood under an autoregressive factorization. The model receives a sequence of tokens, transforms it through stacked self-attention and feedforward blocks, and produces a probability distribution over the vocabulary at each position. The loss compares that distribution to the actual next token. Stacking this across billions of positions gives the familiar next-token cross-entropy objective.

This recipe has known weaknesses. Each token contributes only one bit of supervision per step, which makes pretraining data-hungry. The signal is also myopic: the gradient only ever encourages the model to be right about the immediately following token, so any longer-range planning, such as choosing a function name early in a code completion so that calls many tokens later remain coherent, has to emerge implicitly. Researchers have long argued that this is one reason why language models can be locally fluent but globally incoherent over many tokens.

In parallel, the inference side of language modeling has its own bottleneck. Autoregressive decoding is inherently sequential: each new token requires a full forward pass through the network, and there is no parallelism over the generated tokens themselves. On modern accelerators this is wasteful, because the matrix multiplications in a single forward pass barely saturate the GPU, and most of the time is spent waiting on memory bandwidth. Various forms of parallel decoding have been proposed to break this bottleneck, with speculative decoding being the dominant family.

MTP sits at the intersection of these two threads. By predicting several tokens during training, it provides extra supervision that the model can use to learn richer representations. By exposing the auxiliary heads at inference, it provides a built-in drafter for speculative decoding. The same architectural addition that helps training also helps inference, which is unusual in the literature on language model efficiency.

Origins

The idea of predicting multiple tokens at once is not new. Stern, Shazeer, and Uszkoreit introduced blockwise parallel decoding at NeurIPS 2018 as an inference speedup for sequence-to-sequence models ^[5]. Their scheme makes predictions for several time steps in parallel and then backs off to the longest prefix that a scoring model validates, so the accepted prefix advances the output and any rejected tail is recomputed. On machine translation and image super-resolution they reported iteration reductions of up to 2 times over greedy decoding with no loss in quality (up to 7 times with a slight quality decrease) and wall-clock speedups of up to 4 times ^[5]. This is essentially the same recipe used by modern MTP drafters, although the 2018 paper did not frame the auxiliary heads as a tool for improving the base model itself.

For several years after, the parallel decoding line of work stayed focused on inference. Speculative decoding in its now-canonical form, with a separate draft model verifying against a larger target, was introduced by Leviathan et al. and Chen et al. in 2022 and 2023 ^[12][13], and quickly became standard in production serving stacks. The training objective itself, however, remained almost universally next-token prediction.

The modern MTP literature changed this by treating multi-token prediction as a training objective rather than an inference trick. Gloeckle et al.'s Meta paper in April 2024 was the watershed moment: it demonstrated that adding several auxiliary heads at training time, with no extra training-time cost, both improved downstream performance and yielded a usable drafter for free at inference ^[1]. DeepSeek-V3 then operationalized the idea in a flagship open model later in 2024 ^[2], and a string of follow-up papers in 2025 and 2026 refined the technique with self-distillation targets, register tokens, leap prediction, and shared drafter heads.

The Meta multi-token prediction paper

Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Roziere, David Lopez-Paz, and Gabriel Synnaeve published "Better and Faster Large Language Models via Multi-token Prediction" at ICML 2024 as a Meta FAIR contribution ^[1]. The paper trained transformer LMs of up to 13 billion parameters on hundreds of billions of tokens of code and natural text, with each model duplicated under two training objectives: standard next-token prediction and 4-token prediction.

Architecture

The Meta architecture keeps a single shared transformer trunk and adds n independent output heads on top of it. Each head is itself a transformer layer followed by an unembedding, and head i is responsible for predicting the token at position t plus i given the prefix up to position t. The heads share the unembedding matrix of the main model, so the auxiliary cost is the per-head transformer layer plus n softmaxes ^[1].

Loss formulation

Given a shared trunk that produces a hidden state z_t at position t, the multi-token prediction loss factorizes the joint probability of the next n tokens through n independent head distributions:

L_n = - sum over t of sum over i = 1 to n of log P_theta(x_{t+i} | z_t)

In this factorization the n head distributions are conditionally independent given z_t. Because the latent z_t has to support all n predictions simultaneously, the trunk is pushed toward representations that encode information about several future tokens, not just the next one. The total training loss is the sum of the per-head cross-entropy losses, often averaged and scaled by a constant.

Memory-efficient implementation

A naive implementation would materialize n logit tensors of shape (batch, sequence, vocabulary), which at modern vocabulary sizes is the dominant memory cost of training. Gloeckle et al. propose a sequential forward and backward over the heads: for each head, the trunk hidden state is fed in, the logits and loss are computed, the backward pass is run, the gradients are accumulated at the trunk, and the logits and their gradient buffers are released before moving on to the next head ^[1]. This reduces the peak memory requirement from O(nV + d) to O(V + d), where V is the vocabulary size and d is the hidden dimension, and brings the per-step memory of MTP training back to roughly the same as next-token training despite the n heads.

Key results

Gloeckle et al. report several findings. On code generation benchmarks, 13 billion parameter models trained with 4-token prediction solved about 12 percent more HumanEval problems and 17 percent more MBPP problems than next-token baselines trained on the same data ^[1]. On eight abstractive summarization benchmarks (including CNN/DailyMail and Multi-News), 7B natural-language models trained with 2-token and 4-token prediction both improved ROUGE-L over the next-token baseline after finetuning, with the gap shrinking as pretraining data grew from 200B to 500B tokens ^[1]. On byte-level models, where each byte carries less information than a BPE token, 8-byte prediction was the best setting: a 7B byte-level model solved 67 percent more MBPP pass@1 problems and 20 percent more HumanEval pass@1 problems than next-byte prediction, and decoded 6.4 times faster with self-speculative decoding ^[1]. The gains grew with model size: across six model sizes from 300M to 13B parameters trained on code, MTP models were worse than the baseline at small sizes and better at scale ^[1].

At inference, the best 7B 4-token model reached a 3.0 times speedup on code (accepting on average 2.5 of 3 suggested tokens) and 2.7 times on text when its auxiliary heads were used for greedy self-speculative decoding, with the main head still acting as the verifier ^[1]. On small synthetic tasks, 2-token prediction vastly improved the formation of induction capability in models of 30M non-embedding parameters and below (the advantage disappeared at 100M and above), and MTP improved generalization on a polynomial arithmetic task ^[1].

DeepSeek-V3 multi-token prediction

DeepSeek-V3 was the first widely deployed flagship model to ship with multi-token prediction baked in at scale. The DeepSeek-V3 technical report describes an MTP design that is similar in spirit to Gloeckle et al. but differs in several important details ^[2].

Architecture

DeepSeek-V3 uses D sequential MTP modules to predict the next D tokens, where each MTP module is a full transformer block of the same flavor as the main trunk (with Multi-head Latent Attention and a Mixture-of-Experts feedforward) ^[2]. The base trunk and the D MTP modules share the input embedding and the output head ^[2]. DeepSeek-V3 was trained with D = 1, meaning that besides the exact next token each position predicts one additional token ^[2]. The Hugging Face release totals 685B parameters: 671B of main-model weights and 14B of MTP module weights ^[26].

The critical architectural difference from Gloeckle et al. is that DeepSeek-V3 keeps the predictions causally chained. For input position i, the k-th MTP module takes the representation of token i from the previous depth (the main model's output when k = 1) together with the embedding of token i+k, applies RMSNorm to each, concatenates them, and projects back to the hidden dimension with a learned matrix M_k. The output of this MTP transformer block is fed to the shared output head to predict token i+k+1 ^[2]. Unlike Gloeckle et al., who predict the additional tokens in parallel with independent heads, DeepSeek-V3 predicts them sequentially and "keep[s] the complete causal chain at each prediction depth", a principle the report compares to the EAGLE drafter ^[2].

Loss formulation

The total MTP loss is the average of the cross-entropy losses across the D modules, scaled by a weighting factor lambda:

L_MTP = (lambda / D) * sum over k = 1 to D of L^k_MTP

The total training loss combines the main next-token loss with L_MTP. In the DeepSeek-V3 recipe, lambda is 0.3 for the first 10 trillion training tokens and 0.1 for the remaining 4.8 trillion ^[2]. With D = 1, MTP adds one transformer block per forward pass. Under DeepSeek's DualPipe pipeline schedule, the shallowest layers (including the embedding) and the deepest layers (including the output head) sit on the same pipeline rank, so the MTP module and the main model physically share those parameters and their gradients ^[2].

Training results

DeepSeek-V3's ablations appended a 1-depth MTP module to two baseline MoE models (15.7B total parameters trained on 1.33T tokens, and 228.7B total parameters trained on 540B tokens) with the training data otherwise unchanged; the MTP versions performed better on most evaluation benchmarks. The MTP module was discarded at inference, so the compared models had identical inference cost ^[2]. In DeepSeek's evaluation the acceptance rate of the second-token prediction ranged between 85 and 90 percent across generation topics, which is what enables the inference speedup described below ^[2].

Inference use

DeepSeek-V3 supports two inference modes. In the discard-MTP mode, the MTP module is dropped and the model behaves as a standard next-token autoregressive transformer. In the MTP speculative mode, the MTP module drafts one extra candidate token per step, and the main head verifies it; if the verified token agrees with the draft, the model effectively advances two tokens per forward pass. At an 85 to 90 percent acceptance rate, this delivers approximately a 1.8 times speedup in tokens per second ^[2]. In September 2025 AMD's ROCm team served DeepSeek-V3 with SGLang on eight AMD Instinct MI300X GPUs and reported end-to-end latency speedups of 1.25 to 2.11 times on a random dataset and 1.36 to 1.80 times on ShareGPT from enabling MTP; the speedup fell as maximum concurrency rose ^[3].

How DeepSeek-V3 differs from the Meta paper

The two designs aim at the same goal but make different trade-offs. The table below summarizes the contrasts.

Inference acceleration and speculative decoding

MTP heads are a natural drafter for speculative decoding, the most widely used inference-time acceleration for autoregressive models. In standard speculative decoding, a small draft model produces several candidate tokens, and the larger target model verifies them in a single parallel forward pass. Any prefix of the draft that matches what the target would have sampled is accepted; the first disagreement triggers a fresh sample from the target's distribution and the rest of the draft is discarded.

The ceiling on speedup is set by the average accepted run length. With perfect drafting, k drafted tokens are accepted per call and the user sees up to a (k+1)-fold throughput improvement, modulo verification overhead. In practice, draft acceptance is well below perfect, and the speedup is closer to 1.5 to 3 times for modern systems.

MTP makes drafting cheap and naturally aligned. Because the drafter heads are trained jointly with the target model on the same data, they predict from the same distribution that the target was trained to match, which gives high acceptance rates. Because they share the embedding and unembedding with the main model, the drafter is small, though not free: DeepSeek-V3's single MTP module adds 11.5 billion unique parameters (excluding the shared embedding and output head) to the 671 billion of the main model ^[28]. And because they sit on top of the same hidden state as the main head, verification is a forward pass of the existing trunk over the drafted positions, with the small MTP block as the only additional drafting compute; for mixture-of-experts models that verification pass can still cost noticeably more than a normal decode step (see below) ^[14]. The result is a self-contained drafter that does not require a separate small model and avoids the calibration mismatch that often hurts external drafters.

Reported speedups across recent MTP-equipped systems include:

When vLLM announced Qwen3-Next support in September 2025, it said the engine natively supports the model's MTP mode, letting it decode multiple tokens per step without application changes ^[22]. SGLang runs MTP drafting on both AMD ROCm and NVIDIA CUDA GPUs ^[3][16]. The llama.cpp project merged MTP support on May 16, 2026 (pull request 22673). Its author tested it on Qwen3.6 27B and 35B-A3B and reported a steady-state acceptance of around 75 percent with 3 draft tokens, which he described as more than a 2 times speedup over baseline; prompt processing typically slows when MTP is enabled, mainly because of device-to-host embedding transfers ^[4]. The next section covers how engines expose MTP drafting and what limits its payoff.

Deploying MTP layers as speculative drafters

A serving engine that supports MTP treats the model's own MTP layers as the draft model in a speculative decoding loop. The MTP layer proposes a few tokens in sequence, the full model scores all proposed positions in one forward pass, and the correct prefix is accepted ^[3][14]. Because the MTP weights are trained jointly with the base model, AMD's ROCm team notes, their predictive distribution is consistent with the base model's, and no separate drafter has to be trained ^[3].

NextN naming and engine settings

Inference code often calls the MTP layer "NextN". AMD describes the single MTP module that DeepSeek-V3 and R1 use at inference as NextN, and its DeepSeek-V3 recipe loaded the NextN weights as a separate draft model path ^[3]. Model configuration files record the number of these layers in a num_nextn_predict_layers field: Ling-3.0-flash sets it to 1, and NVIDIA notes that this layer "can be used directly as a draft model without additional training or downloads" ^[14]. DeepSeek-V3's configuration file sets the same field to 1 ^[27].

Published recipes use short drafts. AMD's DeepSeek-V3 benchmark used 2 draft steps, top-k 1, and 3 draft tokens ^[3]. The Qwen3-Next-80B-A3B-Instruct model card recommends 3 steps, top-k 1, and 4 draft tokens in SGLang, and {"method":"qwen3_next_mtp","num_speculative_tokens":2} in vLLM ^[18]. NVIDIA's Ling-3.0-flash example used 3 steps, top-k 1, and 4 draft tokens ^[14].

Acceptance length and verification cost

NVIDIA's DGX Spark write-up names two quantities that decide the payoff: the accept length (average tokens accepted per round) and the verification-cost ratio (the cost of one verification step relative to a normal decode step) ^[14]. Huang et al.'s MoESD paper (NeurIPS 2025) writes speculative-decoding speedup in the same terms: the average number of accepted tokens per round, divided by the sum of drafting time, verification time, and rejection-sampling time, each expressed relative to one ordinary target forward pass ^[15]. For a dense model, generating one token and verifying several cost roughly the same at small batch sizes, because both load the full set of parameters once ^[15].

Acceptance falls with draft depth. In Xiaomi's evaluation on AIME24, MiMo-7B's first MTP layer was accepted about 90 percent of the time and its third above 75 percent ^[20]. NVIDIA reports that acceptance decreases monotonically with draft position for Nemotron 3 Super, Qwen3-Next, and DeepSeek-R1. It attributes part of the decline to a training-inference mismatch: a head trained on ground-truth hidden states conditions on its own generated states when reused autoregressively. Nemotron 3 Super shares parameters across its MTP heads during training to reduce that mismatch ^[21]. Workload and load also matter. In AMD's DeepSeek-V3 tests the speedup fell from 2.11 times at a maximum concurrency of 1 to 1.25 times at 64 ^[3], and NVIDIA found that a synthetic, greedy-decoded workload overstated the benefit compared with sampled GSM8K answers (see the example below) ^[14].

Why MoE models pay more for verification

In a mixture-of-experts model, the draft tokens in a verification pass can route to different experts, so verification activates more experts than a single-token decode step. MoESD describes this as larger memory access and a significantly longer verification step, which is why speculative decoding had been considered ineffective for MoE models at small batch sizes ^[15]. The same paper argues that at moderate batch sizes, where nearly all experts are already active, extra draft tokens add little memory traffic, so MoE models can gain more from speculative decoding than dense ones, and the effective batch-size range widens as MoE models become sparser. It reports up to 2.29 times speedup for Qwen2-57B-A14B at medium batch sizes ^[15]. Google made a similar observation for Gemma 4's MTP drafters: the 26B mixture-of-experts model "presents unique routing challenges" at batch size 1 on Apple Silicon, while batch sizes of 4 to 8 reached up to about 2.2 times speedup locally ^[6].

Example: Ling-3.0-flash on NVIDIA DGX Spark

An October 27, 2026 NVIDIA technical blog post by Yang Liu worked through three optimizations for batch-size-1 inference of inclusionAI's Ling-3.0-flash on DGX Spark with SGLang ^[14]. The model checkpoint used (Ling-3.0-flash-fp4) has 124 billion total and 5.1 billion active parameters, 42 layers (35 Kimi Delta Attention linear-attention layers and 7 multi-head latent attention layers), and routes each token to 8 of 512 routed experts plus one shared expert. Profiling showed decoding was memory-bound, dominated by the MoE computation and the LM-head projection from a 2,560-dimensional hidden state to a 157,184-token vocabulary ^[14].

The three steps were the Humming MXFP4 MoE kernel backend in place of FlashInfer, online FP8 quantization of the BF16 LM head, and MTP speculative decoding with NEXTN using the model's single NextN layer, 3 draft steps, and a 4-token verification window. NVIDIA measured a verification-cost ratio of about 2.26. It attributed the cost mainly to MoE expert activation rather than attention: at batch size 1, KV-cache reads were small relative to weight reads, but a normal decode step activates 8 experts for one token, while verifying four tokens can activate up to 32 ^[14].

Source: NVIDIA ^[14]. The accept length in the synthetic run was 3.90 tokens per round.

On all 1,319 GSM8K test questions (five-shot, temperature 0.6, top_p 0.95, top_k 29, chain of thought enabled), the combined configuration raised decode throughput from 37.81 to 53.89 tokens per second (+42.5 percent), cut mean end-to-end latency by 24.5 percent, and moved accuracy from 96.89 to 96.82 percent, which NVIDIA describes as no statistically detectable change. The MTP accept length fell to 2.68. NVIDIA gives three reasons the synthetic test overstated the gain: synthetic outputs are more repetitive and easier to draft, the synthetic run used greedy decoding, and its fixed 2K-token outputs differ from GSM8K's median of about 310 tokens. It cautions that the 185.1 percent synthetic gain "does not establish the gain for other workloads" ^[14].

The post also documents an integration failure. When the FP8 LM head and MTP were enabled together, the draft layer had to share the target's whole LM-head module rather than only its weight tensor, and the original Ling NEXTN implementation did not expose that interface. Without it the framework silently fell back: the draft received FP8 weights but interpreted them as BF16, accept length dropped to 1.00, and throughput fell below that of the unoptimized baseline. Sharing the full module restored normal acceptance, and NVIDIA recommends confirming at startup that every optimization is active rather than inferring it from later throughput numbers ^[14]. vLLM's documentation describes a related option: because most MTP heads share the target's unquantized LM head, "draft_lm_head_quantization": "fp8" or "nvfp4" gives the drafter a quantized copy, while the target still verifies with its own head, so outputs are unchanged and only acceptance length can move ^[17].

Self-distillation and other extensions

The MTP heads need a training target. In the simplest setting, the target is the actual next-token sequence in the training data, treated as a hard label. Several extensions instead use the main model's predictive distribution as a soft target, in a form of self-distillation.

MTP via self-distillation

Kirchenbauer and colleagues (February 2026) convert a pretrained next-token model into a standalone multi-token predictor with an online distillation objective. A student model predicts a span of tokens simultaneously, and a frozen copy of the original model scores those predictions as a teacher, instead of comparing them with ground-truth text ^[8]. The resulting model keeps the same implementation as the original checkpoint and needs no auxiliary verifier or specialized inference code; the authors report decoding more than 3 times faster at under a 5 percent accuracy drop on GSM8K relative to single-token decoding of the same checkpoint ^[8].

Xu and Che (April 2026) take a post-hoc approach for models that were not trained with MTP: they freeze the target LLM and train an MTP module for it by self-distillation from the target's own next-token predictions ^[11].

MTP-D (Zhao et al., March 2026) distills from the main head to the MTP heads during pretraining, using gradient-detached, top-N-selected logits. The authors report a 7.5 percent increase in MTP head acceptance rates while largely preserving main-head performance ^[23].

MTP needs registers

Gerontopoulos et al. (May 2025) proposed MuToR, which interleaves learnable register tokens into the input sequence, each tasked with predicting a future target ^[9]. The method adds a negligible number of parameters, requires no architectural changes (so it works with off-the-shelf pretrained models), stays aligned with the next-token pretraining objective, and supports scalable prediction horizons. The authors note that MTP's benefits had not consistently carried over to settings such as fine-tuning, and they report results for supervised fine-tuning, parameter-efficient fine-tuning, and pretraining on generative tasks in both language and vision ^[9].

Leap multi-token prediction

L-MTP (Liu et al., NeurIPS 2025) drops the assumption that MTP must predict adjacent future tokens ^[10]. Instead, the MTP heads predict tokens at strategically chosen non-adjacent positions, skipping over intermediate tokens to extend the effective prediction horizon at the same compute cost. The decoder is then modified to consume these non-sequential predictions for accelerated generation. Reported gains include both improved language model quality and faster decoding compared to vanilla MTP.

FastMTP

FastMTP (Cai et al., September 2025) fine-tunes a single MTP head with position-shared weights on self-distilled data, so the same head can be applied recursively across several draft steps while keeping acceptance high ^[7]. It also adds language-aware dynamic vocabulary compression to the MTP head to reduce drafting cost. On seven diverse benchmarks the system delivers an average 2.03 times speedup over next-token decoding, outperforming the vanilla DeepSeek-V3-style MTP drafter by 82 percent.

Future-summary pretraining

Mahajan et al. ("Beyond Multi-Token Prediction", October 2025, ICLR 2026) argue that MTP mostly captures short-range dependencies and propose future summary prediction (FSP): an auxiliary head predicts a compact representation of the long-term future rather than individual tokens ^[24]. They test handcrafted summaries (for example a bag of words of the future sequence) and learned summaries from a reverse language model trained right to left, and report improvements over both next-token and multi-token prediction in 3B and 8B pretraining runs on math, reasoning, and coding benchmarks ^[24].

MTP-D and looped extensions

The MTP-D paper also introduces a looped extension strategy for adding MTP heads economically, and reports a further inference speedup of 220.4 percent relative to 1-head MTP ^[23].

Adoption and deployment

As of 2026, MTP has moved out of research papers into production systems.

An engine's MTP mode only works for model families with MTP layers; for other models vLLM's documentation points users to EAGLE or draft-model speculation instead ^[17]. Google's Gemma 4 drafters and post-hoc methods such as Xu and Che's add MTP-style drafting to models after training ^[6][11].

Why MTP helps training

The Meta paper offered several hypotheses for why predicting more tokens improves the base model itself, not just inference ^[1]:

• Denser supervision per token. Each hidden state z_t is used n times in the loss rather than once. The signal-per-FLOP ratio at training time is higher, which is consistent with the observed improvement in sample efficiency on code benchmarks.
• Forces lookahead. Because the trunk has to encode information about future tokens, it is pushed to develop representations with longer effective horizons. In small models, 2-token prediction vastly improved the formation of induction capability, and MTP improved generalization on a polynomial arithmetic task ^[1].
• Lookahead reinforces choice points. The authors argue that n-token prediction gives a hard-to-predict, consequential transition (a "choice point") a loss weight of n(n+1)/2 through its correlated future predictions, against a weight of n for inconsequential transitions, so the objective implicitly emphasizes the decisions that matter for the rest of a sequence ^[1].
• Better behavior on generative tasks. The gains are largest on benchmarks that require longer output generation (HumanEval, MBPP, summarization). On short discriminative benchmarks, gains are smaller. This is consistent with MTP teaching the model to generate longer coherent sequences rather than to score isolated tokens better.

Not all of these effects are equally well established. The paper labels its explanations as speculation, and later work argues that MTP mostly captures short-range dependencies and offers limited improvement on long-horizon tasks ^[24]. The best value of n also varied with tokenization and data: 8 bytes was consistently best for byte-level models, while for BPE-token code models the best window differed by benchmark ^[1].

When MTP does not help

MTP is not a universal improvement. The Meta paper itself shows several settings where it neither helps nor hurts:

• Small models do worse. In the 300M to 13B parameter sweep on code, MTP models underperformed next-token baselines at the small end and outperformed them at scale ^[1].
• Multiple-choice benchmarks do not improve. On six standard NLP benchmarks, 7B models trained on 200B tokens of natural language matched the baseline with 2-token prediction and regressed slightly with 4-token prediction ^[1].
• The optimal n depends on tokenizer and data. In 7B code ablations, n = 4 was best on HumanEval and MBPP, n = 6 led on APPS/Intro, and 8-byte prediction was consistently best for byte-level models ^[1].

Production models mostly train a single MTP layer. DeepSeek-V3 uses D = 1 ^[2], GLM-4.5 adds one MTP layer ^[19], and Xiaomi trained MiMo-7B with a single MTP layer because its preliminary studies showed that more layers gave no further pretraining improvement, then fine-tuned two extra layers with the main model frozen to draft further ahead at inference ^[20]. Nemotron 3 Super trains two MTP layers with shared parameters ^[21].

Implementation considerations

Implementing MTP in a training stack involves a few non-obvious choices.

Sharing the unembedding

The unembedding matrix (also called the output projection or LM head) is one of the largest single tensors in a transformer. DeepSeek-V3, for example, has a 128K-token vocabulary and a hidden dimension of 7,168 ^[2], so its output head holds roughly 0.9 billion parameters. Sharing it across the main head and the MTP heads keeps memory and parameter count manageable, and both the Meta paper and DeepSeek-V3 share it ^[1][2]. Sharing also has a deployment consequence: an optimization applied to the LM head, such as FP8 quantization, has to reach the draft path as well, as NVIDIA's DGX Spark example shows ^[14].

Sequential forward and backward over heads

The key memory optimization is to compute the loss and gradient for each head sequentially rather than materializing all n logit tensors at once. This brings the peak memory back to roughly the same as next-token training ^[1].

Lambda weighting

The weighting factor lambda controls how much the MTP objective influences the trunk. Too small and the gains vanish; too large and the main head suffers. DeepSeek-V3 used 0.3 and then 0.1 ^[2]; Nemotron 3 Super used an MTP loss scaling factor of 0.3 in pretraining and again during supervised fine-tuning ^[21].

Causal chain versus parallel heads

The Meta paper uses parallel heads conditionally independent given z_t. DeepSeek-V3 chains the predictions causally, with each MTP module conditioning on the previous module's output and the next token's embedding. Even at D = 1 the designs differ, because the DeepSeek-style module receives the embedding of the next token in addition to the trunk's hidden state ^[2].

Drafter compatibility at inference

For MTP heads to be useful at inference, the serving stack must support speculative decoding with the model's own MTP modules as the drafter. Implementations exist in vLLM, SGLang, TensorRT-LLM, and llama.cpp as of 2026 ^{[4][16][17][21]}. The engineering work includes KV-cache management for the draft layer (llama.cpp gives the MTP model its own context and KV cache ^[4]) and keeping the draft path consistent with other optimizations. NVIDIA's DGX Spark example shows that a mismatch can fail silently, leaving MTP enabled with an accept length of 1.00 ^[14].

Relationship to other techniques

MTP overlaps with several other ideas in language model training and inference. The table summarizes how it compares.

Medusa and EAGLE are particularly close in spirit to MTP at inference. Medusa adds extra heads to a frozen base model and trains them on top, while MTP heads are trained jointly with the base. EAGLE uses a small separate drafter that reads the base model's hidden states. DeepSeek's report says its causal-chain MTP design follows a principle similar to EAGLE's, but EAGLE's primary objective is speculative decoding, whereas DeepSeek uses MTP to improve training ^[2].

Open questions and current research

As of 2026, several questions about MTP remain open:

• Optimal n. The Meta paper found n = 4 was best for BPE and n = 8 for byte models, but newer work suggests that with self-distillation or register tokens the curve shifts. There is no consensus default for production training.
• Training cost trade-off. Large auxiliary head counts pay off in inference speed but make pretraining slightly more expensive. The frontier appears to be small n at training (D = 1 to 4) with shared-weight inference-time expansion via methods like FastMTP.
• Quality versus speed trade-off. MTP heads improve the base model at the same step count, but if compute is held fixed, the increase in per-step cost has to be paid back via faster convergence. DeepSeek-V3's ablations, which held the training data fixed, found MTP beneficial at both scales tested ^[2], while the Meta paper found it harmful for small models ^[1].
• Interaction with reasoning training. Models trained for long chain-of-thought reasoning often produce many tokens per query, so MTP-driven speedups are particularly valuable. Whether MTP also improves reasoning quality is less clear, although DeepSeek-R1 inherits MTP from V3 and benefits from the same drafter at inference.
• Generalization to non-text modalities. MTP for diffusion models, discrete diffusion language models, and multimodal inputs is still early-stage. Some 2025 papers show that parallel-decoding style techniques transfer naturally to diffusion LLMs, suggesting that future work may unify these objectives.
• Long-context behavior. Whether MTP helps or hurts long-context understanding is an active area. The auxiliary heads can in principle force the trunk to track longer-range dependencies, but they also burn capacity that might otherwise serve long-context attention.
• Compatibility with reinforcement learning. Post-training with RLHF or GRPO-style methods on MTP-trained base models has been done in practice (DeepSeek-R1, Nemotron 3 Super). NVIDIA added an "MTP healing" stage after Nemotron 3 Super's reinforcement learning, training only the MTP heads on generated responses with the rest of the model frozen, and reports that it significantly improved MTP accuracy ^[21]. How MTP interacts with policy-gradient updates is otherwise not well documented.

Continue reading

• DeepSeek V3DeepSeek-V3 is a 671-billion-parameter open-weights Mixture of Experts large language model from Chinese AI lab DeepSeek, released on December 26, 2024, that activates only 37 billion parameters per token and…Mentioned 34 times above
• SSpeculative DecodingSpeculative decoding is a lossless inference acceleration technique for autoregressive transformer models in which a small, fast draft model proposes several future tokens at once and the larger target model…Mentioned 21 times above
• LLanguage ModelA language model is a model of patterns in language that assigns probabilities or comparable scores to linguistic sequences. Its units may be characters, words, subwords, bytes, or other tokens.Mentioned 12 times above
• SGLangSGLang (short for Structured Generation Language) is a high-performance, open-source serving framework for large language models and multimodal models, originally developed at UC Berkeley's Sky Computing Lab…Mentioned 11 times above
• vLLMvLLM is an open-source inference and serving engine for large language models. It is software for running already trained models, either through a Python interface for offline work or through a network server…Mentioned 9 times above
• DeepSeek-R1DeepSeek-R1 is an open-weight reasoning model and large language model family developed by the Chinese artificial intelligence laboratory DeepSeek. The original model was released on January 20, 2025.Mentioned 6 times above

More from See also

• Next-token prediction
• Transformers
• Knowledge Distillation
• Large Language Model
• Mixture of Experts (MoE)
• Pre-training
• NVIDIA DGX Spark

Topics:AI InferenceLarge Language ModelsMachine Learning

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26Gloeckle, F., Youbi Idrissi, B., Roziere, B., Lopez-Paz, D., and Synnaeve, G. (2024). "Better and Faster Large Language Models via Multi-token Prediction." Proceedings of ICML 2024, PMLR 235:15706-15734. arxiv.org/...2404.19737
2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22DeepSeek-AI (2024). "DeepSeek-V3 Technical Report." arxiv.org/...2412.19437
3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12AMD ROCm Blogs (2025). "Efficient LLM Serving with MTP: DeepSeek V3 and SGLang on AMD Instinct GPUs." rocm.blogs.amd.com/...README
4. ^1 ^2 ^3 ^4 ^5 ^6 ^7am17an (2026). "llama + spec: MTP Support." ggml-org/llama.cpp pull request 22673, merged May 16, 2026. github.com/...22673
5. ^1 ^2 ^3Stern, M., Shazeer, N., and Uszkoreit, J. (2018). "Blockwise Parallel Decoding for Deep Autoregressive Models." Proceedings of NeurIPS 2018. proceedings.neurips.cc/...64dc0f5bf2c93bc-Abstract
6. ^1 ^2 ^3 ^4 ^5Google AI (2026). "Accelerating Gemma 4: Faster Inference with Multi-Token Prediction Drafters." blog.google/...multi-token-prediction-gemma-4
7. ^1 ^2Cai, Y. et al. (2025). "FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction." arXiv:2509.18362. arxiv.org/...2509.18362
8. ^1 ^2Kirchenbauer, J. et al. (2026). "Multi-Token Prediction via Self-Distillation." arXiv:2602.06019. arxiv.org/...2602.06019
9. ^1 ^2Gerontopoulos, A. et al. (2025). "Multi-Token Prediction Needs Registers." arXiv:2505.10518. arxiv.org/...2505.10518
10. ^Liu, X. et al. (2025). "L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models." Proceedings of NeurIPS 2025. arxiv.org/...2505.17505
11. ^1 ^2Xu and Che (2026). "Evolving LLMs from Next-Token Prediction to Multi-Token Prediction via Self-Distillation." MDPI Electronics, Vol. 15, Issue 7, 1533. mdpi.com/...1533
12. ^Leviathan, Y., Kalman, M., and Matias, Y. (2023). "Fast Inference from Transformers via Speculative Decoding." Proceedings of ICML 2023. arxiv.org/...2211.17192
13. ^Chen, C., Borgeaud, S., Irving, G., et al. (2023). "Accelerating Large Language Model Decoding with Speculative Sampling." arxiv.org/...2302.01318
14. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17Liu, Y. (October 27, 2026). "3 Ways to Boost MoE Inference on NVIDIA DGX Spark." NVIDIA Technical Blog. developer.nvidia.com/...erence-on-nvidia-dgx-spark
15. ^1 ^2 ^3 ^4Huang, Z. et al. (2025). "MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE." NeurIPS 2025. arXiv:2505.19645. arxiv.org/...2505.19645
16. ^1 ^2 ^3SGLang Documentation. "Speculative Decoding." docs.sglang.io/...speculative_decoding
17. ^1 ^2 ^3 ^4 ^5 ^6vLLM Documentation. "MTP (Multi-Token Prediction)." docs.vllm.ai/...mtp
18. ^1 ^2 ^3Qwen Team. "Qwen3-Next-80B-A3B-Instruct" model card. Hugging Face. huggingface.co/...Qwen3-Next-80B-A3B-Instruct
19. ^1 ^2 ^3GLM-4.5 Team (2025). "GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models." arXiv:2508.06471. arxiv.org/...2508.06471
20. ^1 ^2 ^3 ^4 ^5Xiaomi LLM-Core Team (2025). "MiMo: Unlocking the Reasoning Potential of Language Model, From Pretraining to Posttraining." arXiv:2505.07608. arxiv.org/...2505.07608
21. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10NVIDIA (2026). "Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning." arXiv:2604.12374. arxiv.org/...2604.12374
22. ^vLLM Team (September 11, 2025). "vLLM Now Supports Qwen3-Next: Hybrid Architecture with Extreme Efficiency." vLLM Blog. blog.vllm.ai/...qwen3-next
23. ^1 ^2Zhao, G. et al. (2026). "Self-Distillation for Multi-Token Prediction." arXiv:2603.23911. arxiv.org/...2603.23911
24. ^1 ^2 ^3Mahajan, D. et al. (2025). "Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries." ICLR 2026. arXiv:2510.14751. arxiv.org/...2510.14751
25. ^Gat, I. et al. (2025). "Set Block Decoding is a Language Model Inference Accelerator." arXiv:2509.04185. arxiv.org/...2509.04185
26. ^DeepSeek-AI. "DeepSeek-V3" model card. Hugging Face. huggingface.co/...DeepSeek-V3
27. ^DeepSeek-AI. "DeepSeek-V3 config.json." Hugging Face. huggingface.co/...config.json
28. ^DeepSeek-AI. "DeepSeek-V3 Weight File Documentation" (README_WEIGHTS.md). Hugging Face. huggingface.co/...README_WEIGHTS.md

Show all 28 references