Medusa

Medusa is a large language model inference acceleration framework that speeds up text generation by adding multiple lightweight decoding heads on top of an existing model to predict several future tokens in parallel, then verifying the candidates with a tree-based attention mechanism in a single forward pass. Unlike conventional speculative decoding, Medusa needs no separate draft model: the extra heads read directly from the base model's own final hidden states, so a single model both proposes and verifies tokens. The original paper reports that Medusa-1 (a frozen backbone with only the heads trained) delivers more than 2.2 times wall-clock speedup with no loss of generation quality, while Medusa-2 (joint fine-tuning) raises that to 2.3-3.6 times. ^[1][2]

The framework was introduced in the 2024 paper "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads" by Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao, with institutional affiliations spanning Princeton University, the University of Illinois Urbana-Champaign, Carnegie Mellon University, the University of Connecticut, and Together AI. ^[1][9] The authors summarize the core idea as "adding extra decoding heads to predict multiple subsequent tokens in parallel" and using "a tree-based attention mechanism" that "constructs multiple candidate continuations and verifies them simultaneously in each decoding step." ^[1] The work appeared first as arXiv preprint 2401.10774 on 19 January 2024 and was subsequently published at the 41st International Conference on Machine Learning (ICML 2024). ^[1] The official implementation is maintained at github.com/FasterDecoding/Medusa under an Apache 2.0 license. ^[2] Medusa is one of the most widely cited methods in the broader field of inference optimization for transformer models.

What is the autoregressive bottleneck Medusa solves?

Transformer-based language models generate text one token at a time. At each decoding step the model performs a full forward pass through all its transformer layers, attending to every previously generated token via the key-value cache, and produces a probability distribution over the vocabulary. Exactly one token is sampled or taken greedily, the token is appended to the sequence, and the process repeats. Because each step depends on the output of the previous step, the decoding loop is inherently sequential and cannot be parallelized across the output dimension.

This design creates a pronounced latency bottleneck, particularly on modern GPUs. Large matrix multiplications that dominate transformer inference can saturate GPU arithmetic units on large batch sizes, but at batch size 1 (the typical configuration for interactive or on-demand generation) the same operations are heavily memory bandwidth bound. The GPU reads model parameters from DRAM to perform each forward pass, and those reads consume far more time than the arithmetic itself. The result is that the GPU is underutilized: arithmetic throughput is not the limiting factor; bandwidth to model weights is. The Medusa paper frames this directly, noting that each decoding step "necessitates moving the full model parameters from High-Bandwidth Memory (HBM) to the accelerator's cache." ^[1] Because each step takes roughly the same time whether it produces one token or could have produced several, any method that generates multiple tokens per step at negligible additional cost directly reduces latency. ^[1]

How does Medusa differ from speculative decoding?

Speculative decoding was the dominant method for exploiting this observation before Medusa. In speculative decoding a small draft model generates a candidate sequence of several tokens in a single pass. Those candidates are then verified by the target model in a single batched forward pass that evaluates all candidate positions in parallel, using rejection sampling to decide how many tokens to accept. The expected number of accepted tokens per verification pass exceeds 1.0 if the draft model's predictions correlate well with the target model's, reducing the total number of target model forward passes required.

Speculative decoding can be highly effective, but it introduces several practical difficulties. The draft model must be architecturally compatible with the target model so that probability distributions can be compared under the rejection sampling criterion. Finding or training an appropriate draft model for a custom fine-tuned target model requires significant effort. The system must run two models concurrently, doubling memory management complexity and complicating multi-GPU or multi-node deployments. Furthermore, the gains from speculative decoding are sensitive to how closely the draft model's output distribution matches the target model's: poorly matched models yield low acceptance rates and may produce no speedup at all. For fine-tuned or domain-adapted models, there is often no ready-made draft model available. The Medusa authors note that speculative decoding's implementation "is impeded by the challenges associated with acquiring and maintaining a separate draft model." ^[1]

The Medusa paper benchmarks standard speculative decoding on the same Vicuna model family used in its own evaluation and reports speedup factors of 1.47 times on Vicuna-7B, 1.56 times on Vicuna-13B, and 1.60 times on Vicuna-33B. ^[1] These figures are considerably below what Medusa achieves, motivating the question of whether the separate draft model can be eliminated entirely.

How does the Medusa approach work?

Medusa's central insight is that the base model's final hidden state already contains rich information about what token sequences are likely to follow. The original language model head maps this hidden state to a next-token probability distribution. There is no fundamental barrier to adding additional heads that map the same hidden state to probability distributions over the token two positions ahead, three positions ahead, and so on. These additional heads are structurally independent of each other and of the base model weights; they can be trained while the base model is held frozen, or trained jointly with the base model under a modified training recipe. ^[1]

Formally, let $h_t$ denote the hidden state produced by the last transformer layer at decoding step $t$. The $k$-th Medusa head computes:

$$p_t^{(k)} = \text{softmax}!\left(W_2^{(k)} \cdot \left(\text{SiLU}\left(W_1^{(k)} \cdot h_t\right) + h_t\right)\right)$$

where $W_1^{(k)} \in \mathbb{R}^{d \times d}$ and $W_2^{(k)} \in \mathbb{R}^{d \times V}$ are the learned parameters of head $k$, $d$ is the model's hidden dimension, and $V$ is the vocabulary size. The residual connection ($+ h_t$) is important for stability: it allows the head to start from a position close to zero correction and gradually learn to refine the base model's implicit belief about future tokens. The base model's original language model head is treated as head 0, predicting the next token, while heads 1 through $K$ predict tokens at offsets 2 through $K{+}1$. Five heads is the configuration found empirically most useful; adding more yields diminishing returns because prediction accuracy for positions further in the future declines. ^[1]

The next-next-token prediction accuracy of the first additional head (head 1) is approximately 60% top-1 and over 80% top-5 in practice, which is high enough to make tree-based candidate verification highly efficient. ^[1]

How does tree attention verify candidates in parallel?

Given the top-$s_k$ predictions from each of the $K$ heads, the candidate set for a single decoding step consists of all possible continuations formed by picking one token per head. With $K = 5$ heads and $s = 2$ top candidates per head, the Cartesian product produces $2^5 = 32$ five-token candidate sequences. Verifying each sequence independently would require 32 forward passes, which would be far slower than ordinary decoding.

Tree attention solves this by recognizing that many candidate sequences share a common prefix. Instead of 32 independent sequences, the candidates can be arranged as a tree in which the root is the current token, the next level contains the top-$s_1$ predictions from head 1, each node at that level branches into the top-$s_2$ predictions from head 2, and so on. Any two paths through the tree that share a prefix node can share the key-value computations for those prefix tokens. A single forward pass processes all unique tree nodes simultaneously by using a custom attention mask that restricts each node's attention to only its direct ancestors in the tree, rather than to all other nodes in the batch. ^[1]

This modified attention mask is the key technical mechanism. In standard multi-query batched decoding, every token in the batch attends to the full context plus all other batch tokens, which would allow candidates from different branches to incorrectly influence each other. By masking out tokens that are not ancestors of the current node, the tree attention ensures that each candidate position sees exactly the context it would see if that path through the tree were the true generated sequence. The resulting forward pass is semantically equivalent to running $s_1 \times s_2 \times \ldots \times s_K$ separate forward passes but performs only as much compute as a single pass with the number of unique tree nodes.

The tree need not be dense. With five heads and two top candidates per head the full Cartesian product yields 62 tree nodes (including the root). The paper evaluates different tree configurations and finds that a sparse tree of approximately 64 nodes achieves a better trade-off between coverage and computational overhead than either the full dense tree or a very small tree. ^[1] The tree configuration is treated as a hyperparameter that can be tuned per model and use case. In TensorRT-LLM's implementation the tree structure is a runtime parameter, allowing operators to adjust it without recompiling the model. ^[6][7]

How does Medusa decide which tokens to accept?

Once the tree attention forward pass completes, the system must decide how many of the proposed tokens to accept. Standard speculative decoding uses rejection sampling, which requires comparing the target model's probability of each token against the draft model's probability, accepting with probability equal to the minimum of their ratio and 1. This scheme guarantees that the accepted sequence follows exactly the target model's distribution, which is important for correctness at non-zero sampling temperatures.

Medusa introduces a different scheme called typical acceptance, which is appropriate given that the proposal heads are part of the same model rather than a separate draft model. Typical acceptance accepts a candidate token from position $k$ if its probability under the base model exceeds a threshold that depends on the entropy of the base model's distribution at that position:

$$p^{(0)}(x_{n+k} \mid \ldots) > \min!\left(\epsilon,; \delta \cdot \exp!\left(-H\left(p^{(0)}\right)\right)\right)$$

where $H$ denotes the entropy of the distribution, $\epsilon$ is a hard lower bound, and $\delta$ is a scaling factor. At low entropy (confident predictions) the threshold is tight, meaning only very probable tokens are accepted. At high entropy (uncertain predictions) the threshold is looser, accepting a broader range of plausible tokens. This scheme is inspired by typical sampling from information theory, where tokens near the entropy of the distribution, rather than only the highest-probability tokens, are considered typical outputs. The paper reports that typical acceptance provides approximately 10% speedup over greedy acceptance by approving longer candidate sequences, while maintaining generation quality closely comparable to random sampling. ^[1]

At temperature zero (greedy decoding), typical acceptance reduces to simple comparison against the argmax, and no stochasticity is involved. At higher temperatures, the entropy-adaptive threshold allows more liberal acceptance without requiring rejection sampling against a separate draft model's probabilities.

After acceptance, the longest valid prefix of the candidate tree that satisfies the acceptance criterion is appended to the context, the KV cache is updated, and decoding continues. The average number of new tokens accepted per decoding step, called the acceleration rate, ranges from about 3.0 to 3.5 for Medusa-2 configurations, meaning each forward pass produces three to three-and-a-half tokens on average instead of one. ^[1]

How is Medusa trained: Medusa-1 versus Medusa-2?

Medusa offers two distinct training regimes, chosen based on available compute and the desired trade-off between training cost and inference speedup. ^[1][2]

Medusa-1: frozen backbone

In the Medusa-1 regime the base language model's weights are held completely frozen, and only the additional heads are trained. The training objective is a weighted sum of cross-entropy losses across the $K$ additional heads:

$$\mathcal{L}{\text{Medusa-1}} = \sum{k=1}^{K} -\lambda_k \log p_t^{(k)}!\left(y_{t+k+1}\right)$$

where $\lambda_k = 0.8^k$ is a decaying weight that places more emphasis on near-future positions, which are easier to predict and more likely to be accepted. The frozen backbone ensures that the base model's output distribution is completely unaffected; the paper describes Medusa-1 as "enabling lossless inference acceleration" because the set of possible outputs is unchanged. ^[1]

Training Medusa-1 requires minimal compute. The reference implementation trains the heads for Vicuna-7B on approximately 60,000 ShareGPT conversation samples in about five hours on a single NVIDIA A100 PCIE GPU. ^[2] The heads contain only a small fraction of total parameters compared to the base model, so memory requirements are modest and training can be performed on hardware that cannot accommodate full fine-tuning of the base model. Medusa-1 achieves speedup factors of 2.18 times on Vicuna-7B and 2.33 times on Vicuna-13B. ^[1]

Medusa-2: joint training

Medusa-2 trains the heads and the base model backbone jointly, using a combined loss that includes both the standard next-token prediction loss and the Medusa head losses:

$$\mathcal{L}{\text{Medusa-2}} = \mathcal{L}{\text{LM}} + \lambda_0 \cdot \mathcal{L}_{\text{Medusa-1}}$$

Joint training allows the backbone to learn internal representations that are more informative for the heads' multi-step predictions, producing higher head accuracy and thus higher acceptance rates. The paper notes that this regime needs "a special training recipe that preserves the backbone model's capabilities." ^[1] The additional training complexity is managed through two techniques. First, differential learning rates are used: the backbone is updated at a lower rate (around $1 \times 10^{-4}$) while the heads are updated at a rate approximately four times higher ($2 \times 10^{-3}$) to allow the heads to adapt quickly to the evolving backbone representations. Second, a two-stage warmup procedure trains the heads alone for an initial period before enabling backbone updates, preventing the joint optimization from destabilizing head training early in the run. ^[1]

The LoRA (Low-Rank Adaptation) adapter technique is used for the backbone component of Medusa-2 training to limit memory consumption. A rank-32 LoRA with $\alpha = 16$ and dropout of 0.05 is applied to the backbone, meaning the actual parameter updates are factored into low-rank matrices and do not require storing full-rank gradient tensors. This makes Medusa-2 training feasible on hardware with limited memory, though it remains more demanding than Medusa-1. ^[1]

Medusa-2 achieves substantially higher speedups: 2.83 times on both Vicuna-7B and Vicuna-13B, 2.66 times on Zephyr-7B-Beta, and 2.35 times on Vicuna-33B. ^[1] On certain task categories the gains are larger: the paper reports 3.29 times speedup on coding tasks and 3.62 times on extraction tasks with Vicuna-7B under Medusa-2, reflecting higher head prediction accuracy when the output is more structured and predictable. ^[1]

Self-distillation for custom models

A practical challenge arises when applying Medusa to a model that has been fine-tuned on proprietary or unavailable data. If the original training data cannot be used to train the heads, standard supervised training is not possible. Medusa addresses this through a self-distillation mechanism: the model itself is used to generate training data from freely available seed prompts (such as ShareGPT or UltraChat). The generated outputs, which reflect the model's own behavior, are used as the supervision signal for head training. ^[1][2]

For Medusa-2 self-distillation, the backbone loss is replaced with a KL divergence loss that keeps the backbone's next-token distribution close to the original model's distribution:

$$\mathcal{L}{\text{LM-distill}} = \text{KL}!\left(p{\text{original},t}^{(0)} ,|, p_t^{(0)}\right)$$

This prevents the backbone from drifting away from its original behavior while still allowing it to develop internal representations useful for the heads. Self-distillation makes it practical to add Medusa to any instruction-tuned or RLHF-trained model without access to the original fine-tuning data, which is important for models trained with proprietary instruction datasets or human preference annotations. ^[1]

How fast is Medusa?

The benchmark results in the paper use the MT-bench evaluation suite, which covers eight categories of tasks including writing, reasoning, math, coding, extraction, STEM, humanities, and roleplay. Speedup is measured as wall-clock tokens generated per second relative to vanilla autoregressive decoding at batch size 1. MT-bench quality scores are used to verify that output quality is preserved. The headline result reported in the abstract is that "Medusa-1 can achieve over 2.2x speedup without compromising generation quality, while Medusa-2 further improves the speedup to 2.3-3.6x." ^[1]

Speedup vs. vanilla autoregressive decoding

Scroll sideways for more →

Acceptance rate and overhead

Scroll sideways for more →

The acceleration rate column reports the average number of new tokens accepted per decoding step (including the base model's own token). The overhead column reports the cost of a Medusa forward pass relative to a standard forward pass. An acceleration rate of 3.47 with 1.22x overhead yields a net speedup of roughly $3.47 / 1.22 \approx 2.84$, consistent with the reported 2.83 times speedup for Vicuna-7B. ^[1]

Quality preservation on MT-Bench

Medusa-2 models achieve MT-Bench scores within a small margin of the original models, confirming that the joint training does not meaningfully degrade output quality: ^[1]

Scroll sideways for more →

Task-specific speedups (Vicuna-7B, Medusa-2)

Scroll sideways for more →

Structured tasks like coding and extraction, where the model's next-token predictions are more deterministic and the Medusa heads achieve higher accuracy, yield the largest speedups. Open-ended creative writing and mathematical reasoning, where token probabilities are more diffuse, yield somewhat lower but still substantial gains.

How does Medusa compare with vanilla speculative decoding?

Medusa significantly outperforms standard speculative decoding (using a separately trained draft model) on the same model family in the paper's evaluation: ^[1]

Scroll sideways for more →

The comparatively modest gains from speculative decoding in this benchmark reflect the difficulty of finding a well-matched draft model for instruction-tuned Vicuna models. The draft model used in the paper's speculative decoding baseline is a smaller general-purpose language model, and the distribution mismatch between draft and target leads to low acceptance rates. Medusa avoids this problem by deriving its proposals directly from the target model's own hidden states, which are by construction well-correlated with the target model's output distribution.

How does Medusa compare with EAGLE?

EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) is a subsequent speculative decoding method that, like Medusa, adds prediction machinery to the base model to avoid requiring a separate draft model. EAGLE was published in January 2024 alongside the Medusa paper. ^[4] Rather than predicting multiple future tokens from the current hidden state, EAGLE trains a small autoregressive draft head that operates on the base model's feature sequence, producing speculative tokens that are contextually conditioned on the preceding hidden states. ^[4]

Comparative evaluations conducted as part of the EAGLE-2 paper (arXiv 2406.16858, 2024) show EAGLE and its successors achieving higher speedups than Medusa on a shared benchmark: ^[5]

Scroll sideways for more →

These EAGLE-2 benchmark numbers use temperature 0 on the MT-bench dataset and are measured on the same hardware under the same conditions, providing a fair cross-method comparison. ^[5] The results indicate that EAGLE's autoregressive draft head, by conditioning each speculative token on prior hidden states rather than predicting all positions from a single state snapshot, achieves higher acceptance rates and thus higher practical speedup. Medusa's simpler per-position feed-forward heads are easier to train and integrate but cannot match the richer conditional structure of EAGLE's draft model.

The Medusa-2 numbers in the original Medusa paper (2.83 times on Vicuna-7B) are higher than the EAGLE-2 benchmark numbers for Medusa (1.91 times) because they measure different things: Medusa's own evaluation uses Medusa-2 jointly trained models with typical acceptance, while the EAGLE-2 benchmark uses Medusa in a configuration closer to Medusa-1 with greedy acceptance. The discrepancy illustrates how significantly training regime, acceptance scheme, and evaluation methodology affect reported speedup figures.

Medusa's practical advantage over EAGLE is its simplicity: training requires only a few hours on a single GPU, the implementation is a minimal extension of the base model with no autoregressive draft head training loop, and integration into existing serving frameworks is straightforward. For practitioners who need a working acceleration solution quickly and are not optimizing for maximum possible speedup, Medusa's ease of use is a meaningful benefit.

Where is Medusa used?

TensorRT-LLM

NVIDIA's TensorRT-LLM library includes native support for Medusa decoding. ^[6] The implementation provides a convert_checkpoint.py script that loads Medusa head weights alongside base model weights and produces a TensorRT engine that runs the combined forward pass. The Medusa tree configuration is exposed as a runtime parameter (medusa_choices), allowing operators to vary the candidate tree structure without rebuilding the engine. ^[7] The TensorRT-LLM integration supports greedy decoding (temperature 0) and has been tested with Vicuna-7B, Vicuna-13B, and the quantized Llama-3.1-8B-Medusa-FP8 model, combining Medusa acceleration with FP8 weight quantization for further efficiency. ^[7][13] Beam search and certain other decoding strategies are incompatible with the Medusa tree structure and are not supported in this implementation. ^[7]

Text Generation Inference (TGI)

HuggingFace's Text Generation Inference server added support for Medusa in its inference path. ^[2] TGI's integration allows users running Medusa-augmented checkpoints stored on the Huggingface Hub to transparently benefit from tree-based acceleration without code changes beyond specifying the model path. This integration extended Medusa's practical reach to the large community of practitioners using TGI for model deployment.

RTP-LLM

Ant Group's RTP-LLM inference framework, used in large-scale production deployments, also incorporates Medusa support. ^[2] RTP-LLM targets high-throughput serving at scale, and the Medusa integration follows the same principle of adding the head-based candidate tree to the serving pipeline.

Together AI

Together AI, one of the institutional sponsors of the research, deployed Medusa in its inference API to accelerate generation for customers. The Together AI blog noted that the 33B Vicuna model with Medusa acceleration could operate at speeds comparable to the 13B Vicuna baseline, effectively doubling throughput for larger models where memory bandwidth pressure is most acute. ^[3] Together AI's involvement in the research ensured that the implementation was tested at production scale from early in the project's life.

vLLM

vLLM supports Medusa as one of several speculative decoding strategies. ^[8] The vLLM implementation integrates Medusa heads into the scheduler and token generation pipeline alongside draft-model-based speculative decoding and prompt lookup decoding. As of 2024, vLLM's Medusa support is described as preliminary relative to the draft-model pathway, with ongoing kernel optimization work aimed at improving throughput at larger batch sizes where the tree attention mechanism is more complex to schedule efficiently. ^[8] vLLM uses PagedAttention for KV cache management, and coordinating block allocation for tree-based speculative candidates adds additional complexity compared to single-sequence decoding.

What are Medusa's limitations?

Batch size dependency. Medusa's speedup is most pronounced at batch size 1 (single concurrent request), where the bottleneck is memory bandwidth per token and the cost of the extra forward pass for tree verification is well amortized. At large batch sizes, where the model is already operating near peak arithmetic efficiency, the overhead of verifying multiple candidate sequences does not yield as much net improvement. Most serving applications involve mixed batch sizes, and the benefit diminishes as concurrency increases.

Temperature and acceptance rates. At very low generation temperatures, token predictions are highly concentrated, and Medusa heads achieve high acceptance rates. At high temperatures, where output distributions are flatter and more random, the heads predict incorrectly more often, reducing acceptance rates and shrinking the effective speedup. The typical acceptance scheme partially mitigates this but does not fully recover the speedup at high temperatures that rejection-sampling-based speculative decoding with a well-matched draft model can provide.

EAGLE and successors outperform Medusa. As shown in the comparison section, EAGLE-2 and subsequent methods that use richer draft models have surpassed Medusa's speedup numbers on shared benchmarks. ^[5] For practitioners whose primary goal is maximum speedup and who are willing to accept greater implementation complexity, EAGLE-family methods are generally preferred as of 2024 and 2025. Medusa occupies a different point on the complexity-speedup trade-off curve: easier to train, integrate, and operate, but not the highest-speedup option available.

Greedy-only support in some implementations. TensorRT-LLM's Medusa integration supports only temperature 0 (greedy decoding) as of its initial release, limiting its applicability to sampling-based generation. ^[7] This is a constraint of the specific implementation rather than the method itself, but it affects users of that framework.

No gain at long contexts. Medusa's heads predict future tokens based on the current hidden state, which encodes the full context. The quality of those predictions does not systematically degrade with context length, but the overhead of tree attention grows with the number of tree nodes and is fixed regardless of context length. For very short prompts where generation terminates quickly, the training and deployment overhead may not be worth the speedup. Medusa is most valuable for workloads generating hundreds to thousands of tokens per request.

Memory overhead. Each Medusa head adds parameters proportional to the model's hidden dimension times the vocabulary size. For a model with hidden dimension $d = 4096$ and vocabulary size $V = 32000$, each head contributes approximately $4096 \times 32000 \times 2 \approx 250$ million parameters at float16 precision. Five heads add roughly 1.25 billion parameters, equivalent to a small fraction of a large model's total size but not negligible for memory-constrained deployments.

Influence and subsequent work

Medusa's publication contributed to a productive period of research into draft-model-free speculative decoding. The Hydra method (2024) extends Medusa's multi-head concept with additional conditioning mechanisms. ^[11] EAGLE (2024) and EAGLE-2 (2024) replace the per-position feed-forward heads with an autoregressive draft head operating on features from the base model, achieving higher acceptance rates at the cost of a more complex training procedure. ^[4][5] EAGLE-3 (2025) further scales this approach with training-time modifications to the base model's feature representations. ^[14] The Lookahead Decoding method (2024) takes a different approach, using Jacobi iteration to generate candidate continuations from a fixed-point approximation rather than trained heads.

Medusa itself underwent revisions. The expanded paper (arXiv 2401.10774, which reached v3) incorporated the Medusa-2 training recipe, the self-distillation mechanism, the typical acceptance scheme, and expanded evaluation across additional models. ^[1] These additions addressed several of the limitations identified in the initial preprint. The expanded version (v3) was posted on 14 June 2024, ahead of the ICML 2024 camera-ready release. ^[1]

The framework's code repository, github.com/FasterDecoding/Medusa, served as a reference implementation that was studied and extended by multiple inference framework teams. ^[2][9] The FasterDecoding GitHub organization also hosts subsequent related projects.

Relationship to Hydra

Hydra (arXiv 2402.05109), introduced by Zachary Ankner and colleagues in February 2024, is the most direct extension of Medusa. ^[11] It identifies that Medusa's draft heads are sequentially independent: each head predicts its token position purely from the base hidden state, with no awareness of which tokens the earlier heads proposed. Hydra makes the draft heads sequentially dependent by conditioning each head on the candidate tokens generated so far, which raises draft accuracy and acceptance length. Hydra reports about 2.70 times throughput improvement over vanilla autoregressive decoding and roughly 1.31 times improvement over Medusa on Vicuna models at batch size 1, while keeping the single-model, no-separate-draft-model property that makes Medusa attractive. ^[11]

2025-2026 developments

Medusa remained an actively referenced baseline and a deployed serving option through 2025 and into 2026, even as the EAGLE family overtook it on raw speedup.

NVIDIA published production benchmarks for Medusa on Llama 3.1. In a September 2024 low-latency inference report, NVIDIA measured Medusa heads attached to Llama 3.1 models on an HGX H200 system with NVLink Switch, reporting up to 1.9 times higher token throughput per user for Llama 3.1 405B (108 versus 56 tokens per second per user) and about 1.5 times for Llama 3.1 70B (268 versus 184 tokens per second per user). ^[12] NVIDIA also released a ready-to-serve checkpoint, Llama-3.1-8B-Medusa-FP8, that combines three Medusa heads with FP8 quantization and reports an average acceptance length of 2.07 tokens per generation step on MT-Bench while preserving accuracy (68.3% on 5-shot MMLU). ^[13] The TensorRT-LLM Medusa example was subsequently extended beyond Vicuna to cover Qwen2 and a Llama-3.1-70B-Medusa configuration. ^[7]

The method continued to be integrated into mainstream serving stacks. vLLM carried Medusa into its rewritten V1 engine as a dedicated MedusaProposer alongside n-gram and EAGLE proposers, and the vLLM v0.13.0 release of December 2025 added a Medusa GPU-CPU synchronization optimization, indicating ongoing maintenance rather than deprecation. ^[17] At the same time, some newer engines did not adopt it: SGLang's speculative decoding stack standardized on EAGLE, EAGLE-3, n-gram, and native multi-token prediction draft models, and does not list Medusa among its supported algorithms, recommending EAGLE-3 as the default high-throughput option. ^[15]

A related shift was the move from Medusa-style heads bolted onto a finished model toward multi-token prediction (MTP) heads trained jointly with the base model from the start. Models such as DeepSeek-V3 ship MTP modules that double as speculative-decoding draft heads, and analyses describe Medusa as effectively a post-hoc version of the same multi-token-prediction idea. ^[16] Academic work also continued to refine Medusa directly: a February 2025 paper replaced Medusa's fixed candidate tree with a dynamically constructed tree (dynamic tree attention), reporting improved decoding efficiency at equal generation quality. ^[18] By 2026, industry write-ups characterized Medusa as a lightweight, low-cost option that is often retained as a fallback tier in serving stacks where EAGLE-3 handles chat workloads and MTP-based drafting handles long-context generation. ^[19]

See also

• Speculative Decoding
• Inference optimization
• Multi-token prediction
• vLLM
• PagedAttention
• Flash Attention
• Together AI
• Continuous Batching
• GPTQ
• Transformer
• Tri Dao

References

1. Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., & Dao, T. (2024). Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. *Proceedings of the 41st International Conference on Machine Learning (ICML 2024)*. arXiv:2401.10774. https://arxiv.org/abs/2401.10774 ↩
2. FasterDecoding. (2024). Medusa GitHub Repository. https://github.com/FasterDecoding/Medusa ↩
3. Together AI. (2023). Medusa: Simple framework for accelerating LLM generation with multiple decoding heads. Together AI Blog. https://www.together.ai/blog/medusa ↩
4. Li, Y., Wei, F., Zhang, C., & Zhang, H. (2024). EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. arXiv:2401.15077. https://arxiv.org/abs/2401.15077 ↩
5. Li, Y., Wei, F., Zhang, C., & Zhang, H. (2024). EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. *Proceedings of EMNLP 2024*. arXiv:2406.16858. https://arxiv.org/abs/2406.16858 ↩
6. NVIDIA. (2024). Speculative Decoding in TensorRT-LLM. https://nvidia.github.io/TensorRT-LLM/advanced/speculative-decoding.html ↩
7. NVIDIA. (2024). TensorRT-LLM Medusa Example. https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/medusa/README.md ↩
8. vLLM Team. (2024). How Speculative Decoding Boosts vLLM Performance by up to 2.8x. vLLM Blog. https://blog.vllm.ai/2024/10/17/spec-decode.html ↩
9. University of Illinois Coordinated Science Laboratory. (2024). Team effort leading to significantly higher LLM execution speed and high-profile industrial adoption. https://csl.illinois.edu/news-and-media/team-effort-leading-to-significantly-higher-llm-execution-speed-and-high-profile-industrial-adoption ↩
10. Medusa Project Website. https://sites.google.com/view/medusa-llm
11. Ankner, Z., Parthasarathy, R., Nrusimha, A., Rinard, C., Ragan-Kelley, J., & Brandon, W. (2024). Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding. arXiv:2402.05109. https://arxiv.org/abs/2402.05109 ↩
12. NVIDIA. (2024). Low Latency Inference Chapter 1: Up to 1.9x Higher Llama 3.1 Performance with Medusa on NVIDIA HGX H200 with NVLink Switch. NVIDIA Technical Blog. https://developer.nvidia.com/blog/low-latency-inference-chapter-1-up-to-1-9x-higher-llama-3-1-performance-with-medusa-on-nvidia-hgx-h200-with-nvlink-switch/ ↩
13. NVIDIA. (2024). Llama-3.1-8B-Medusa-FP8 Model Card. Hugging Face. https://huggingface.co/nvidia/Llama-3.1-8B-Medusa-FP8 ↩
14. Li, Y., Wei, F., Zhang, C., & Zhang, H. (2025). EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. arXiv:2503.01840. https://arxiv.org/abs/2503.01840 ↩
15. SGLang. (2025). Speculative Decoding. SGLang Documentation. https://docs.sglang.ai/advanced_features/speculative_decoding.html ↩
16. DeepSeek-AI. (2024). DeepSeek-V3 Technical Report. arXiv:2412.19437. https://arxiv.org/abs/2412.19437 ↩
17. vLLM Project. (2025). Release v0.13.0. GitHub. https://github.com/vllm-project/vllm/releases/tag/v0.13.0 ↩
18. Zhang, Z. (2025). Acceleration Multiple Heads Decoding for LLM via Dynamic Tree Attention. arXiv:2502.05947. https://arxiv.org/abs/2502.05947 ↩
19. Nguyen, D. (2026). Speculative Decoding 2026: 2.8x Faster LLM Inference. SyncSoft.AI Blog. https://www.syncsoft.ai/en/blog/speculative-decoding-eagle3-medusa-deepseek-mtp-chinese-chuhai-2026 ↩