DFlash

DFlash is a speculative decoding method in which the draft model is a small block-diffusion network rather than an autoregressive one. The drafter proposes a whole block of future tokens in a single forward pass, conditioned on hidden features taken from the large target model, and the target then verifies the block in parallel, so the final output keeps the target model's distribution.^[1] The method was introduced in "DFlash: Block Diffusion for Flash Speculative Decoding" by Jian Chen, Yesheng Liang and Zhijian Liu of UC San Diego, posted to arXiv on 5 February 2026 and accepted at ICML 2026. The abstract reports "over 6x lossless acceleration across a range of models and tasks" and up to 2.5 times the speedup of EAGLE-3.^[1]

Code and draft checkpoints are published by Z Lab, Liu's group at UC San Diego, in the MIT-licensed z-lab/dflash repository (created 4 January 2026).^[2][3][7] Within six months DFlash support had been merged into SGLang, vLLM and llama.cpp, NVIDIA reported running it in TensorRT-LLM, and several model developers began shipping DFlash drafters alongside their own models.^{[4][7][9][10][11]} On 18 August 2026 Inco AI, a company Liu co-founded, released DFlash 2, which adds a candidate path selector and short convolutions to the drafter and is the draft model built into Inco's Splash engine for Macs.^[4][7][18]

Background

In standard speculative decoding a cheap drafter guesses several future tokens and the target model checks them all in one forward pass, keeping the longest correct prefix plus one token of its own. The DFlash paper writes the per-token latency as the draft time plus the verification time, divided by the expected number of tokens accepted per cycle, so a method gets faster either by accepting more tokens or by drafting more cheaply.^[1] Autoregressive drafters, including EAGLE-3, still produce draft tokens one at a time, so drafting cost grows linearly with the number of tokens drafted. To keep that cost low they are kept very shallow (a single transformer layer in EAGLE-3), which limits how many of their tokens are accepted. The authors argue that this caps practical speedups at about 2 to 3 times.^[1]

Diffusion language models can denoise a block of masked tokens at once, but the paper notes that open diffusion models generally trail autoregressive models in quality and need many denoising steps to stay accurate.^[1] Earlier attempts to use diffusion for drafting either used large drafters (DiffuSpec and SpecDiff-2 use 7B-parameter diffusion models, which the authors say limits speedups to 3 to 4 times) or small models with limited capacity (PARD, which the authors put at a ceiling of about 3 times).^[1] DFlash's premise, summarized in the paper as "the target knows best," is that a small diffusion drafter can be accurate if it reads the target model's own hidden states, which already carry information about upcoming tokens.^[1]

Method

Parallel block drafting

A DFlash drafter takes the last token confirmed by the target (the "anchor") followed by mask tokens, and fills every masked position of the block in one forward pass. Because the cost of that pass barely changes with block size, drafting time is "largely insensitive" to the number of drafted tokens for moderate block sizes. The drafter can therefore afford several layers. The paper reports that a five-layer DFlash drafter generating 16 tokens has lower latency and a longer acceptance length than EAGLE-3 generating 8 tokens.^[1]

Conditioning on the target model

After the target's prefill pass, DFlash extracts hidden states from a fixed set of target layers (five layers, uniformly spaced from the second layer to the third-to-last), concatenates them and projects them into a single context feature.^[1] EAGLE-3 also uses target features, but fuses them with token embeddings at the drafter's input only, where the paper says their influence is diluted as the drafter gets deeper. DFlash instead injects the projected feature into the key and value projections of every draft layer and keeps it in the drafter's KV cache, which the authors credit for acceptance length continuing to rise as draft layers are added.^[1] The only extra parameterized component is the shared projection; for a Qwen3.5-35B-A3B target the paper puts it at about 42 MB in BF16.^[1]

Training

Drafters are trained against a frozen target, using a mixture of about 800,000 samples from NVIDIA's Nemotron Post-Training Dataset V2 and CodeAlpaca whose responses are regenerated by the target model.^[1] Rather than splitting each response into fixed blocks as standard block-diffusion training does, DFlash samples random anchor positions (512 per sequence), starts a block at each anchor and trains the drafter to predict the following block_size minus one tokens, which matches how the drafter is used at inference. All blocks in a sequence are trained together with a sparse attention mask: tokens attend bidirectionally within their own block and to the injected target features, but not across blocks.^[1] The loss is weighted toward early positions with an exponential decay, because an error early in a block invalidates everything after it. The drafter shares the target's token embedding and language-model head and keeps them frozen, so only the draft transformer layers are trained.^[1]

The paper's default drafters have five layers (eight for Qwen3-Coder) and a block size of 16 (10 for Llama 3.1).^[1] Training can be online, computing target features on the fly, or offline from cached features.^[1]

Comparison with other drafting methods

Sources: the DFlash paper for the classic, Medusa, EAGLE and DFlash rows; the DSpark abstract; Inco's DFlash 2 post.^[1][21][4] Like speculative decoding methods that use exact verification, DFlash is lossless in the sense that verification accepts or corrects each drafted token; it does not change the target model's accuracy.^[1][4]

Reported results

Paper benchmarks

With thinking disabled on Qwen3 models, using the Hugging Face Transformers backend on NVIDIA H200 GPUs, the paper reports these speedups over autoregressive decoding and average acceptance lengths (tau, tokens per verification step including the target's bonus token) for Qwen3-8B at temperature 0:^[1]

Across Qwen3-4B and Qwen3-8B the authors summarize an average 4.9 times speedup at temperature 0 (2.4 times EAGLE-3 with tree size 16) and 4.1 times at temperature 1.^[1] With thinking enabled the speedups were roughly 4.5 times at temperature 0 and 3.9 times at temperature 1.^[1] Chat is the weakest category in every table that includes a chat benchmark; for Qwen3-8B at temperature 0, MT-Bench reached 2.75 times against 6.08 times on MATH-500.^[1]

Serving results are lower and fall as concurrency rises. In SGLang on a single B200 with the FlashAttention-4 backend, Qwen3-8B on MATH-500 went from 230 to 1,175 output tokens per second at concurrency 1 (5.1 times) and from 5,694 to 16,076 at concurrency 32 (2.8 times).^[1] Against the official EAGLE-3 checkpoints for Llama 3.1 8B Instruct, trained on the same data, DFlash with block size 10 gave 2.4 times on GSM8K at concurrency 1 against 1.6 and 1.9 times for the two EAGLE-3 settings; at concurrency 32 DFlash kept 1.6 times while EAGLE-3 fell to 1.0 and 0.6 times.^[1] An appendix table on SGLang at concurrency 8 compares DFlash with the native multi-token prediction of Qwen3.5 models: on Qwen3.5-9B MATH-500, MTP accepted 6.7 tokens per step for 1.7 times, while DFlash accepted 7.3 for 3.5 times.^[1]

Ablations

• Depth. Eight draft layers gave longer acceptance than five, but five gave the best end-to-end speedup.^[1]
• Block size. A drafter trained with block size 16 still works at inference block size 8, but a block-8 drafter does not generalize up to 16. The authors suggest shrinking the block under compute-bound, large-batch serving, where verifying long blocks costs more.^[1]
• KV injection. On Qwen3-4B, injecting target features into every layer's keys and values beat EAGLE-3-style input fusion for both autoregressive and block-diffusion drafters.^[1]
• Without target features. A five-layer block-diffusion drafter with no target conditioning managed only 2.65 to 3.73 times across four math benchmarks at temperatures 0 and 1.^[1]
• Long context. A Qwen3.5-27B drafter trained at 4K context lost acceptance length beyond 4K (on the LongBench hotpotqa task it fell to 3.61 at 16K). Fine-tuning on 1,600 LongAlign samples raised that figure to 6.05.^[1]

Vendor and partner measurements

The NVIDIA post was co-written by five NVIDIA staff (engineers and product managers), UC San Diego's Hao Zhang and Zhijian Liu. The Google post describes work by a UC San Diego team led by Hao Zhang, with guidance from Google Cloud engineers.^[7][8]

DFlash 2

Inco AI published "DFlash 2: Keep Drafting Parallel" on 18 August 2026.^[4] Inco's diagnosis was that DFlash predicts each position independently, so a block can contain individually plausible tokens that do not fit together, and that accuracy also decays toward the end of the block. On a five-layer Qwen3-4B DFlash drafter on GSM8K, the top pick at the first position was correct 85.4 percent of the time, but the correct token was among the top 16 candidates 99.5 percent of the time. An oracle choosing correctly from those candidates would raise acceptance length from 4.27 to 6.79.^[4] DFlash 2 adds two parts:^[4]

• Path selector. The drafter keeps the top 16 candidates per position and scores every adjacent pair with DFlash's own logit plus a gated low-rank bilinear term over 256-dimensional token embeddings. The only sequential step is a walk over these precomputed scores; under sampling, rejection sampling restores the target distribution. On the Qwen3-4B drafter the selector alone raised acceptance length from 4.27 to 4.61 at temperature 0, for 2.0 million added parameters and 0.6 percent added cycle latency. A DSpark-style correction reached 4.49 with 77.8 million parameters and 9.6 percent added latency.
• Two-tap dynamic convolution. Inco found that within-block attention shrank in later drafter layers and attributed "suffix decay" to a local modeling problem. It inserts a two-tap depthwise convolution, whose kernel adapts to the content, before and after each attention and feed-forward sublayer. This added 16.5 million parameters (3 percent) and 0.7 percent latency and brought a five-layer drafter close to a 15-layer one, which would have added 15.2 percent latency.

Together the two additions cost 1.3 percent of draft-verify cycle latency. On Qwen3.5-4B (thinking on, temperature 1.0), with DFlash and DSpark drafters Inco trained under matched setups and the MTP module that ships with the model, mean acceptance length across five benchmarks was 4.54 for native MTP, 4.92 for DFlash, 5.49 for DSpark and 5.97 for DFlash 2. That is 1.05 tokens (21 percent) more than DFlash; Inco puts the per-benchmark gain at 16 to 25 percent.^[4]

Inco released DFlash 2 drafters for Qwen3.8-27B and Meta's Muse Glimmer the same day, later adding drafters for GLM-5.3 and GLM-5.3-Flash.^[4][19] The model cards (throughput) and the DFlash 2 post (mean acceptance length) report SGLang results on one H200 with each model's default sampling:^[4][5][6]

At concurrency 32 on Qwen3.8-27B, native MTP and a community DSpark drafter fell below autoregressive throughput on several tasks, while DFlash 2 stayed at or above it.^[5] These are Inco's own measurements. The Qwen and Muse Glimmer drafters are Apache-2.0; the two GLM drafters carry CC BY-NC-ND 4.0.^[19]

The DFlash 2 integration pull requests were opened on 18 August and merged into SGLang on 19 August, vLLM on 21 August (shipped in vLLM 0.28.0 on 26 August) and llama.cpp on 27 August. An Ollama pull request opened on 19 August was still open on 23 September 2026.^{[12][13][10][14]} Inco's post also links a prebuilt oMLX build with DFlash 2 support for Apple silicon, published in Z Lab's omlx-fork repository.^[4][2]

Adoption

The paper thanks David Wang for leading the SGLang integration and members of the Modal team for engineering support; the repository also thanks NVIDIA's Benjamin Chislett for bringing DFlash to vLLM.^[1][2] Z Lab's DFlash collection covers Qwen3, Qwen3.5 and Qwen3.6, Gemma 4, MiniMax M2.5 and M2.7, Kimi K2.5, K2.6 and K2.7-Code, gpt-oss, Llama 3.1, GLM 5.1 and NVIDIA Alpamayo targets, and its DFlash 2 collection adds Qwen3.8-27B and Muse Glimmer.^[2]

Other organizations have published their own DFlash drafters on Hugging Face. Inco's DFlash 2 post names several of them:^[4][17]

CoreWeave said on 18 June 2026 that its production Kimi K2.7-Code endpoint runs an NVFP4 checkpoint with a DFlash speculator it trained itself, served on vLLM by default.^[15] Xiaomi's MiMo-V2.6 technical report describes its speculative decoder as a multi-token prediction module "following the block diffusion design of DFlash." It also reports that RL rollouts used a block-6 DFlash drafter by default, raising average accepted length by 31.3 percent over the MTP-3 configuration inherited from supervised fine-tuning.^[20]

Inco reported more than 3.5 million Hugging Face downloads of DFlash models as of August 2026, and "more than 6 million" in its 17 September Splash post. These are the company's own counts.^[4][18]

Related methods

DFlash is one of several 2026 parallel drafters. DSpark (Xin Cheng and co-authors, arXiv, 6 July 2026) couples "a parallel backbone with a lightweight sequential module" to model dependencies inside the block, and adds confidence-scheduled verification that shortens the verified block for drafts likely to be rejected. Its abstract reports 60 to 85 percent faster per-user generation than an MTP-1 baseline in DeepSeek-V4 live serving.^[21] Domino (arXiv, 28 May 2026) likewise refines a parallel draft backbone with a lightweight causal head.^[22] Inco's DFlash 2 post presents its pairwise selector as a cheaper alternative to the sequential heads in these two methods.^[4] The block-diffusion technique DFlash borrows is discussed in discrete diffusion and diffusion language models.

People and organizations

All three paper authors are listed at UC San Diego, with Liu as corresponding author.^[1] NVIDIA's June 2026 post describes Liu as an assistant professor at UC San Diego who leads Z Lab, a co-founder of Inco AI, and an MIT PhD advised by Song Han.^[7] Inco's DFlash 2 post says "our team released DFlash in January." The z-lab/dflash README now links both the paper and the DFlash 2 post. Identical DFlash 2 drafter weights also sit under the z-lab organization on Hugging Face; Inco's model cards call those copies mirrors, although the z-lab repositories were created on 14 and 15 August 2026, before the incoai repositories on 18 August.^[4][2][17] The paper acknowledges research support from Qualcomm and Amazon, and compute from Modal, Yotta Labs, Eigen AI and InnoMatrix.^[1]

Limitations

• Target-specific drafters. A DFlash drafter reads a particular target's hidden layers and shares its embedding and output head, so each target model needs its own trained drafter.^[1]
• Gains shrink under load. Every published serving table shows speedups falling as concurrency rises, because verification competes with other requests for compute.^[1][5][6]
• Workload dependence. Math and code accept far more draft tokens than open-ended chat, in the paper and in Google's TPU work.^[1][8]
• Context length. Drafters trained at short context lose acceptance on long prompts unless fine-tuned for them.^[1]
• Self-reported numbers. Most published figures come from the authors, Inco, or partners who co-wrote the posts.^[1][4][7][8]

Timeline

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39Jian Chen, Yesheng Liang, and Zhijian Liu. "DFlash: Block Diffusion for Flash Speculative Decoding." arXiv:2602.06036, submitted February 5, 2026; v2 May 28, 2026 (ICML 2026 camera-ready). arxiv.org/...2602.06036
2. ^1 ^2 ^3 ^4 ^5 ^6z-lab. "dflash" GitHub repository, README and metadata (created January 4, 2026; MIT License). github.com/...dflash (accessed September 23, 2026)
3. ^Z Lab, UC San Diego. "DFlash: Block Diffusion for Flash Speculative Decoding" project page. z-lab.ai/...dflash (accessed September 23, 2026)
4. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17Inco AI. "DFlash 2: Keep Drafting Parallel." Inco AI blog, August 18, 2026. inco.ai/...dflash2
5. ^1 ^2 ^3Inco AI. "Qwen3.8-27B-DFlash2" model card. Hugging Face. huggingface.co/...Qwen3.8-27B-DFlash2 (accessed September 23, 2026)
6. ^1 ^2Inco AI. "Muse-Glimmer-30B-DFlash2" model card. Hugging Face. huggingface.co/...Muse-Glimmer-30B-DFlash2 (accessed September 23, 2026)
7. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11Amr Elmeleegy, Benjamin Chislett, Fernando Xiong, Michael Iovine, Omri Almog, Hao Zhang, and Zhijian Liu. "Boost Inference Performance up to 15x on NVIDIA Blackwell Using DFlash Speculative Decoding." NVIDIA Technical Blog, June 23, 2026. developer.nvidia.com/...flash-speculative-decoding
8. ^1 ^2 ^3 ^4 ^5 ^6Weiren Yu, Yarong Mu, Lihao Ran, Zhaoxiang Feng, Yiming Zhao, and Hao Zhang. "Supercharging LLM inference on Google TPUs: Achieving 3X speedups with diffusion-style speculative decoding." Google Developers Blog, May 4, 2026. developers.googleblog.com/...-speculative-decoding
9. ^1 ^2 ^3sgl-project/sglang. Pull request #22077, "[Feature] Add DFLASH speculative decoding support," merged April 7, 2026. github.com/...22077
10. ^1 ^2 ^3 ^4 ^5vLLM Project. Pull request #36847, "[Feat][Spec Decode] DFlash," merged March 30, 2026; release notes for v0.20.0 (April 27, 2026) and v0.28.0 (August 26, 2026). GitHub. github.com/...36847 github.com/...v0.20.0 github.com/...v0.28.0
11. ^1 ^2 ^3ggml-org/llama.cpp. Pull request #22105, "[Speculative decoding] feat: add DFlash support," merged June 28, 2026. github.com/...22105
12. ^1 ^2sgl-project/sglang. Pull request #35371, "[Spec] DFlash2: local convolution + candidate selector," merged August 19, 2026. github.com/...35371
13. ^vllm-project/vllm. Pull request #52816, "[Spec Decode] DFlash2: local convolution + candidate selector," merged August 21, 2026. github.com/...52816
14. ^1 ^2ggml-org/llama.cpp. Pull request #27342, "spec : add DFlash2 support (local convolution + candidate selector)," merged August 27, 2026; and ollama/ollama pull request #17865, "mlx: add DFlash2 support," opened August 19, 2026 (open as of September 23, 2026). github.com/...27342 github.com/...17865
15. ^1 ^2CoreWeave. "Kimi K2.7 Code Now Available on Serverless Inference with Leading Benchmark Price-Performance." CoreWeave blog, June 18, 2026. coreweave.com/...ading-benchmark-price-performance
16. ^LM Studio. "LM Studio 0.4.22" release notes, August 28, 2026. lmstudio.ai/...lmstudio-v0.4.22
17. ^1 ^2 ^3Hugging Face. Model API records for the z-lab organization and the third-party DFlash repositories listed (creation dates and licenses). huggingface.co/z-lab (accessed September 23, 2026)
18. ^1 ^2 ^3 ^4Inco AI. "Splash: A Local Engine Built Around the Model." Inco AI blog, dated September 17, 2026. inco.ai/...splash
19. ^1 ^2Hugging Face. incoai organization model listing (licenses and creation dates). huggingface.co/incoai (accessed September 23, 2026)
20. ^LLM-Core Xiaomi. "MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement." Technical report, September 2026. huggingface.co/...MiMo_V2_6_technical_report.pdf
21. ^1 ^2Xin Cheng, Xingkai Yu, Chenze Shao, and co-authors. "DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation." arXiv:2607.05147, July 6, 2026. arxiv.org/...2607.05147
22. ^Jianuo Huang, Yaojie Zhang, Qituan Zhang, Hao Lin, Hanlin Xu, and Linfeng Zhang. "Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding." arXiv:2605.29707, May 28, 2026. arxiv.org/...2605.29707