Qwen3.8-Flash-Next

Qwen3.8-Flash-Next is an experimental open-weight multimodal large language model released by Alibaba Group's Qwen team on August 26, 2026. Qwen describes it as an early preview of architectural ideas intended for a later Qwen4 family. It is not itself a Qwen4 release, and its public name remains Qwen3.8-Flash-Next even though the immutable checkpoint configuration uses the internal architecture identifier qwen4_exp.^[1][2][4][5]

The model combines Gated DeltaNet layers with Qwen Sparse Attention (QSA), a four-branch Gated Residual pathway, and a large N-gram Embedding table. Qwen reports a 125-billion-parameter main model with 6 billion parameters active per token, plus 51 billion parameters in the N-gram table and a 4-billion-parameter multi-token-prediction module. The native context window is 262,144 tokens, with a documented YaRN-based extension path to 1,000,000 tokens.^[3][4][5]

Release and identity

Qwen released the weights and technical report on August 26, 2026. The GitHub repository says Flash-Next plays a role similar to Qwen3-Next, which previewed architecture later used in Qwen3.5 and subsequent Qwen3.x models. This framing makes Flash-Next a research and engineering preview, not a renamed Qwen4 model or evidence that every described component will appear unchanged in a final Qwen4 release.^[1][2][3]

The Hugging Face model card distinguishes Flash-Next from the hosted Qwen3.8-Flash service. It says Qwen3.8-Flash is an official version based on Flash-Next with additional production features, including a default one-million-token context and built-in tools. Those service features should not be attributed automatically to the downloadable Flash-Next checkpoint.^[4]

Model structure and parameter accounting

Backbone and expert routing

The language backbone has 48 layers with a hidden size of 2,560. Each layer has a mixture-of-experts feed-forward block containing 512 routed experts and one shared expert. The router selects ten routed experts for each token in addition to the shared expert, with an intermediate dimension of 640 for routed and shared experts.^[4][5]

Qwen's 125B-A6B label describes the main model: 125 billion parameters are in the backbone and about 6 billion are activated for a token. The release separately lists a 51-billion-parameter N-gram Embedding table and a 4-billion-parameter multi-token-prediction (MTP) module. NVIDIA's checkpoint analysis gives the embedding table a more exact size of 51.2 billion parameters, from 320,001,536 padded rows of width 160. These separately identified components should not be hidden inside the 125B headline. The 6B active figure also does not mean that the complete checkpoint can be stored as a six-billion-parameter model.^[4][8]

The MTP module contains one layer and supports multi-step prediction for speculative decoding. The checkpoint configuration records a hybrid MTP layer with sparse attention. Qwen's report says the MTP path can reuse QSA selections across draft steps. NVIDIA AutoModel's current supervised-fine-tuning path does not load the 4B MTP head, so its documented training support does not cover that component.^[3][5][8]

Hybrid token mixing and QSA

The 48-layer decoder repeats a four-layer pattern: three Gated DeltaNet layers followed by one QSA layer. This yields 36 recurrent linear-attention layers and 12 sparse-attention layers. Gated DeltaNet compresses earlier context into a fixed-size state, while QSA retains direct access to selected earlier token groups.^[3][4][5]

QSA uses a lightweight indexer that first compresses keys in groups of four tokens. For each query it selects up to 512 complete micro-blocks, corresponding to a 2,048-token budget, and includes the unfinished causal tail. The main grouped-query attention then runs over those selected token positions. The released indexer has four query heads, one shared key head, and a head dimension of 128.^[3][4][5]

Qwen introduced QSA during continued pre-training in two stages. The report describes a 1,000-step indexer warm-up using about 2 billion tokens, followed by joint sparse training of the backbone and indexer for 8,000 steps using roughly 200 billion tokens. These figures describe the QSA conversion stages, not the entire model's pre-training corpus. Qwen did not disclose a complete data inventory, provenance analysis, or exact all-stage training-token total.^[3]

In kernel tests reported by Qwen at a one-million-token sequence length, QSA was 7.6 times faster than dense attention for prefill and 4.9 times faster for decoding. These are vendor measurements of the attention kernels under the report's setup, not universal end-to-end serving speedups. Application throughput also depends on hardware, batching, expert communication, the vision path, MTP, and runtime implementation.^[3]

Gated Residual

Gated Residual widens the residual state into four persistent branches. Each sublayer reads through a data-dependent elementwise gate and writes back through learned branch weights. The checkpoint uses a bottleneck rank of 320 for the gating projections. Qwen's report compares the design with conventional pre-normalization and Hyper-Connection variants at smaller experimental scales, then reports stress tests intended to examine stability.^[3][4][5]

Those experiments are internal ablations rather than independent validation. In one 25B-A3B experiment trained for 560 billion tokens, Qwen reported a benchmark average of 54.66 for Gated Residual, compared with 50.91 for pre-normalization and 54.47 for dynamic manifold-constrained Hyper-Connections. The result supports Qwen's design choice under that experiment; it does not isolate the effect in the final post-trained Flash-Next model.^[3]

N-gram Embedding

The N-gram Embedding, also called Engram or ple in implementation documentation, appears at decoder layer 2. It hashes local bigram and trigram contexts into a large lookup table and adds the retrieved values to the token representation. Qwen designed the table for host-memory storage with asynchronous prefetching, adding capacity without placing the whole table in accelerator memory.^[3][4][8]

Qwen's ablations found that increasing the N-gram vocabulary reduced language-model loss monotonically while downstream benchmark performance saturated or varied. The report therefore does not support a general claim that a larger table always improves task accuracy. Offloading also does not make lookup free: memory capacity, host-to-device transfers, sharding, and communication remain deployment concerns.^[3]

Muon and AdamW optimization

Qwen used the Muon optimizer for two-dimensional weights that act as linear maps, including attention projections, Gated DeltaNet projections, and expert feed-forward matrices. Input and N-gram embeddings, the output head, MoE router, and low-rank Gated Residual projections remained on AdamW; the N-gram table used Adam without weight decay. The team used eight Newton-Schulz steps for Muon orthogonalization and split fused matrices before applying it so unrelated sub-operators were not orthogonalized together.^[3]

The technical report says Qwen refitted its scaling law for the new architecture, selected a larger learning rate and batch size, and omitted batch-size warm-up. Its stability tests used smaller models and deliberately elevated learning rates, while the production-run comparison covered the first 276 billion consumed tokens. These tests document Qwen's training choices. They do not establish that the same optimizer split is optimal for other models, datasets, or hardware.^[3]

Multimodal interface and context

The released checkpoint accepts text, image, and video inputs and generates text. Its vision encoder has 27 layers, a hidden size of 1,152, 16 attention heads, 16-pixel spatial patches, and two-frame temporal patches. The configuration includes distinct image and video token identifiers, and the Hugging Face pipeline is image-text-to-text.^[4][5]

The configuration's native maximum position setting is 262,144 tokens. Qwen documents an extension to one million tokens through static YaRN overrides in vLLM, SGLang, and TokenSpeed. The model card warns that a constant YaRN scaling factor can affect performance on shorter inputs, so the extended setting is optional rather than the checkpoint default. A configured limit also does not prove uniform retrieval or reasoning quality throughout the whole window.^[4][5]

Checkpoints and runtime support

Qwen released BF16 and FP8 checkpoints in sharded Safetensors format under Qwen/Qwen3.8-Flash-Next and Qwen/Qwen3.8-Flash-Next-FP8. The FP8 repository identifies the BF16 release as its base model. Qwen's official repository documents Transformers Serve, SGLang, vLLM, TokenSpeed, llama.cpp, MLX-VLM, and Unsloth paths. Support levels differ: the repository describes text and vision support for llama.cpp and MLX-VLM, while the main serving recipes expose OpenAI-compatible APIs.^[2][4][7]

NVIDIA NeMo AutoModel added a separate language-only supervised-fine-tuning implementation. NVIDIA validated a 100-step run on 64 H100 GPUs and a context-parallel comparison on 128 H100 GPUs. Its current scope excludes vision and video training, MTP, tensor parallelism, pipeline parallelism, and left or interior padding. QSA indexers are frozen in that implementation because the released inference reference does not expose the auxiliary loss or a straight-through gradient path required to train their discrete selections.^[8]

NVIDIA's general NeMo RL model-support page does not list Qwen3.8-Flash-Next as a dedicated supported model. Its broad Hugging Face route is described for models under 70 billion parameters and sequences up to 32,000 tokens, which does not match this checkpoint's 125B backbone. The AutoModel SFT recipe therefore should not be presented as documented NeMo RL reinforcement-learning support.^[9]

Evaluation

Qwen's technical report evaluates the base model on 14 pre-training benchmarks. It reports that Qwen3.8-Flash-Next-Base exceeded Qwen3.7-Plus-Base on eight benchmarks and trailed it by no more than 2.6 points on the other six, while using about one-third as many activated parameters, about one-third as many training tokens, and roughly one-ninth the training FLOPs. The report does not disclose the absolute full-run token count needed to independently verify the token and FLOP ratios.^[3]

The post-trained model card reports 58.7 on DeepSWE 1.1, 62.5 on SWE-bench Pro, 81.0 on SWE-bench Multilingual, and 73.9 on Qwen's internal CoWorkBench. Qwen says DeepSWE used Claude Code and mini-SWE-agent harnesses and reports the higher of the two results. SWE-bench Pro used Claude Code for most models after Qwen corrected tasks and reevaluated baselines; SWE-bench Multilingual used mini-SWE-agent. CoWorkBench is an in-house suite whose full task set is not supplied in the model card. These results are vendor-reported and should not be treated as independent rankings.^[4]

The technical report's QSA comparisons and long-context tests are also Qwen-run ablations. It reported a QSA average of 76.8 versus 75.9 for full attention across eight short-context benchmarks, and higher QSA scores than its full-attention baseline on RULER and MRCR at several long lengths. These measurements support the architecture discussion but do not establish universal efficiency or accuracy across other prompts, runtimes, and deployments.^[3]

License and limitations

The weights, configuration, inference code, and associated documentation are released under Qwen Community License 1.0, not Apache 2.0 or MIT. The license broadly permits use, modification, redistribution, hosting, fine-tuning, and derivative works, but it imposes conditions. Commercial products above either 100 million monthly active users or US$20 million in monthly revenue must display the model name prominently. Commercial model-as-a-service and AI work-assistant businesses must obtain a separate Qwen license, apart from qualifying internal use.^[6]

The release is open weight, but the custom license and incomplete training disclosure make "fully open source" an imprecise description of the whole project. Qwen did not publish the training data, full data pipeline, complete training code, optimizer states, all-run compute record, independent safety audit, red-team report, or model-specific system card. The report emphasizes architecture and training efficiency rather than production safety. The model card also warns that framework performance varies, static YaRN can affect shorter contexts, and long-video use requires preprocessing changes. Together with the explicit experimental-preview status, these limits rule out assuming production readiness from the release alone.^[2][3][4][6]

References

1. ^Qwen Team. "Qwen3.8-Flash-Next: A New Architecture, Towards Ultimate Cost-Efficiency." August 26, 2026. qwen.ai/blog
2. ^Qwen Team. "Qwen3.8-Flash-Next." GitHub repository, accessed August 28, 2026. github.com/...Qwen3.8-Flash-Next
3. ^Qwen Team. "On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability." Technical report, August 26, 2026. github.com/...tech_report.pdf
4. ^Qwen Team. "Qwen/Qwen3.8-Flash-Next." Hugging Face model card and BF16 checkpoint, accessed August 28, 2026. huggingface.co/...Qwen3.8-Flash-Next
5. ^Qwen Team. "Qwen3.8-Flash-Next config.json." Hugging Face checkpoint configuration, accessed August 28, 2026. huggingface.co/...config.json
6. ^Qwen. "Qwen Community License 1.0." 2026. huggingface.co/...LICENSE
7. ^Qwen Team. "Qwen/Qwen3.8-Flash-Next-FP8." Hugging Face FP8 checkpoint, accessed August 28, 2026. huggingface.co/...Qwen3.8-Flash-Next-FP8
8. ^NVIDIA. "Qwen3.8-Flash-Next." NeMo AutoModel model coverage, accessed August 28, 2026. docs.nvidia.com/...qwen3-8-flash-next
9. ^NVIDIA. "Model Support." NeMo RL documentation, accessed August 28, 2026. docs.nvidia.com/...model-support