Hy4 Preview

Hy4 Preview is an open-weight large language model released by the Tencent Hy Team on August 28, 2026. It is a text-generating mixture-of-experts model with a 770-billion-parameter backbone that activates 49 billion parameters for each token. Tencent released BF16 and FP8 checkpoints and described the model as an early version of the Hy4 family rather than a finished production release.^[1][2][4][5]

The model combines Gated DeepSeek Sparse Attention with IndexCache and identity Hyper-Connections. Its published configuration supports a 1,048,576-token context window, tool calling, and two response modes. The released weights and repository are under the Apache License 2.0, but Tencent did not publish the training dataset, a full training recipe, or a Hy4-specific technical report with the launch.^[2][3][6]

Release and status

Tencent announced Hy4 Preview on August 28, 2026 and made checkpoints available through Hugging Face, ModelScope, GitCode, and CNB. Tencent also offered hosted access through WorkBuddy, CodeBuddy, Yuanbao, ima, Tencent Cloud TokenHub, and OpenRouter. The company said that WorkBuddy and CodeBuddy access would be free for two weeks after launch. These distribution terms concern hosted products and do not change the checkpoint license.^[1][2]

The word "Preview" is part of the release name and describes its maturity. Tencent calls it an early Hy4 version with room for additional pre-training and post-training. The repository lists excessive reasoning time on complex tasks and a tendency to over-verify work as known issues. Tencent said later Hy4 models would follow, but it did not give a date or specifications for a final Hy4 release.^[1][2]

Architecture

Backbone and expert routing

The backbone has 78 transformer layers and a hidden size of 6,144. The first layer uses a dense feed-forward network with an intermediate size of 18,432. Each of the other 77 layers uses an MoE feed-forward block with 256 routed experts and one shared expert. The router selects eight routed experts per token in addition to the shared expert, and each expert has an intermediate size of 2,048.^[2][3]

Tencent's headline figures of 770 billion total parameters and 49 billion active parameters apply only to this backbone. The model also has one native multi-token-prediction (MTP) layer with 10 billion total parameters and 0.7 billion active parameters. The published specification table explicitly excludes that layer. Accordingly, the 770B/49B label should not be treated as an all-components count. The MTP layer is used for speculative decoding, in which draft tokens can be checked by the main model to improve generation throughput.^[2][3]

Gated sparse attention and IndexCache

Every backbone layer is configured for DeepSeek Sparse Attention (DSA), a form of sparse attention that uses an indexer to select a limited set of earlier tokens for the more expensive attention calculation. The original DeepSeek-V3.2 report describes DSA as reducing the core attention calculation from quadratic attention over the whole sequence to attention over the selected positions. Hy4 adds elementwise gating and uses 32 indexer heads with a head dimension of 128 and an indexer top-k of 2,048.^[2][3][7]

The configuration alternates full indexer layers with shared-index layers. This implements IndexCache, which reuses sparse token selections across nearby layers instead of recomputing an index at every layer. The IndexCache paper reports speedups on other experimental models, including a 30-billion-parameter DSA model, but those paper results are not measurements of Hy4 Preview. Tencent did not publish an ablation that isolates the quality or speed contribution of IndexCache in Hy4.^[3][8]

Hy4 uses multi-head latent attention with a query compression rank of 2,048 and a key-value compression rank of 512. Its attention configuration has 64 attention heads and eight key-value heads. These configuration values describe the released checkpoint; they do not establish how reliably the model uses all parts of a million-token prompt.^[3]

Identity Hyper-Connections

The residual pathway uses identity Hyper-Connections (iHC) with four residual streams. Hyper-Connections generalize a single residual stream into multiple streams with connections across depth and width. The original Hyper-Connections paper evaluated dense and sparse language models at much smaller scales and described identity-initialized and dynamic variants. Hy4's configuration enables iHC with a multiplier of four, but Tencent did not release a Hy4 ablation comparing it with an otherwise identical conventional residual architecture.^[2][3][9]

Checkpoints, modes, and deployment

Tencent published a BF16 checkpoint as tencent/Hy4-preview and a separately quantized FP8 checkpoint as tencent/Hy4-preview-FP8. Both repositories use sharded Safetensors files and identify the architecture as HYV4ForCausalLM. The BF16 configuration specifies a vocabulary of 120,832 tokens and a maximum position setting of 1,048,576. The FP8 repository is a derived deployment format, not a separate trained model.^[3][4][5]

The chat template supports high and no_think reasoning settings. The repository recommends temperature=0.9 and top_p=1.0, with high as the default. These are vendor-recommended inference settings rather than guarantees that one mode is best for every task.^[2]

Official serving recipes cover vLLM and SGLang. The vLLM recipe uses the FP8 checkpoint, the FLASHMLA_SPARSE attention backend, Hy4-specific reasoning and tool parsers, and MTP speculative decoding. The SGLang recipe likewise uses the FP8 checkpoint and a NEXTN speculative configuration. Tencent supplied prebuilt container tags for both runtimes. Repository examples demonstrate an OpenAI-compatible chat-completions interface after a server is running.^[2]

Tencent also published supervised fine-tuning scripts for DeepSpeed, LLaMA-Factory, and ms-swift. Its tested minimums are substantial: the guide lists 64 accelerators with at least 96 GB of memory each for LoRA fine-tuning and 128 such accelerators for full fine-tuning, with at least 2 TB of CPU memory per machine. Those figures are Tencent's tested minima and vary with sequence length and batch size. They are not general minimums for inference.^[2]

Evaluation

Tencent published a large benchmark appendix covering agentic coding, search, office work, scientific tasks, and reasoning. The table mixes public benchmarks, Tencent internal suites, official scores for some comparison models, and Tencent-run comparison scores marked with an asterisk. Tencent says models were evaluated at their highest available reasoning setting. Harnesses, turn budgets, timeouts, judges, and resource limits differ by benchmark, so the rows should not be combined into a single universal ranking.^[2]

Among the vendor-reported public results, Hy4 Preview scored 82.9 on SWE-bench Multilingual, 65.7 on SWE-bench Pro, 85.4 on Terminal-Bench 2.1, and 92.3 on GPQA Diamond. Tencent's appendix says SWE-bench Multilingual and SWE-bench Pro used the swe-agent scaffold, while Terminal-Bench used the Claude Code harness with up to 500 turns and a 12-hour timeout. These numbers have not been independently reproduced for this article.^[2]

Tencent separately ran a blind side-by-side study in which 163 internal experts rated outputs on 203 engineering tasks. It reported an average score of 2.99 for Hy4 Preview, compared with 2.92 for GLM 5.3 and 2.94 for Kimi K3. Against GLM 5.3, Tencent reported 46.8 percent wins, 12.8 percent ties, and 40.4 percent losses. Against Kimi K3, it reported 51.2 percent wins, 7.9 percent ties, and 40.9 percent losses. Because Tencent selected the tasks, operated the study, and did not release the individual ratings, this evidence is a vendor evaluation rather than independent validation.^[2]

Tencent also reported that Hy4 Preview analyzed its inference system and produced changes that improved end-to-end throughput by 31.8 percent over an internal baseline. The announcement does not provide enough code, logs, hardware details, or baseline configuration for outside reproduction, so the result should not be read as a general inference-speed claim.^[1]

License and evidence limits

The code and released checkpoint repositories state an Apache 2.0 license. That license permits use, modification, and redistribution subject to its terms. It is accurate to describe the published weights and repository materials as openly licensed. It is broader than the available evidence to call the entire model-development process fully open source, because Tencent did not release the training corpus, data-provenance record, complete pre-training code, optimizer state, training compute, or a Hy4-specific technical report.^[2][4][6]

The release materials do not include an independently audited safety evaluation, model-specific system card, red-team report, documented knowledge cutoff, or detailed training-data disclosure. The benchmark appendix measures selected tasks under specified harnesses; it does not establish production reliability, safety in high-stakes use, or dependable recall across the full configured context length. The preview label and Tencent's own known-issues section further limit claims about readiness.^[1][2]

References

1. ^Tencent. "Tencent Releases and Open-Sources Tencent Hy4 preview." August 28, 2026. tencent.com/...nd-open-sources-tencent-hy4-preview
2. ^Tencent Hy Team. "Hy4-preview." GitHub repository and model documentation, accessed August 28, 2026. github.com/...Hy4-preview
3. ^Tencent. "Hy4-preview config.json." Hugging Face checkpoint configuration, accessed August 28, 2026. huggingface.co/...config.json
4. ^Tencent. "tencent/Hy4-preview." Hugging Face model card and BF16 checkpoint, accessed August 28, 2026. huggingface.co/...Hy4-preview
5. ^Tencent. "tencent/Hy4-preview-FP8." Hugging Face model card and FP8 checkpoint, accessed August 28, 2026. huggingface.co/...Hy4-preview-FP8
6. ^Tencent Hy Team. "Hy4-preview LICENSE." GitHub, accessed August 28, 2026. github.com/...LICENSE
7. ^DeepSeek-AI et al. "DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models." arXiv:2512.02556, December 2025. arxiv.org/...2512.02556
8. ^Yushi Bai et al. "IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse." arXiv:2603.12201, March 2026. arxiv.org/...2603.12201
9. ^Defa Zhu et al. "Hyper-Connections." arXiv:2409.19606, September 2024. arxiv.org/...2409.19606