Hy4 Preview

Hy4 Preview is an open-weight large language model released by the Tencent Hy Team on August 28, 2026. It is a text-generating mixture-of-experts model with a 770-billion-parameter backbone that activates 49 billion parameters for each token. Tencent released BF16 and FP8 checkpoints and described the model as an early version of the Hy4 family rather than a finished production release.^[1][2][4][5]

The model combines Gated DeepSeek Sparse Attention with IndexCache and identity Hyper-Connections. Its published configuration supports a 1,048,576-token context window, tool calling, and two response modes. The released weights and repository are under the Apache License 2.0, but Tencent did not publish the training dataset, a full training recipe, or a Hy4-specific technical report with the launch.^[2][3][6]

Release and status

Tencent announced Hy4 Preview on August 28, 2026 and made checkpoints available through Hugging Face, ModelScope, GitCode, and CNB. Tencent also offered hosted access through WorkBuddy, CodeBuddy, Yuanbao, ima, Tencent Cloud TokenHub, and OpenRouter. The company said that WorkBuddy and CodeBuddy access would be free for two weeks after launch. These distribution terms concern hosted products and do not change the checkpoint license.^[1][2]

At launch, Tencent's research page listed API rates of USD 0.834 per million input tokens, USD 2.501 per million output tokens, and USD 0.042 per million cached input tokens.^[10] Tencent Cloud's China-site TokenHub rate card separately listed CNY 6, CNY 18, and CNY 0.30 for the same three billing meters.^[11] TokenHub lists the model ID as hy4-preview and documents a 1M-token context window, a 960k maximum input, and a 64k maximum output.^[12] Its protocol matrix marks OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages as natively supported for this model.^[13] Tencent documents Guangzhou and Singapore as separate access regions with different base URLs and non-interchangeable API keys.^[13] These are launch-time hosted-service terms, not properties of the Apache-licensed checkpoints, and Tencent can change them.

The word "Preview" is part of the release name and describes its maturity. Tencent calls it an early Hy4 version with room for additional pre-training and post-training. The repository lists excessive reasoning time on complex tasks and a tendency to over-verify work as known issues. Tencent said later Hy4 models would follow, but it did not give a date or specifications for a final Hy4 release.^[1][2]

Architecture

Backbone and expert routing

The backbone has 78 transformer layers and a hidden size of 6,144. The first layer uses a dense feed-forward network with an intermediate size of 18,432. Each of the other 77 layers uses an MoE feed-forward block with 256 routed experts and one shared expert. The router selects eight routed experts per token in addition to the shared expert, and each expert has an intermediate size of 2,048.^[2][3]

Tencent's headline figures of 770 billion total parameters and 49 billion active parameters apply only to this backbone. The model also has one native multi-token-prediction (MTP) layer with 10 billion total parameters and 0.7 billion active parameters. The published specification table explicitly excludes that layer. Accordingly, the 770B/49B label should not be treated as an all-components count. The MTP layer is used for speculative decoding, in which draft tokens can be checked by the main model to improve generation throughput.^[2][3]

Gated sparse attention and IndexCache

Every backbone layer is configured for DeepSeek Sparse Attention (DSA), a form of sparse attention that uses an indexer to select a limited set of earlier tokens for the more expensive attention calculation. The original DeepSeek-V3.2 report describes DSA as reducing the core attention calculation from quadratic attention over the whole sequence to attention over the selected positions. Hy4 adds elementwise gating and uses 32 indexer heads with a head dimension of 128 and an indexer top-k of 2,048.^[2][3][7]

The configuration alternates full indexer layers with shared-index layers. This implements IndexCache, which reuses sparse token selections across nearby layers instead of recomputing an index at every layer. The IndexCache paper reports speedups on other experimental models, including a 30-billion-parameter DSA model, but those paper results are not measurements of Hy4 Preview. Tencent did not publish an ablation that isolates the quality or speed contribution of IndexCache in Hy4.^[3][8]

Hy4 uses multi-head latent attention with a query compression rank of 2,048 and a key-value compression rank of 512. Its attention configuration has 64 attention heads and eight key-value heads. These configuration values describe the released checkpoint; they do not establish how reliably the model uses all parts of a million-token prompt.^[3]

Identity Hyper-Connections

The residual pathway uses identity Hyper-Connections (iHC) with four residual streams. Hyper-Connections generalize a single residual stream into multiple streams with connections across depth and width. The original Hyper-Connections paper evaluated dense and sparse language models at much smaller scales and described identity-initialized and dynamic variants. Hy4's configuration enables iHC with a multiplier of four, but Tencent did not release a Hy4 ablation comparing it with an otherwise identical conventional residual architecture.^[2][3][9]

Checkpoints, modes, and deployment

Tencent published a BF16 checkpoint as tencent/Hy4-preview and a separately quantized FP8 checkpoint as tencent/Hy4-preview-FP8. Both repositories use sharded Safetensors files and identify the architecture as HYV4ForCausalLM. The BF16 configuration specifies a vocabulary of 120,832 tokens and a maximum position setting of 1,048,576. The FP8 repository is a derived deployment format, not a separate trained model.^[3][4][5]

The chat template supports high and no_think reasoning settings. The repository recommends temperature=0.9 and top_p=1.0, with high as the default. These are vendor-recommended inference settings rather than guarantees that one mode is best for every task.^[2]

Official serving recipes cover vLLM and SGLang. The vLLM recipe uses the FP8 checkpoint, the FLASHMLA_SPARSE attention backend, Hy4-specific reasoning and tool parsers, and MTP speculative decoding. The SGLang recipe likewise uses the FP8 checkpoint and a NEXTN speculative configuration. Tencent supplied prebuilt container tags for both runtimes. Repository examples demonstrate an OpenAI-compatible chat-completions interface after a server is running.^[2]

Tencent also published supervised fine-tuning scripts for DeepSpeed, LLaMA-Factory, and ms-swift. Its tested minimums are substantial: the guide lists 64 accelerators with at least 96 GB of memory each for LoRA fine-tuning and 128 such accelerators for full fine-tuning, with at least 2 TB of CPU memory per machine. Those figures are Tencent's tested minima and vary with sequence length and batch size. They are not general minimums for inference.^[2]

AngelSlim GGUF quantizations

On September 1, 2026, Tencent's AngelSlim team published three GGUF conversions of Hy4 Preview. GGUF is a deployment format rather than a separately trained model, and these files are quantized derivatives of the original checkpoint. Their size figures use binary gibibytes, while bits per weight, or bpw, is an average over the complete file rather than a label for every tensor.^[14][15]

The Q4_K_M build keeps most tensors in Q4_K and raises the expert down projections to Q6_K in 37 layers under llama.cpp's selection logic. The UD-IQ1_M build applies a mixture of 1.75-bpw IQ1_M and 2.0625-bpw IQ2_XXS to the routed-expert gate and up projections. These are mixed-precision files, so neither the Q4 nor IQ name describes every tensor in the artifact.^[14]

MIX-STQ1_0 is likewise not a uniform 1.25-bit model. Its file is named Hy4-preview-STQ1_0.gguf, but the model card calls the overall allocation strategy MIX-STQ1_0. The routed-expert gate and up projections use 1.3125-bpw STQ1_0 on 29 layers and 2.0625-bpw IQ2_XXS on the other 48. Expert down projections use IQ3_XXS, with IQ4_XS in the final three layers. Selected attention tensors remain at Q5_K or Q8_0, while the DSA indexer mixes Q8_0 and F32 and several router, normalization, iHC, sink, and output tensors stay in F32. AngelSlim says the three routed-expert tensor families account for 97.7 percent of the model's parameters, allowing most storage savings to come from those families while smaller or more sensitive groups retain higher precision.^[14]

STQ1_0 format and Hy4 encoding

STQ1_0 comes from the Sherry line of ternary quantization work. In the format, a weight takes one of three scaled values: negative, zero, or positive. Each group of four has exactly one zero and three signed nonzero values, a 3:4 sparsity rule. The 32 possible sign and zero-position patterns fit in five bits per four weights, and an FP16 scale is stored for each 256-weight block. Including that scale, the format occupies 42 bytes per 256 weights, or 1.3125 bpw.^[17][18]

The Sherry paper combines this format with quantization-aware training and an annealed residual mechanism called Arenas. The Hy4 GGUF card describes a different conversion path: post-training quantization that preserves the STQ1_0 byte format but chooses the scale by weighted least squares and places zeros using an importance matrix. Re-quantizing the Hy4 STQ file with the supplied recipe therefore requires an importance matrix. The public material does not establish that Hy4 Preview was trained with Sherry's Arenas procedure.^[14][17]

Runtime compatibility

None of the three GGUF files runs with an unmodified upstream llama.cpp build as of September 3, 2026. The hyv4 model architecture had not been added upstream, so Q4_K_M, UD-IQ1_M, and MIX-STQ1_0 all require AngelSlim's 0001-hyv4-architecture.patch. The STQ1_0 file additionally requires 0002-stq1_0-quant-and-cuda.patch. AngelSlim's instructions pin llama.cpp commit 0cea36222, require --jinja for the Hy4 chat template, and advise keeping mmap-backed weights on local storage.^[14]

The related llama.cpp pull request 22836 adds the STQ1_0 format and optimized CPU kernels, not the separate Hy4 architecture implementation. The pull request was still open and unmerged on September 3, 2026. Its existence therefore does not make STQ1_0 or hyv4 available in stock llama.cpp.^[18]

AngelSlim also published an advanced distributed deployment preview for the STQ1_0 file. Its reference setup divides the model between two Linux machines, one with four 16 GB RTX A4000 cards and another with a 16 GB RTX 4090 Laptop GPU. The downloadable runtime packages contain compiled executables and libraries, and the guide states that the required MoE support is not included in the current public open-source edition. This is a hardware-specific preview, not evidence of general single-laptop support or a fully upstream, source-available runtime path.^[16]

Evaluation

Tencent published a large benchmark appendix covering agentic coding, search, office work, scientific tasks, and reasoning. The table mixes public benchmarks, Tencent internal suites, official scores for some comparison models, and Tencent-run comparison scores marked with an asterisk. Tencent says models were evaluated at their highest available reasoning setting. Harnesses, turn budgets, timeouts, judges, and resource limits differ by benchmark, so the rows should not be combined into a single universal ranking.^[2]

Among the vendor-reported public results, Hy4 Preview scored 82.9 on SWE-bench Multilingual, 65.7 on SWE-bench Pro, 85.4 on Terminal-Bench 2.1, and 92.3 on GPQA Diamond. Tencent's appendix says SWE-bench Multilingual and SWE-bench Pro used the swe-agent scaffold, while Terminal-Bench used the Claude Code harness with up to 500 turns and a 12-hour timeout. These numbers have not been independently reproduced for this article.^[2]

Tencent separately ran a blind side-by-side study in which 163 internal experts rated outputs on 203 engineering tasks. It reported an average score of 2.99 for Hy4 Preview, compared with 2.92 for GLM 5.3 and 2.94 for Kimi K3. Against GLM 5.3, Tencent reported 46.8 percent wins, 12.8 percent ties, and 40.4 percent losses. Against Kimi K3, it reported 51.2 percent wins, 7.9 percent ties, and 40.9 percent losses. Because Tencent selected the tasks, operated the study, and did not release the individual ratings, this evidence is a vendor evaluation rather than independent validation.^[2]

Tencent also reported that Hy4 Preview analyzed its inference system and produced changes that improved end-to-end throughput by 31.8 percent over an internal baseline. The announcement does not provide enough code, logs, hardware details, or baseline configuration for outside reproduction, so the result should not be read as a general inference-speed claim.^[1]

License and evidence limits

The code and released checkpoint repositories state an Apache 2.0 license. That license permits use, modification, and redistribution subject to its terms. It is accurate to describe the published weights and repository materials as openly licensed. It is broader than the available evidence to call the entire model-development process fully open source, because Tencent did not release the training corpus, data-provenance record, complete pre-training code, optimizer state, training compute, or a Hy4-specific technical report.^[2][4][6]

The release materials do not include an independently audited safety evaluation, model-specific system card, red-team report, documented knowledge cutoff, or detailed training-data disclosure. The benchmark appendix measures selected tasks under specified harnesses; it does not establish production reliability, safety in high-stakes use, or dependable recall across the full configured context length. The preview label and Tencent's own known-issues section further limit claims about readiness.^[1][2]

References

1. ^1 ^2 ^3 ^4 ^5Tencent. "Tencent Releases and Open-Sources Tencent Hy4 preview." August 28, 2026. tencent.com/...nd-open-sources-tencent-hy4-preview
2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16Tencent Hy Team. "Hy4-preview." GitHub repository and model documentation, accessed August 28, 2026. github.com/...Hy4-preview
3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Tencent. "Hy4-preview config.json." Hugging Face checkpoint configuration, accessed August 28, 2026. huggingface.co/...config.json
4. ^1 ^2 ^3Tencent. "tencent/Hy4-preview." Hugging Face model card and BF16 checkpoint, accessed August 28, 2026. huggingface.co/...Hy4-preview
5. ^1 ^2Tencent. "tencent/Hy4-preview-FP8." Hugging Face model card and FP8 checkpoint, accessed August 28, 2026. huggingface.co/...Hy4-preview-FP8
6. ^1 ^2Tencent Hy Team. "Hy4-preview LICENSE." GitHub, accessed August 28, 2026. github.com/...LICENSE
7. ^DeepSeek-AI et al. "DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models." arXiv:2512.02556, December 2025. arxiv.org/...2512.02556
8. ^Yushi Bai et al. "IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse." arXiv:2603.12201, March 2026. arxiv.org/...2603.12201
9. ^Defa Zhu et al. "Hyper-Connections." arXiv:2409.19606, September 2024. arxiv.org/...2409.19606
10. ^Tencent Hy. "Introducing Hy4 preview." August 28, 2026. hy.tencent.ai/...hy4-preview
11. ^Tencent Cloud. "TokenHub Model Pricing." Updated August 28, 2026. cloud.tencent.com/...130055
12. ^Tencent Cloud. "TokenHub Model List." Updated August 28, 2026. intl.cloud.tencent.com/...78934
13. ^1 ^2Tencent Cloud. "TokenHub Language Model API Overview." Updated August 28, 2026. cloud.tencent.com/...130079
14. ^1 ^2 ^3 ^4 ^5AngelSlim. "Hy4-preview-GGUF." Hugging Face model card and files, accessed September 3, 2026. huggingface.co/...Hy4-preview-GGUF
15. ^Tencent. "AngelSlim." GitHub repository, accessed September 3, 2026. github.com/...AngelSlim
16. ^Tencent. "Hy4 Preview GGUF deployment guideline." GitHub, accessed September 3, 2026. github.com/...hy4_preview_gguf_guideline.md
17. ^1 ^2Hong Huang et al. "Sherry: Hardware-Efficient 1.25-Bit Ternary Quantization via Fine-grained Sparsification." arXiv:2601.07892, January 2026. arxiv.org/...2601.07892
18. ^1 ^2sjl623. "ggml-cpu: add STQ1_0 ternary quantization with ARM NEON vec_dot kernel." llama.cpp pull request 22836, opened May 8, 2026; status checked September 3, 2026. github.com/...22836