Alice AI Foundation 80B-A3B

AliceAI-Foundation-80B-A3B-Base (announced in English as "Alice AI Foundation 80B-A3B Base") is an open-weights base large language model from Yandex, published on Hugging Face and announced on 21 September 2026 under the Apache License 2.0.^[1][3][4] It is a sparse mixture-of-experts (MoE) model with 80 billion total parameters, of which about 3 billion are active for each token, and a maximum context length of 262,144 tokens.^[1] Yandex describes it as a pretrained model that was "trained entirely from scratch", and its Russian-language launch post adds that no weights from third-party open-source models were used.^[1][3] That distinguishes it from Yandex's previous closed Alice AI LLM 235B, whose training started from the weights of Alibaba's Qwen model Qwen3-235B-A22B.^[3]

The model uses a hybrid attention design in which three linear-attention layers based on Kimi Delta Attention alternate with one gated full-attention layer, and every block ends in a 512-expert MoE layer.^[1][2] Alongside the weights, Yandex published a long technical write-up on Habr, two Russian factual-knowledge benchmarks (WikiWebFacts and HardMultiQA), a fine-tuning chat template and a LoRA example script.^[1][3][6][7]

Background

Alice AI is the name of Yandex's AI assistant; Yandex calls the fast answers Alice AI gives in Yandex Search its most widely used generative product.^[9] According to the Foundation launch post, the company's previous large model, Alice AI LLM 235B (base released in October 2025 and described in a December technical report), was built by continuing training from Qwen3-235B-A22B.^[3] For the new model the team rebuilt the training corpus, re-chose the architecture, fitted hyperparameters with its own scaling laws and added reasoning and agentic data; Yandex says the whole path to release took about six months.^[3]

Yandex frames the release as "another step" toward what it calls a unified reasoning model (in Russian, единая рассуждающая модель), on which the agentic capabilities of Alice AI are to be developed.^[3] Yandex has not described the published checkpoint as the model that serves Alice AI users; it is a base model and the repository deliberately ships no default chat template.^[1]

The release followed a related one. On 11 September 2026 the Alice AI Search team open-sourced AliceAI-T5-35B-A0.6B, an encoder-decoder MoE pretrain behind those fast answers in Yandex Search (see below). The Foundation post says an updated pretraining dataset underlies both models.^[3][9]

Release

The repository also contains an export-control notice asking users to comply with applicable export-control law, and a contributor notice requiring external contributors to accept the Yandex Contributor License Agreement.^[10] The model card, written in Russian with an English translation, lists the tags custom_code, mixture-of-experts and vllm.^[1]

Architecture

The model card and config.json give the following configuration.^[1][2]

Hybrid attention. Kimi Delta Attention (KDA) is the linear-attention mechanism introduced by Moonshot AI in Kimi Linear.^[3][11] The Habr post explains the motivation: in full attention the KV cache grows with sequence length, while KDA folds history into a fixed-size state, so its memory use does not grow with context. Unlike Gated DeltaNet in Qwen3-Next, which applies one decay coefficient per attention head, KDA uses a separate coefficient for each row of the state matrix.^[3] Yandex reports that in its experiments on 5-15B models, replacing Gated DeltaNet with KDA lowered loss by about 0.02 and improved several task groups, including roughly 5 points on MMLU-Pro CoT.^[3]

Attention Residuals. Instead of summing the outputs of earlier layers in the residual stream, the model uses Attention Residuals (AttnRes), a method from a Kimi Team paper that computes a learned, per-token softmax weighting over depth.^[3][12] Yandex uses the block variant with a block of eight layers (four transformer blocks, counting attention and MoE sublayers separately) and aligned activation-checkpoint boundaries with the AttnRes operations to limit saved activations. On 5-15B test models, Block AttnRes lowered loss by about 0.017 and added about 3 points on MMLU-Pro CoT.^[3]

Routing. MoE routing follows the auxiliary-loss-free, bias-based load balancing of DeepSeek-V3.^[3]

Training

The launch post describes four sequential training stages.^[3]

Yandex put most of the long-context compute into the 256K stage because extending the 32K stage did not noticeably help long-context benchmarks. The final stage was shortened to 128K context to save compute, on the grounds that the longest reasoning trajectories in its data were often noisy.^[3] Only the checkpoint after the reasoning stage is released.^[3]

Ablations. Yandex validated its changes in a series of from-scratch runs of 2 trillion tokens each.^[1] It compared four configurations in sequence: the previous corpus with a Qwen3-Next architecture, the new corpus with that architecture, the new corpus with Yandex's own architecture, and finally hyperparameters predicted by its scaling laws. The post reports that the new corpus improved all 13 benchmarks in the comparison and that the full configuration had the best macro-average and the best score on 7 of 13.^[3] The scaling-law fits used models of about 5B and 10B total parameters trained on 100B to 600B tokens.^[3]

Optimizer and stability. Training used the Muon optimizer with a linear learning-rate schedule; Yandex wrote a distributed, pipelined Muon step that it says roughly halved optimizer-step time in the final run.^[3] A test run at the learning rate chosen by the scaling law produced a sharp loss spike together with fast growth of the activations at the output of the MoE down-projection. The team tried a router Z-loss, DeepSeek-style routing (kept because it improved quality), a hard-clipped SwiGLU of the kind used in gpt-oss (rejected because it hurt quality) and Gated RMSNorm (stable together with Attention Residuals but about 5% slower). The final fix moved the Z-loss idea from router logits to the MoE output activations, which Yandex says kept training stable without a quality loss or slowdown.^[3]

Training data

The corpus mixes natural data (web pages, public code, books, scientific papers, news, multilingual material and open datasets) with semi-synthetic data (generated or corrected solutions and augmented versions of the most useful web pages) and fully synthetic tasks and solutions.^[3] It started from the corpus of YandexGPT 5 Lite Base plus datasets prepared for Alice AI LLM 235B. Yandex reports that augmentation recipes that worked when continuing training from open models had to be reworked for a longer from-scratch pretrain, and it describes a pipeline for collecting and paraphrasing factual documents, comparing the approach to one described in the Kimi K2 technical report.^[3]

The final stage added reasoning traces and agent data. Yandex's stated finding was that hard-reasoning data alone did not produce agentic skills: a reasoning-only checkpoint trailed a Qwen base model on most agent benchmarks after the same supervised fine-tuning, and adding tool-interaction data to pretraining closed much of the gap and made later RLVR training more stable.^[3] The agent data included about 7 billion tokens of trajectories in synthetic CRUD environments built with a pipeline modelled on the Agent World Model paper, about 1 billion tokens of multi-hop search trajectories generated from 10,000 questions, and software-engineering data following the daVinci-Dev approach.^[3]

Benchmarks

All figures below are Yandex's own measurements of base (pretrained) checkpoints, run on its internal evaluation infrastructure with vLLM.^[1][3]

Knowledge and core skills

This table was run at temperature 0 for every model. Most tasks are 5-shot; CultCat is 4-shot and the two coding tasks are 1-shot. TriviaQA and MATH-500 are graded with an LLM-as-a-judge instead of exact match.^[1] The Alice AI LLM 235B column appears only in the Habr post, not on the model card.^[3]

Bold marks the best score in each row across all six columns. Among the five open base models on the model card, Yandex's model has the top score (or a tie) on 13 of the 20 rows. The larger models lead on English TriviaQA, EduBench English, MMLU-Pro, SuperGPQA, BigCodeBench and LongMemEval, and Qwen3.5-35B-A3B leads on EduBench Math.^[1] The Habr post itself says the model is first on 8 of the 10 factual, educational and expert-knowledge benchmarks.^[3] The EduBench tasks are built from queries submitted to Alice, ExpertFactsQA was written by domain experts, EGE is based on multiple-choice tasks from Russia's Unified State Exam, and FinQA 128k is Yandex's long-context version of the FinQA financial-report benchmark.^[1][3] CultCat is an earlier Yandex benchmark of cultural knowledge.^[1]

Yandex notes that every model in the comparison except Qwen3.5-35B-A3B has more total and more active parameters than its own; the Qwen model has fewer total parameters and the same number of active ones.^[3] Against its own previous closed model, the post says the new model is ahead on factual knowledge, mathematics, coding and long-context work while having almost three times fewer total parameters and about seven times fewer active parameters.^[3] The 235B model still scores higher on EGE, TriviaQA, MMLU-Pro and BigCodeBench in the same table.^[3]

Reasoning benchmarks for base models

For harder math and programming tasks, Yandex scores base models zero-shot with pass@k, counting a task as solved if any of k samples is correct, at temperature 1 with presence_penalty 1.5 and repetition_penalty 1.^[1][3] The post argues that pass@k shows whether a correct trajectory already exists for later reinforcement learning to strengthen.^[3] DeepSeek-V4-Flash-Base is left out because its scores stayed close to zero under several evaluation settings, and GLM-4.5-Air also lost too much quality zero-shot to give a useful signal.^[3]

Source: model card and Habr post; Yandex's measurements.^[1][3]

Yandex also reports agent benchmarks such as tau2-bench, VitaBench, DeepPlanning and SWE-bench Verified (in OpenHands), but only after applying the same supervised fine-tuning to every base model, and it uses them mainly to compare its own pretraining experiments.^[3]

WikiWebFacts and HardMultiQA

Yandex released two Russian-language factual benchmarks with the model, both on Hugging Face under CC BY-SA 4.0 (the repositories are gated: users must log in and agree to share contact information before downloading) with LightEval evaluation scripts for pretrained ("pretrain") and instruction-tuned ("alignment") models.^[6][7] Both dataset cards ask users to share a link to the repository rather than republish the questions, to reduce the chance of contamination; the cards say this is a voluntary request that does not limit the license.^[6][7]

Answers are graded with an LLM judge. Yandex says it checked the judges against a larger auditor model and that judge accuracy exceeded 95%, for each task type in the case of HardMultiQA.^[3] The launch post describes WikiWebFacts as not very hard for current frontier models and nearing saturation for the strongest ones; its main use is a stable signal for comparing from-scratch runs and smaller models. HardMultiQA questions were made deliberately non-trivial by checking drafts against a set of internal and external models during annotation.^[3]

Usage

Transformers. The card names Transformers 5.16.1 as the reference version and requires trust_remote_code=True. Running the KDA layers on GPU needs the flash-linear-attention package (version 0.5.0 in the card's instructions) with KDA support.^[1]

vLLM. For vLLM, the card uses a prebuilt Docker image (yamlbrand/alice-ai-vllm) with tensor parallelism across four GPUs, and it enables speculative decoding through the model's multi-token prediction layer with one speculative token.^[1]

Tokenizer. The tokenizer loads as LlamaTokenizer from a SentencePiece BPE model. Reasoning markers ([COT_ENABLE], [COT_START], [COT_END]) and tool-use markers are ordinary vocabulary tokens rather than Hugging Face special tokens, and the card warns users not to change the legacy: false setting.^[1]

Fine-tuning. Yandex rendered its agentic training data in the OpenAI Messages format (system, user, assistant, tool and meta roles, with tool definitions in a separate tools field) through a Jinja template, which it publishes as finetune/chat_template.jinja and recommends for SFT and RL data. The template is intentionally not set as the tokenizer's chat template, because a base model has no single conversational format.^[1] The repository also includes a minimal PEFT LoRA script that trains on the tatsu-lab/alpaca dataset with loss computed only on responses; the card says a model of this size needs FSDP2 and designs the example for four 80 GB GPUs.^[1]

Within two days of release, third parties had posted converted versions on Hugging Face, including FP8, NVFP4, GGUF and MLX 4-bit builds.^[13] These are not Yandex releases.

Related release: AliceAI-T5-35B-A0.6B

AliceAI-T5-35B-A0.6B-Base (repository yandex/AliceAI-T5-35B-A0.6B) is an encoder-decoder base model with sparse MoE layers: 34.35 billion unique parameters, 512 experts per MoE layer and top-8 routing, a 16-layer encoder and 12-layer decoder, and a 128K context using RoPE with YaRN.^[8] It is also Apache 2.0 licensed.^[8] Yandex's Alice AI Search team describes it as the pretrain behind the fast answers that Alice AI gives in Yandex Search, trained from scratch, and notes that outside users get inference through Hugging Face Transformers while the optimized production inference stack remains internal to Yandex.^[9] The T5 model uses a different architecture from the 80B Foundation model; the Foundation post connects the two through the updated pretraining dataset that underlies both.^[3]

Coverage

iXBT reported the release on 22 September 2026. It summarized it as an experiment on the path toward a unified reasoning model for Alice AI, trained from scratch without third-party open weights, and highlighted the two open Russian benchmarks.^[14]

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26Yandex. "AliceAI-Foundation-80B-A3B-Base" (model card, English version). Hugging Face. huggingface.co/...README_en.md
2. ^1 ^2 ^3 ^4Yandex. "config.json", AliceAI-Foundation-80B-A3B-Base. Hugging Face. huggingface.co/...config.json
3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41 ^42 ^43Yandex (Habr company blog). "Открываем AliceAI-Foundation-80B-A3B-Base: новую языковую модель Яндекса, обученную с нуля" (Opening AliceAI-Foundation-80B-A3B-Base, Yandex's new language model trained from scratch). Habr, 21 September 2026. habr.com/...1083300
4. ^1 ^2Yandex (@yandexcom). Post announcing the Alice AI Foundation 80B-A3B Base weights. X, 21 September 2026. x.com/...2102050970939383810
5. ^1 ^2Yandex. "LICENSE" (Apache License 2.0, Copyright 2026 YANDEX LLC), AliceAI-Foundation-80B-A3B-Base. Hugging Face. huggingface.co/...LICENSE
6. ^1 ^2 ^3 ^4Yandex. "WikiWebFacts" (dataset card). Hugging Face. huggingface.co/...WikiWebFacts
7. ^1 ^2 ^3 ^4Yandex. "HardMultiQA" (dataset card). Hugging Face. huggingface.co/...HardMultiQA
8. ^1 ^2Yandex. "AliceAI-T5-35B-A0.6B-Base" (model card). Hugging Face. huggingface.co/...AliceAI-T5-35B-A0.6B
9. ^1 ^2 ^3Yandex (Habr company blog). "Открываем претрейн Alice AI Search: как устроена модель быстрых ответов Алисы на Поиске" (Opening the Alice AI Search pretrain). Habr, 11 September 2026. habr.com/...1080654
10. ^1 ^2Yandex. AliceAI-Foundation-80B-A3B-Base repository files (safetensors shards, NOTICES, CONTRIBUTING.md). Hugging Face. huggingface.co/...main
11. ^Kimi Team. "Kimi Linear: An Expressive, Efficient Attention Architecture". arXiv:2510.26692. arxiv.org/...2510.26692
12. ^Kimi Team. "Attention Residuals". arXiv:2603.15031. arxiv.org/...2603.15031
13. ^Hugging Face. Model search results for "AliceAI-Foundation". huggingface.co/models
14. ^iXBT. "Яндекс открыл 80-миллиардную языковую модель, обученную с нуля" (Yandex opens an 80-billion-parameter language model trained from scratch). 22 September 2026. ixbt.com/...vdvoe-mensem-cisle-aktivnyx-parametrov
15. ^Hugging Face. Model API record for yandex/AliceAI-Foundation-80B-A3B-Base (safetensors parameter count, file list). huggingface.co/...AliceAI-Foundation-80B-A3B-Base