# Mixture of Experts (MoE)

> Source: https://aiwiki.ai/wiki/mixture_of_experts
> Updated: 2026-07-28
> Fact-checked: 2026-07-28
> Categories: Deep Learning, Machine Learning, Neural Networks
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Mixture of Experts (MoE)." aiwiki.ai, 28 Jul 2026. https://aiwiki.ai/wiki/mixture_of_experts
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

A mixture of experts (MoE) is a [machine-learning](https://aiwiki.ai/wiki/machine_learning) architecture that contains multiple component functions called experts and a gating or routing mechanism that determines how their outputs contribute to a result. In the original formulation, the gate assigned continuous weights to several local expert networks. In many modern sparse MoE systems, the router instead selects only a small subset of experts for each input or token.[1][2][3]

Sparse activation separates two quantities that are identical in a conventional dense layer: the total parameters stored in the layer and the parameters used for one input. Adding experts can therefore increase parameter capacity without requiring every expert to run on every token. It does not make computation or memory constant. Shared layers, the router, selected experts, data movement, and output combination still incur costs, while all expert weights must be stored, sharded, compressed, or loaded somehow.[5][6][12]

An expert is normally a trainable subnetwork inside one [neural network](https://aiwiki.ai/wiki/neural_network), not a separately deployed model. Modern language-model implementations often use independent feed-forward networks as experts in selected blocks of a [Transformer](https://aiwiki.ai/wiki/transformers), while leaving attention and other parts dense. MoE is not limited to Transformers, language models, or sparse routing, and learned experts are not guaranteed to correspond to human-readable topics.[3][9]

## Core architecture

Let $$x$$ be an input representation and let $$E_i(x)$$ denote the output of expert $$i$$. A router produces a score $$s_i(x)$$ for each of $$N$$ experts. A common token-choice implementation converts the scores to probabilities, selects a set $$S_k(x)$$ containing the $$k$$ highest-scoring experts, and combines their outputs:

$$
y(x)=\sum_{i\in S_k(x)} \alpha_i(x)E_i(x).
$$

The coefficients $$\alpha_i(x)$$ are derived from the router scores. They may be normalized across all experts before selection, renormalized across the selected experts, or handled in another implementation-specific way. The sparse selection means that only the chosen expert functions run for that token, but dense attention, normalization, residual paths, embeddings, and other shared components still run if the surrounding model contains them.[4][5]

In a typical language-model MoE block, the router operates on each token representation and replaces one or more dense feed-forward sublayers with a bank of expert feed-forward networks. GShard used a form of top-2 token routing in a multilingual [machine-translation](https://aiwiki.ai/wiki/machine_translation) Transformer, while the [Switch Transformer](https://aiwiki.ai/wiki/switch_transformer) simplified this design to one selected expert per token.[4][5] These are influential designs, not definitions of MoE.

Three parameter counts should be distinguished:

- **Total parameters** include every expert and all shared components.
- **Active parameters for one token** include the selected experts plus the shared components used for that token.
- **Parameters touched by a batch or sequence** are the union of the parameters activated by all of its tokens and can approach the total model size even when each individual token uses few experts.[12]

The same distinction applies to computation. Sparse expert activation can reduce expert arithmetic relative to evaluating every expert, but per-token floating-point operations also include the dense parts of the model. Router and communication work can be material at distributed scale.[5][11][20]

## Historical development

Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton introduced adaptive mixtures of local experts in 1991. Their supervised learning procedure trained several networks on different subsets of the cases and used a gating network to assign input-dependent mixture weights.[1] Michael Jordan and Robert Jacobs extended the idea in 1994 with a hierarchical mixture in which gating and expert components were arranged as a tree and trained with an expectation-maximization procedure.[2]

The modern large-scale sparse line developed from conditional computation. In 2017, Noam Shazeer and colleagues described a sparsely gated MoE layer with many feed-forward experts and a trainable gate that chose a sparse combination for each example. Their experiments placed MoE layers between recurrent layers for language modeling and translation, showing that the idea did not depend on the Transformer architecture.[3]

GShard subsequently combined sparse MoE layers, top-2 routing, and automatic sharding in a Transformer trained for multilingual translation.[4] Switch Transformer studied top-1 routing, expert capacity, load-balancing losses, selective router precision, and combinations of data, model, and expert parallelism.[5] Later work broadened the design space rather than establishing one standard recipe: balanced assignment, expert-choice routing, soft routing, dropless sparse kernels, fine-grained experts, shared experts, and continuous sparse routers each address different optimization or systems constraints.[7][8][10][15][16][18][19]

## Routing methods

### Token-choice routing

In token-choice routing, each token independently selects its highest-scoring $$k$$ experts. Top-1 routing minimizes expert computation and routing fan-out, while top-2 or higher $$k$$ evaluates more experts and combines their outputs. Increasing $$k$$ changes both compute and communication, so models with different $$k$$ values are not directly comparable from total parameter count alone.[4][5]

The top-$$k$$ operation is discrete and discontinuous. The selected experts and the router probabilities on the selected path can receive gradients, but the selection boundary itself is not an ordinary differentiable mapping. Training therefore relies on the particular gating formulation, auxiliary objectives, noise, or other estimators used by the implementation.[3][4][19]

### Balanced and expert-choice routing

Token-choice routing does not guarantee equal expert loads. BASE Layers instead posed allocation as a balanced linear-assignment problem so that each expert received the same number of tokens in the studied training setup.[7] Expert Choice Routing reversed the usual direction: each expert selected a fixed-size bucket of high-scoring tokens, allowing the number of experts used by a token to vary.[8]

These methods make balancing part of the assignment, but their scope matters. An assignment computed across a batch lets one token's route depend on other tokens in that batch. For an autoregressive [large language model](https://aiwiki.ai/wiki/large_language_model), a naive global expert-choice assignment can also let later tokens affect earlier assignments. A causal implementation must restrict or modify that computation.[17] Balanced assignment is therefore not a universal drop-in replacement for token-local routing.

### Soft and continuous routing

Soft MoE forms a fixed number of weighted input slots for each expert and then combines expert outputs back into token representations. It is fully differentiable and avoids hard token dropping, but it changes the computation being performed and was evaluated primarily in vision settings.[15] ReMoE is a different continuous approach that uses ReLU-based sparse routing instead of top-$$k$$ plus [softmax](https://aiwiki.ai/wiki/softmax).[19] Results for either method are evidence about the tested models and workloads, not proof that continuous routing is always preferable.

Routing can also be static, hashed, task-conditioned, or hierarchical. The term MoE describes the use of gated experts, not a single routing algorithm.

## Load balancing and expert capacity

If too many tokens choose the same expert, that expert can become a memory or latency bottleneck while other experts remain underused. Persistent imbalance can also leave some experts with relatively few training examples. Load balancing is therefore both an optimization problem and a systems-utilization problem.[4][5][8]

Many distributed implementations allocate a fixed token buffer to each expert. For a routing group containing $$T$$ tokens, $$N$$ experts, $$k$$ selected experts per token, and capacity factor $$c$$, a useful general approximation is

$$
C=\left\lceil c\frac{Tk}{N}\right\rceil .
$$

For Switch-style top-1 routing, this reduces to $$C=\lceil cT/N\rceil$$. Exact rounding, minimum capacity, routing-group boundaries, and whether second-choice assignments count in the same way vary by implementation. The formula should therefore be read as an average-assignment budget, not a universal API definition.[4][5]

A larger capacity factor leaves more room for imbalanced routes but may allocate more memory, communication volume, or padded computation. A smaller capacity can overflow. Depending on the design, an overflowing assignment may be skipped through a residual path, redirected to another expert, or processed by a variable-size or dropless kernel.[4][5][10]

Common balancing approaches include:

- an auxiliary loss that encourages the mean router probability and realized token fraction to be more uniform across experts;
- a balanced assignment or fixed expert bucket;
- router-score biases updated from recent load without backpropagating a balancing loss; and
- systems measures such as expert replication or adaptive placement.

Auxiliary losses add a second objective that can compete with the main learning objective if weighted too strongly. The auxiliary-loss-free bias method reported by Lean Wang and colleagues avoids those balancing gradients, but its published evidence is bounded to the studied models, data, and training scales.[17] MegaBlocks addressed a different part of the problem by using block-sparse operations that processed variable expert loads without dropping tokens or padding every expert to the maximum load.[10]

Router stability is related to, but distinct from, load balance. The ST-MoE study found training instabilities associated with large router logits and introduced a router z-loss to keep those logits small in its tested setup.[9] A z-loss does not by itself guarantee balanced expert use.

## Distributed execution

Sparse MoE becomes a distributed-systems problem when experts are placed on different accelerators. [Expert parallelism](https://aiwiki.ai/wiki/expert_parallelism) assigns different experts, or replicas of experts, to different devices. A common forward pass has four stages:

1. compute router scores and assignments;
2. group and dispatch token representations to the devices that own the selected experts;
3. evaluate each expert on its received tokens; and
4. return and combine the expert outputs in the original token order.

The dispatch and return stages are commonly implemented as two all-to-all collective communications.[5][11] Their cost depends on the interconnect topology, message size, number of participating devices, token distribution, precision, and whether communication can overlap with expert computation. Tutel showed that the best parallel layout and pipelining strategy changed with workload and scale, while COMET continued this line of work with finer-grained communication-computation overlap.[11][20] Their measured speedups are properties of their experimental systems, not fixed characteristics of MoE.

Expert parallelism complements rather than replaces other forms of [distributed training](https://aiwiki.ai/wiki/distributed_training). [Data parallelism](https://aiwiki.ai/wiki/data_parallelism) distributes examples, [tensor parallelism](https://aiwiki.ai/wiki/tensor_parallelism) partitions operations within layers, and [pipeline parallelism](https://aiwiki.ai/wiki/pipeline_parallelism) partitions groups of layers. [Model parallelism](https://aiwiki.ai/wiki/model_parallelism) is the broader category that can include several of these approaches. A deployment may combine them, which changes memory placement and collective-communication patterns.[5][11]

Load balance at the model level does not guarantee balanced wall-clock execution. Experts can have equal token counts but different effective costs if sequence grouping, token shapes, kernel efficiency, replicas, or device placement differ. Conversely, some imbalance may be acceptable if popular experts are replicated or placed on faster links. The relevant target is end-to-end time and resource use under the intended workload.

## Training behavior

Sparse routing changes which parameters receive updates. On a given step, each expert is trained only from the tokens routed to it. If early routing strongly favors a small subset, positive feedback can produce overloaded experts and poorly trained alternatives. Balancing objectives, capacity policies, router initialization, routing noise, expert assignment methods, and numerical precision are among the controls studied in the literature.[3][4][5][9]

Expert specialization can emerge, but it should be measured rather than assumed. The 2017 sparsely gated study found syntactic and semantic patterns in some language experts.[3] ST-MoE later reported visible specialization in selected encoder experts, much less meaningful specialization in decoder experts, and no simple language-per-expert partition in its multilingual analysis.[9] An expert number is therefore not a dependable topic label, and inspecting a few routed examples does not establish a complete semantic role.

Sparse upcycling initializes an MoE from a pretrained dense checkpoint, commonly by copying a dense feed-forward layer into multiple experts and then continuing training with a new router. Komatsuzaki and colleagues demonstrated this approach for language and vision models, while also showing that its benefit depended on the additional training budget and configuration.[13] Upcycling reuses a checkpoint; it does not eliminate the need to train routing and differentiate initially identical experts.

The size and organization of experts are additional design variables. Fine-grained MoE divides a comparable expert parameter budget into more, smaller experts and activates a larger combination of them. Scaling-law experiments have treated this granularity as a separate hyperparameter rather than assuming a fixed expert size.[18] DeepSeekMoE combined fine-grained routed experts with shared experts that were always active in its tested architecture.[16] Those designs illustrate alternatives to uniform, wholly routed experts, but their reported quality and efficiency comparisons do not transfer automatically to other models.

## Inference and deployment

Sparse arithmetic does not imply a proportionally small deployment. For low-latency [inference](https://aiwiki.ai/wiki/inference), all experts that might be selected generally need to be resident somewhere in the serving system or available through a loading policy. Sharding spreads their weights across devices; replication trades more memory for reduced contention; offloading reduces accelerator residency but adds data movement; and [quantization](https://aiwiki.ai/wiki/quantization) changes storage and arithmetic precision. The memory per device can therefore be much lower than the total model size, but the aggregate model weights still exist.[12]

A token may activate only one or two experts while a batch of heterogeneous tokens activates many experts. This makes active parameters per token an incomplete predictor of memory traffic and latency. Prompt processing often offers more tokens to group by expert than single-token autoregressive decoding, while decoding efficiency depends strongly on concurrent request batching. Small expert batches can underuse matrix hardware, and communication or weight reads can dominate.[12]

Serving metrics also pull in different directions. Replicating a frequently selected expert may improve throughput but consume memory needed for other experts. Increasing capacity may reduce overflow but enlarge buffers. Restricting an all-to-all group may lower communication distance while reducing placement flexibility. A configuration should be evaluated on the intended hardware, traffic distribution, sequence lengths, batch policy, and latency objective rather than inferred from an architecture label.

## Evaluation and tradeoffs

MoE and dense models have more than one scale axis. Routed-language-model experiments found that total parameter count and computation both affected performance and could not be collapsed into the usual dense-model parameter count without an additional model of their relationship.[6] Fine-grained MoE experiments added expert granularity and training tokens to that comparison.[18] As a result, neither total parameters nor active parameters alone establishes model quality or efficiency.

A useful comparison reports at least:

| Dimension | Information needed |
| --- | --- |
| Model | total parameters, shared parameters, expert count and size, layers using MoE, routing method, and $$k$$ |
| Training | data, tokens, objective, optimizer, precision, total training compute, and failed or dropped-token rate |
| Per-token work | active expert parameters, dense shared work, measured or estimated operations, and router work |
| Placement | devices, weight precision, sharding, replication, offloading, and aggregate memory |
| Communication | expert-parallel group, topology, bytes moved, collective implementation, and overlap policy |
| Quality | held-out loss and downstream evaluations under matched data and evaluation procedures |
| Serving | prompt and decode latency, throughput, batch policy, sequence lengths, and service-level target |

Wall-clock speedups in GShard, Switch, DeepSpeed-MoE, Tutel, MegaBlocks, or COMET were measured with particular models, baselines, software, hardware, and parallel layouts.[4][5][10][11][12][20] They should not be presented as a general MoE multiplier. The same caution applies to energy use and cost.

Reported operation counts also need a declared boundary. Expert matrix multiplications may dominate an arithmetic estimate while routing, padding, collective communication, synchronization, and weight movement are measured elsewhere or omitted. FLOPs can support a matched algorithmic comparison, but they do not replace elapsed time, memory, network traffic, or service-level measurements on the target system.

MoE also differs from an ordinary ensemble. An ensemble usually combines predictions from multiple component models, often after separate training. An MoE commonly learns routing and experts jointly and can place routed layers inside a larger network. The terminology overlaps in some fields, so the training and routing structure is more informative than the name alone.

## Applications

The first neural mixtures of local experts were general supervised architectures.[1][2] Large sparse MoE work later focused heavily on language modeling and translation.[3][4] The technique also applies beyond language: V-MoE inserted sparse expert layers into a [Vision Transformer](https://aiwiki.ai/wiki/vision_transformer) and evaluated them on image recognition, while Soft MoE studied differentiable slot-based routing in visual recognition.[14][15]

These examples show that MoE is a conditional-computation pattern rather than a synonym for a particular [language model](https://aiwiki.ai/wiki/language_model). Whether it helps depends on the data, task, optimization method, expert design, router, and execution system. A dense model can be preferable when parameter storage, communication, small-batch latency, implementation complexity, or training stability matters more than expanding conditional capacity.

## References

1. Jacobs, Robert A., Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. "Adaptive Mixtures of Local Experts." Neural Computation 3, no. 1 (1991): 79-87. https://www.cs.toronto.edu/~hinton/mixex.html
2. Jordan, Michael I., and Robert A. Jacobs. "Hierarchical Mixtures of Experts and the EM Algorithm." Neural Computation 6, no. 2 (1994): 181-214. https://doi.org/10.1162/neco.1994.6.2.181
3. Shazeer, Noam, et al. "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer." International Conference on Learning Representations (2017). https://arxiv.org/abs/1701.06538
4. Lepikhin, Dmitry, et al. "GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding." International Conference on Learning Representations (2021). https://arxiv.org/abs/2006.16668
5. Fedus, William, Barret Zoph, and Noam Shazeer. "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity." Journal of Machine Learning Research 23, no. 120 (2022): 1-39. https://jmlr.org/papers/v23/21-0998.html
6. Clark, Aidan, et al. "Unified Scaling Laws for Routed Language Models." Proceedings of the 39th International Conference on Machine Learning (2022): 4057-4086. https://proceedings.mlr.press/v162/clark22a.html
7. Lewis, Mike, et al. "BASE Layers: Simplifying Training of Large, Sparse Models." Proceedings of the 38th International Conference on Machine Learning (2021): 6265-6274. https://proceedings.mlr.press/v139/lewis21a.html
8. Zhou, Yanqi, et al. "Mixture-of-Experts with Expert Choice Routing." Advances in Neural Information Processing Systems 35 (2022). https://proceedings.neurips.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abstract-Conference.html
9. Zoph, Barret, et al. "ST-MoE: Designing Stable and Transferable Sparse Expert Models." arXiv (2022). https://arxiv.org/abs/2202.08906
10. Gale, Trevor, Deepak Narayanan, Cliff Young, and Matei Zaharia. "MegaBlocks: Efficient Sparse Training with Mixture-of-Experts." Proceedings of Machine Learning and Systems 5 (2023). https://proceedings.mlsys.org/paper_files/paper/2023/hash/5a54f79333768effe7e8927bcccffe40-Abstract-mlsys2023.html
11. Hwang, Changho, et al. "Tutel: Adaptive Mixture-of-Experts at Scale." Proceedings of Machine Learning and Systems 5 (2023). https://proceedings.mlsys.org/paper_files/paper/2023/hash/5616d34cf8ff73942cfd5aa922842556-Abstract-mlsys2023.html
12. Rajbhandari, Samyam, et al. "DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale." Proceedings of the 39th International Conference on Machine Learning (2022): 18332-18346. https://proceedings.mlr.press/v162/rajbhandari22a.html
13. Komatsuzaki, Aran, et al. "Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints." International Conference on Learning Representations (2023). https://openreview.net/forum?id=T5nUQDrM4u
14. Riquelme, Carlos, et al. "Scaling Vision with Sparse Mixture of Experts." Advances in Neural Information Processing Systems 34 (2021). https://proceedings.neurips.cc/paper/2021/hash/48237d9f2dea8c74c2a72126cf63d933-Abstract.html
15. Puigcerver, Joan, Carlos Riquelme Ruiz, Basil Mustafa, and Neil Houlsby. "From Sparse to Soft Mixtures of Experts." International Conference on Learning Representations (2024). https://proceedings.iclr.cc/paper_files/paper/2024/hash/79fea214543ba263952ac3f4e5452b14-Abstract-Conference.html
16. Dai, Damai, et al. "DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (2024): 1280-1297. https://aclanthology.org/2024.acl-long.70/
17. Wang, Lean, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. "Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts." arXiv (2024). https://arxiv.org/abs/2408.15664
18. Ludziejewski, Jan, et al. "Scaling Laws for Fine-Grained Mixture of Experts." Proceedings of the 41st International Conference on Machine Learning (2024): 33270-33288. https://proceedings.mlr.press/v235/ludziejewski24a.html
19. Wang, Ziteng, Jun Zhu, and Jianfei Chen. "ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing." International Conference on Learning Representations (2025). https://proceedings.iclr.cc/paper_files/paper/2025/hash/94dc604e115237a7f4a758b3146cd976-Abstract-Conference.html
20. Zhang, Shulai, et al. "COMET: Fine-grained Computation-communication Overlapping for Mixture-of-Experts." Proceedings of Machine Learning and Systems 7 (2025). https://proceedings.mlsys.org/paper_files/paper/2025/hash/e27ea0cd50b798ff8942caf9203f0992-Abstract-Conference.html