Mixture of Experts

Mixture of Experts (MoE) is a neural network architecture technique in machine learning and artificial intelligence that enables large language models to achieve computational efficiency through conditional computation, where only a small subset of specialized networks processes each input. The architecture allows models to scale to trillions of parameters while maintaining manageable computational costs by activating only a fraction of the network for each input token.

Overview

Mixture of Experts represents a revolutionary approach to scaling neural networks by decomposing a large model into multiple specialized subnetworks called "experts," coordinated by a learned gating network or router that determines which experts should process each input. Unlike traditional dense neural networks where all parameters activate for every input, MoE architectures implement sparse activation where only selected experts process each token, enabling constant computational cost independent of total parameter count.

This approach is considered a form of ensemble learning, essentially a "committee" of expert models that cooperate under the guidance of a gating mechanism. By activating only a small number of experts for each input, MoE architectures can scale up to models with extremely large numbers of parameters while keeping the per-input computational cost relatively fixed.

DeepSeek-V3, released in December 2024, demonstrates the architecture's maturity with 671 billion total parameters but only 37 billion activated per token, achieving GPT-4-level performance while training for approximately $6 million. This sparse activation principle fundamentally changes how the AI industry approaches model scaling, decoupling model capacity from computational requirements.

The approach traces back to 1991 when researchers at the University of Toronto first introduced adaptive mixtures of local experts for neural networks, but found practical realization only in 2017 when Google Brain demonstrated that sparsely-gated MoE layers could scale to 137 billion parameters.

History

Early foundations (1991-2016)

The intellectual foundation for Mixture of Experts emerged in 1991 when Robert Jacobs, Michael I. Jordan, Steven Nowlan, and Geoffrey Hinton published "Adaptive Mixtures of Local Experts" in Neural Computation. Their seminal work introduced the core concept of training multiple specialized neural networks (experts) where each learns to handle specific subsets of training data, coordinated by a gating network that determines expert selection. The original implementation used Expectation-Maximization algorithms and demonstrated the approach on vowel discrimination tasks. In this work, a set of simple expert neural networks were trained alongside a gating network that learned to weight each expert's contribution based on the input. The gating network effectively partitions the input space into regions, assigning higher weight to the expert that is most competent for each region. This pioneering MoE model was shown to reach a target accuracy in about half the training epochs required by a single large network, demonstrating faster training through divide-and-conquer specialization.

Around the same time, Hampshire and Waibel (1992) developed the Meta-Pi network, a related multi-network approach that also combined multiple subnetworks via an adaptive weighting function for robust speech recognition. The Meta-Pi network used a similar formula for mixing expert outputs and can be seen as an early instance of the MoE principle applied to phoneme classification tasks. It was observed that one expert specialized in a subset of speakers, while another expert handled a different subset, and some inputs were handled by combinations of experts for interpolation.

Jordan and Jacobs extended this in 1994 with hierarchical architectures featuring tree-structured gating across multiple levels, introducing the hierarchical mixture of experts (HME) model, organizing experts in a tree-like hierarchy of gating networks. They showed that the parameters of an HME can be learned effectively using the Expectation-Maximization (EM) algorithm, treating the gating outputs as latent variables in a probabilistic model. This allowed a principled maximum-likelihood training of MoEs and improved convergence properties, with EM training achieving 10x faster convergence than gradient descent methods. Subsequent work provided theoretical guarantees for MoE training—for example, Jordan and Xu (1995) analyzed convergence of the EM-based learning in MoE architectures.

Variations and refinements of the MoE approach were explored throughout the 1990s, sometimes under different names (e.g. "gating networks" or "mixture of experts networks"), and the technique was recognized as one of the architectures in the family of ensemble learning and modular neural network methods.

In the early 2000s, the MoE idea was applied beyond neural networks. Collobert and Bengio (2001) introduced a "parallel mixture of SVMs", an MoE model where each expert was a support vector machine trained on a subset of the data. This demonstrated that the MoE gating concept could scale to large datasets by splitting the problem across multiple learners, each handling a portion of the input space. While MoE did not see widespread use during the 2000s compared to other ensemble methods like boosting or random forests, it continued to be a subject of research in various domains.

For two decades, MoE remained primarily a theoretical framework explored in classical machine learning literature. The deep learning era initially passed over MoE architectures, with researchers in 2013 attempting to incorporate gating at each layer of deep networks but failing to achieve practical sparsity.

Modern deep learning era (2017-present)

A major resurgence of interest in Mixture of Experts came in the mid-2010s with the rise of deep learning and the demand for scaling models. The breakthrough came in 2017 when Noam Shazeer and colleagues at Google Brain published "Outrageously Large Neural Networks," introducing sparsely-gated MoE layers with noisy top-k selection and auxiliary load balancing losses. This work demonstrated that conditional computation could achieve 1000x capacity increases with minor computational overhead, scaling to 137 billion parameters in LSTM-based language models when dense models were limited to hundreds of millions of parameters. Their MoE prototype network had over 137 billion parameters but only activated a small fraction (e.g. two experts) for each input, achieving significant improvements in model capacity and training speed at fixed computational cost. This breakthrough showed the viability of training "outrageously" large neural networks using MoE's conditional computation, and it rekindled widespread interest in MoE for scaling neural network models.

The transformer era accelerated MoE adoption dramatically. Google's GShard in 2020 demonstrated 600 billion parameter machine translation models trained in four days on 2048 TPU cores, using top-2 routing with automatic sharding across devices. Lepikhin et al. (2021) reported training a multilingual translation model with 600 billion parameters using MoE (with top-2 gating) and an automatic model-sharding system (GShard) to distribute training across 2048 TPU cores. They introduced additional techniques to improve training, such as random routing (the second expert is chosen probabilistically among the top candidates to encourage load balancing) and expert capacity limits (each expert processes at most a fixed number of tokens per batch to prevent any single expert from overload). GShard demonstrated that MoE could scale to hundreds of billions of parameters and achieve superior quality in translation tasks (100 languages to English) compared to dense models, all while training efficiently through conditional computation.

Switch Transformers followed in 2021, simplifying routing to top-1 selection and achieving 7x pretraining speedup over T5 while scaling to 1.6 trillion parameters across 2048 experts. In 2021, Fedus, Zoph, and Shazeer introduced the Switch Transformer, which simplified the MoE routing by selecting only one expert (top-1) per input token. By using a single expert (the one with highest gating score) for each token, the Switch Transformer further reduced the communication and computation overhead of MoE. The authors showed that a Switch Transformer based on Google's T5 model achieved a 4× speedup in pre-training compared to a dense T5 of similar quality, and they successfully trained models up to 1.6 trillion parameters (with 128 experts per MoE layer). This work proved that sparse models could train stably in bfloat16 precision through selective precision techniques and router z-loss mechanisms.

Google's *GLaM* (Generalist Language Model) is another MoE-based large language model, introduced in late 2021. GLaM uses MoE layers with top-2 gating: for each token, the two highest-weight experts are activated per layer. The largest GLaM model has 1.2 trillion parameters spread across 64 experts in each of 32 MoE layers (interleaved with shared layers), but only about 1/32 of those parameters are active for a given token (roughly 37 billion active parameters per token). Thanks to this sparsity, GLaM achieved better average performance on NLP benchmarks than the dense 175B-parameter GPT-3, while using only about one-third of the energy to train and half the inference FLOPs. GLaM demonstrated the practical benefits of MoE at scale: it essentially delivered more than 7× the parameter count of GPT-3 at a fraction of the computational cost, validating MoE as a path toward bigger-but-efficient models.

A series of other advanced models and research efforts have continued to refine MoE techniques. For instance, Meta AI's NLLB-200 (2022) used a hierarchical MoE for an extreme multilingual translation system: first a gate chooses between a shared universal expert vs. language-specific experts, then another gate picks among the language-specific experts if that path is chosen.

In 2022, Google researchers proposed ST-MoE (Stable and Transferable MoE), exploring strategies to stably fine-tune sparse MoE models. They found, for example, that fine-tuning only the dense layers (or a subset like the feed-forward layers) while freezing the expert layers can sometimes yield better performance than naively fine-tuning all parameters, presumably because it avoids destabilizing the expert specializations.

The open-source revolution began in December 2023 when Mistral AI released Mixtral 8x7B under Apache 2.0 license, proving that production-grade MoE models could be democratized. With 46.7 billion total parameters but only 12.9 billion active per token, Mixtral matched GPT-3.5 performance while achieving 6x faster inference than Llama 2 70B. Mixtral's release under Apache 2.0 (with an instruction-tuned variant) marked an important step in making MoE practical in the broader AI community. In March 2024, Databricks followed with an MoE model (dubbed DBRX 132B) featuring 16 experts (132B total, 4 experts used per token) along with an instruction-tuned version, further indicating the growing adoption of MoE in large-scale models.

Research in 2023 also showed that MoE models can benefit from techniques like instruction tuning: Shen et al. (2023) demonstrated that applying instruction-tuning methods to an MoE LLM produced strong results, suggesting that MoE and instruction tuning are complementary for building versatile LLMs.

Another notable point is the rumored use of MoE in cutting-edge proprietary models. For example, some experts speculated that OpenAI's GPT-4 might internally be an MoE model — one hypothesis (attributed to George Hotz) posited that GPT-4 could consist of 8 expert models of ~220B parameters each (total ~1.76 trillion) combined via a MoE architecture. OpenAI has not confirmed this, but the possibility underscores how MoE is seen as a plausible path to scaling AI systems beyond what's feasible with dense models alone.

Architecture

Core components

At the core of a Mixture of Experts model is a set of expert models and a gating function. Formally, an MoE consists of the following components: Expert networks ($f_{1},f_{2},\dots ,f_{n}$) are multiple specialized feedforward neural networks (or other models), typically with identical architectures but independent parameters, where each expert can specialize in processing different aspects of the input data. All experts receive the same input $x$ but may produce different outputs: $f_{1}(x),f_{2}(x),\dots ,f_{n}(x)$. In transformer architectures, MoE layers replace the dense feedforward networks that appear after self-attention blocks. Each expert implements a standard two-layer feedforward network with an intermediate dimension and activation function. The gating network (router), denoted $g(x)$ (also called a gating function or router), serves as the coordination mechanism determining which experts process each input token. This learned component takes the input $x$ and computes a probability distribution over all available experts, producing a set of non-negative weights or probabilities $w_{1}(x),w_{2}(x),\dots ,w_{n}(x)$, one for each expert. These gating outputs determine how much each expert's output will contribute for the given input. Typically, the gating network is itself a lightweight neural network or logistic regression that outputs a probability distribution over experts (often via a softmax function). The gating network is parameterized by a learnable weight matrix $W_{g}$, computing: $G(x)={\text{Softmax}}(W_{\text{gate}}\cdot x)$. The combiner (mixture output) aggregates expert outputs, usually via a weighted sum. The final output of the MoE, $y$, is computed by combining the expert outputs using the gating weights. In the basic formulation: $y=\sum _{i=1}^{n}w_{i}(x),f_{i}(x)$ In other words, the gating network's output $w_{i}(x)$ serves as an adaptive weight on expert $i$'s prediction. (In some cases, a hard selection is used instead: the gating chooses the single best expert for the input, rather than a weighted sum.) Both the experts and the gating network have trainable parameters. The MoE model is typically trained end-to-end by defining a loss function $L$ on the mixture output (for example, mean squared error for regression or cross-entropy for classification) and minimizing $L$ with respect to all parameters.

Mathematical formulation

The mathematical formulation of MoE output captures the conditional combination of expert outputs. For input $x$, the MoE layer computes: $y=\sum _{i=1}^{N}g_{i}(x)\cdot E_{i}(x)$ where $E_{i}(x)$ represents expert $i$'s output and $g_{i}(x)$ represents the gating probability (with $g_{i}(x)\geq 0$ and typically $\sum _{i}g_{i}(x)=1$ through softmax normalization). In sparse implementations with top-k selection, this summation includes only $k$ non-zero terms, maintaining constant computational cost regardless of total expert count.

Routing mechanisms

The routing mechanism in an MoE determines how input tokens are assigned to experts. The process typically involves several steps: Scoring: The router, parameterized by a learnable weight matrix $W_{g}$, computes a vector of logits (scores) $s$ for the input token $x$: $s=W_{g}\cdot x$ Noise Injection (Training Only): To improve load balancing during training, tunable noise is often added to the logits. This encourages exploration and prevents the router from collapsing to always choosing the same few experts. Noisy top-k gating adds controlled randomness to prevent premature convergence to few popular experts. The noisy logits can be calculated as: $H(x)_{i}=(W_{g}\cdot x)i+{\text{StandardNormal}}()\cdot {\text{Softplus}}((W{\text{noise}}\cdot x)_{i})$ before selecting the top-k experts. This noise encourages exploration during training, helping achieve balanced expert utilization without solely relying on auxiliary loss functions. Selection: A TopK function is applied to the logits, which identifies the indices of the k highest-scoring experts. The logits for all non-selected experts are masked by setting them to $-\infty$. Weighting: Finally, a softmax function is applied to the masked logits to produce the final sparse gating weights $G(x)$. This normalizes the scores of the top k experts so they sum to 1, while ensuring the weights for all other experts are 0. The choice of $k$ represents a crucial engineering trade-off between computational efficiency and model expressivity. A dense MoE, where $k=N$, activates all experts and is computationally expensive. At the other extreme, the Switch Transformer pioneered Top-1 routing ($k=1$), which maximizes computational savings by activating only a single expert per token. Models like Mixtral 8x7B use Top-2 routing ($k=2$), offering a balance that allows the model to synthesize information from two specialized pathways for each token at a modest increase in computational cost compared to Top-1. This makes $k$ a critical hyperparameter that defines the model's position on the spectrum between full density and extreme sparsity. Expert Choice routing, introduced by Google Research in 2022, inverts the standard paradigm by having experts select top-k tokens rather than tokens selecting experts. This architectural innovation guarantees perfect load balancing by construction, as each expert processes exactly k tokens without needing auxiliary loss functions or capacity constraints.

Hierarchical and other variants

In a hierarchical MoE (HME), experts themselves can be MoE modules or there can be multiple layers of gating. The model forms a tree of experts: at the top level, a gating network decides which high-level expert branch to follow; each branch may itself have a gating network deciding among lower-level experts, and so on. This hierarchical structure can represent more complex partitionings of the input space. Jordan and Jacobs demonstrated that a two-level hierarchy of experts could recursively partition the input into nested regions and learn separate mappings in each region, with the EM algorithm naturally handling the tree structure. Hierarchical MoEs can be very powerful, though they add complexity. An example of a hierarchical MoE in modern usage is Meta AI's NLLB-200 language translator. Other variations of MoE include using different model types as experts (not just neural networks), or combining MoE with other ensemble techniques. For instance, one can have experts that are decision trees or SVMs, as long as a gating function can be trained to choose between them. Another variation is the mixture of experts with shared bottom layers: here, some lower-layer representation is common, and only the higher layers are divided into experts (this is common in transformer MoEs, where the self-attention layers might be shared and only the feed-forward layers are split into experts). MoEs have also been extended to contexts like reinforcement learning (to mix policies) and have parallels in mixture-of-experts decision systems in statistics.

Training

Both the experts and the gating network have trainable parameters. The MoE model is typically trained end-to-end by defining a loss function on the mixture output and minimizing it with respect to all parameters.

Training algorithms

In early formulations, training was done via gradient-based learning (backpropagation) where the gating function's outputs provide a soft assignment to each expert, allowing gradients to flow into both the gating network and the expert networks.

An alternative training approach is to use an expectation-maximization algorithm (EM): treating the selection of an expert as a latent variable, one can derive EM updates that iteratively refine the experts and gating probabilities. In the E-step, responsibilities are assigned to experts based on their current performance. In the M-step, parameters are updated based on these responsibilities. Jordan and Jacobs (1994) showed that EM can efficiently train hierarchical mixtures of experts by maximizing the likelihood of the data under the MoE probabilistic model.

Deep learning variants use backpropagation with auxiliary losses for load balancing and router z-loss for stability. MoE models are trained using gradient descent with specialized loss functions that balance the primary task objective with architectural stability requirements.

Load balancing

The load balancing problem represents MoE training's most persistent challenge, manifesting as routing collapse where all tokens concentrate on a few popular experts while others remain underutilized. A key challenge in training MoE models is routing collapse, a phenomenon where the gating network learns to favor a small subset of experts, leaving others under-trained and effectively wasting model capacity. This can lead to the problem of expert under-utilization or one expert dominating.

The traditional solution employs an auxiliary load balancing loss. Regularization terms are often added to the training objective to encourage the load to be shared among experts (for instance, penalizing the gating entropy or the variance of expert utilization). A common formulation, introduced in the "Outrageously Large Neural Networks" paper, is:

$L_{\text{aux}}=\alpha \cdot N\cdot \sum _{i=1}^{N}f_{i}\cdot P_{i}$

where:

• $N$ is the number of experts
• $\alpha$ is a tunable hyperparameter that controls the strength of the loss (typically 0.01)
• $f_{i}$ is the fraction of tokens in the batch that are routed to expert $i$
• $P_{i}$ is the average routing probability (gating value) assigned to expert $i$ across the tokens in the batch

The inclusion of this loss term creates a fundamental tension during training. The primary task loss (for example cross-entropy) pushes the router to select the expert that will yield the most accurate prediction, while the auxiliary loss pushes it toward a uniform distribution, regardless of which expert is truly best for a given token. This makes training an MoE model a multi-objective optimization problem, where successfully balancing task performance against architectural stability is critical.

For training, the loss includes the main task loss plus auxiliary terms for balancing:

$L_{\text{total}}=L_{\text{task}}+L_{\text{balance}}$

DeepSeek-V3's breakthrough in late 2024 introduced auxiliary-loss-free load balancing that separates expert selection from expertise weighting through bias terms. The mechanism applies expert-wise bias to routing scores before top-k selection, dynamically updating these biases based on recent expert load to encourage underutilized experts. Instead of adding a term to the main loss function, it directly adjusts the router's logits with a set of learnable biases. These biases are updated based on recent expert utilization, depressing the scores for overloaded experts and elevating them for underloaded ones, thereby enforcing balance without introducing conflicting gradients.

Router z-loss

Router z-loss addresses numerical instability from exponential functions in softmax computation, particularly critical when training in bfloat16 precision. The loss:

$L_{z}={\frac {1}{B}}\sum _{i}\left[\log !\left(\sum _{j}\exp !\left((\mathrm {router_{l}ogits} )_{j}\right)\right)\right]^{2}$

penalizes large logit magnitudes, keeping values in ranges where exponential functions remain numerically stable.

Distributed training

Distributed training of MoE models introduces expert parallelism as a fourth parallelism dimension beyond data, tensor, and pipeline parallelism. In expert parallelism, different devices host different expert subsets, requiring all-to-all communication to route tokens to their selected experts. This communication pattern dominates training time at scale, with costs proportional to $(\mathrm {num_{t}okens} )\times (\mathrm {capacity_{f}actor} )\times (\mathrm {num_{e}xperts} )$.

To train large MoE models, experts are typically distributed across multiple accelerator devices. The process of routing tokens from all devices to their assigned experts on other devices requires a high-bandwidth, all-to-all communication step. This communication overhead can become a significant bottleneck, increasing the actual wall-clock time per training step beyond what would be predicted by FLOPs alone.

Variants

Variant	Routing Type	Experts per Token	Key Characteristics
Sparse MoE (Top-1)	Token Choice	1	Simplest implementation, lowest overhead
Sparse MoE (Top-2)	Token Choice	2	Industry standard, balance of quality and efficiency
Fine-grained MoE	Token Choice	2-4	Many small experts, more specialization options
Soft MoE	Weighted Combination	Variable	Fully differentiable, no discrete routing
Expert Choice	Expert Choice	Variable	Perfect load balancing by construction
Hierarchical MoE	Hierarchical	Variable	Tree-structured gating for multi-level specialization
Hard MoE	Hard Selection	1	Selects only one expert per input

Sparse MoE

The sparse MoE family with top-k token choice routing represents the dominant production architecture. Top-1 routing used in Switch Transformers activates exactly one expert per token, offering the simplest implementation with lowest communication overhead. Top-2 routing employed by GShard, GLaM, ST-MoE, and Mixtral activates two experts per token, providing a strong accuracy-efficiency tradeoff that has become the de facto standard for production systems.

Fine-grained MoE

Fine-grained MoE architectures introduced by DeepSeek and Databricks use many small experts rather than fewer large experts, enabling more precise specialization and exponentially more expert combinations. DeepSeek-V3's design with 256 experts each sized at 0.25x standard expert width allows selecting from vastly more expert combinations than traditional designs.

Soft MoE

Soft MoE, introduced by Google Research in 2023, fundamentally reimagines routing by creating weighted combinations of input tokens for each expert rather than assigning discrete tokens to experts. Each expert processes a "slot" containing a learned weighted mixture of multiple tokens, with dispatch weights determining how tokens combine and combine weights determining how expert outputs merge back.

Hierarchical MoE

As described earlier, hierarchical MoE organizes experts in tree-like structures with multiple levels of gating, allowing for more complex partitionings of the input space and recursive specialization.

Adaptive Mixtures of Local Experts

The original formulation uses Gaussian distributions for experts and maximum likelihood training with EM algorithms, as introduced by Jacobs et al. in 1991.

Hard MoE

Hard MoE performs a hard selection, choosing only one expert per input rather than a weighted combination. This can make training non-differentiable, but techniques like smooth approximations or stochastic hard gating can be used.

Orthogonal MoE

Orthogonal MoE enforces orthogonality between expert weights to enhance diversity and prevent experts from becoming too similar, improving overall model performance.

Modern implementations

Switch Transformers

Switch Transformers established the trillion-parameter benchmark with 1.6 trillion total parameters across 2048 experts, demonstrating that top-1 routing with capacity factor 1.25 and router z-loss enables stable training in bfloat16 precision. The architecture achieved 7x pretraining speedup over T5-Base while maintaining quality.

Mixtral

Mixtral 8x7B revolutionized accessible AI by releasing a production-grade 46.7 billion parameter sparse MoE model under Apache 2.0 license in December 2023. With 8 experts per layer and top-2 routing activating 12.9 billion parameters per token, the model matched or exceeded GPT-3.5 performance across most benchmarks while achieving 6x faster inference than Llama 2 70B.

Mixtral 8x22B followed with 141 billion total parameters and 39 billion active, extending context to 64,000 tokens and achieving state-of-the-art results among open models on reasoning and code benchmarks.

DeepSeek-V3

DeepSeek-V3, released in December 2024, represents the current state-of-the-art in MoE architecture with multiple breakthrough innovations. The 671 billion parameter model with 37 billion activated per token introduced auxiliary-loss-free load balancing through bias-based expert selection, multi-token prediction for improved data efficiency, and FP8 mixed precision training validated at extreme scale for the first time.

Training cost only $6 million using 2.788 million H800 GPU hours, an order of magnitude less than closed-source competitors, while achieving performance comparable to GPT-4 and Claude 3.5 Sonnet on benchmarks including MMLU (88.5), MATH-500 (90.2), and code generation tasks.

GPT-4

GPT-4's architecture remains officially unconfirmed, but widespread industry speculation suggests MoE implementation based on multiple signals including non-deterministic behavior even at temperature zero, varied performance across different domains suggesting expert specialization, and cost structure significantly higher than GPT-3.5. However, Sam Altman explicitly denied specific parameter count claims, and OpenAI's GPT-4 technical report deliberately omits architectural details. This information should be treated as unverified industry speculation rather than confirmed fact.

Notable implementations table

Model	Year	Parameters (Total/Active)	Experts	Sparsity (top-k)	Developer	Notes
Switch Transformers	2021	1.6T / ~25B	2048	1	Google	First trillion-parameter MoE LLM
GLaM	2021	1.2T / ~37B	64	2	Google	Outperformed GPT-3 at 1/3 energy cost
Mixtral 8x7B	2023	46.7B / ~12.9B	8	2	Mistral AI	Open-source, outperforms GPT-3.5
Mixtral 8x22B	2024	141B / ~39B	8	2	Mistral AI	Extended context to 64K tokens
DBRX	2024	132B / ~36B	16	4	Databricks	Strong in coding and math
DeepSeek-V3	2024	671B / ~37B	256	8	DeepSeek	Auxiliary-loss-free balancing, $6M training cost
Qwen1.5-MoE	2023	14.3B / ~2.7B	60	8	Alibaba	Efficient for edge devices
Grok-1	2023	314B / ~86B	8	2	xAI	Large-scale Top-2 MoE architecture

Applications

Natural language processing

Natural language processing represents MoE's most mature application domain, with production deployments powering machine translation at Google, multilingual language models supporting 200 languages, and large language models from OpenAI, Anthropic, Google, and open-source providers. MoE has been particularly impactful in NLP and large language models. DeepSeek-V3 exemplifies current capabilities with superior performance on mathematical reasoning, competitive results on general knowledge, and strong code generation.

Computer vision

MoE models have also seen application in other domains. In computer vision, researchers have applied sparse MoE to scale up vision transformers. Riquelme et al. (2021) introduced Vision MoE (V-MoE), demonstrating an image classification model with 15B parameters using sparse expert layers for vision tasks. They found that MoE could significantly improve the scaling of vision models similar to its effect in NLP.

Computer vision applications began with Vision-MoE (V-MoE) adapting sparse routing to Vision Transformers by replacing dense FFN layers with MoE alternatives. Experts develop specialization for different visual features with some focusing on specific object types or image characteristics.

Another study by Oksuz et al. (2024) proposed MoCaE (Mixture of Calibrated Experts) for object detection, which combines multiple expert object detectors and calibrates their outputs to achieve state-of-the-art accuracy on benchmarks like COCO.

Multimodal systems

Multimodal systems represent the frontier of MoE deployment, combining text, image, audio, and video understanding through expert networks specialized in different modalities or cross-modal reasoning. MoE-LLaVA demonstrated superior visual question answering and image captioning in January 2024.

Recommendation systems

Recommendation systems at scale employ Multi-gate Mixture-of-Experts (MMoE) for multi-task learning, particularly when optimizing for multiple objectives simultaneously. Google YouTube's MMoE optimizes video recommendations for both engagement metrics and user satisfaction indicators.

These examples show that MoE is a general paradigm not limited to language models: any scenario where different subsets of the input data or feature space can benefit from specialized treatment is a potential fit for a mixture-of-experts approach.

Performance characteristics

Efficiency advantages

One key property of MoEs is that they can effectively address different subsets of a problem, which can reduce overall model complexity. Each expert can focus on a more homogeneous subset of the data, potentially modeling it better than a monolithic model would. Meanwhile, the gating network learns which inputs correspond to which expert. This can be viewed as a form of modular learning, decomposing a task into subtasks handled by different modules.

The fundamental efficiency advantage comes from sparse activation enabling constant computational cost independent of total parameter count. Mixtral 8x7B with 46.7 billion total parameters activates only 12.9 billion per token, processing each input with computational cost equivalent to a 13B dense model while accessing the capacity of a 47B parameter system.

Training efficiency improvements manifest through faster convergence to target quality metrics, with Switch Transformers demonstrating 7x speedup over T5-Base during pretraining. DeepSeek-V3's $6 million training cost represents revolutionary efficiency, using 2.788 million H800 GPU hours to train a model matching GPT-4 capabilities.

Challenges

Despite the successes, deploying MoE models also brings challenges.

Memory requirements present the primary disadvantage, as all expert parameters must reside in memory simultaneously despite sparse activation. MoE models present a distinct resource profile compared to dense models. While computationally sparse, MoE models are dense in terms of memory requirements. All expert parameters must be loaded into memory simultaneously, even though only a fraction are used for any given forward pass. This means the memory footprint is proportional to the model's total parameter count, not its active parameter count, making MoE models challenging to deploy on resource-constrained hardware. Mixtral 8x7B requires memory equivalent to a 47B dense model even though only 13B parameters process each token, while 8x22B requires 90GB+ GPU RAM.

Communication overhead in distributed training dominates time budgets through all-to-all communication patterns required for expert parallelism. To train large MoE models, experts are typically distributed across multiple accelerator devices. The process of routing tokens from all devices to their assigned experts on other devices requires a high-bandwidth, all-to-all communication step. This communication overhead can become a significant bottleneck, increasing the actual wall-clock time per training step beyond what would be predicted by FLOPs alone. DeepSeek-V3's achievement of full computation-communication overlap through FP8 quantization represents a major systems engineering breakthrough.

The Shrinking Batch Problem: Sparsity introduces a challenge for achieving high hardware utilization. For a global batch of $b$ tokens routed to $N$ experts with a Top-K of $k$, each expert receives a much smaller effective batch of approximately $(k\cdot b)/N$ tokens. Modern accelerators like GPUs achieve peak efficiency with large batches; when an expert receives a very small sub-batch, much of its computational power goes unused. This necessitates complex hybrid parallelism strategies to re-aggregate larger batches for each expert, which in turn can exacerbate communication overhead.

Training instability is another well-known issue. Early in training, if the gating network becomes imbalanced (favoring a few experts too much), some experts receive little gradient and stagnate. The complex, dynamic interplay between the router and the experts can make MoE models more susceptible to training instabilities than their dense counterparts. Shazeer et al. (2017) addressed this by adding noisy gating (a small random noise added to gating logits to encourage exploration) and a load-balancing loss term that penalizes the gate for uneven expert usage.

Fine-tuning challenges: The Switch Transformer paper noted that models with very large numbers of experts could be harder to fine-tune on downstream tasks. Furthermore, their immense parameter counts make them prone to overfitting when fine-tuned on smaller, specialized datasets. This requires careful application of regularization techniques and specialized fine-tuning strategies to adapt them to downstream tasks without degrading their performance.

Comparison with dense architectures

The choice between a sparse MoE and a traditional dense architecture involves a complex set of trade-offs across training efficiency, inference performance, and hardware requirements.

Training and inference efficiency

Training: MoE models are more FLOP-efficient during training. For a fixed computational budget (for example a set number of GPU hours), an MoE model can be trained with a much larger total parameter count than a dense model, which often translates to superior performance. However, due to communication overhead from token routing, the actual wall-clock time per training step for an MoE model can be higher than for a dense model with the same number of active parameters.

Inference: An MoE model is significantly faster (has lower latency) than a dense model with the same total parameter count. However, it is generally slower than a dense model with the same active parameter count. This is because the MoE model incurs overhead from the router computation and suffers from memory bandwidth limitations, as the parameters for all experts must be read from memory.

The "efficiency" of an MoE model is therefore highly context-dependent. In a training-centric context, where the goal is to create the most capable model for a fixed compute budget, MoE is extremely efficient. It trades memory for compute, allowing researchers to build models with far more knowledge. In a deployment-centric context, particularly on consumer hardware where memory is the primary constraint, the calculus changes. A 47B parameter MoE model requires enough VRAM to hold all 47B parameters, making it much more demanding to run than a 13B dense model, even if their inference FLOPs are similar. This makes MoE a strategic architectural choice that is best suited for scenarios where maximizing model capacity is prioritized and the necessary memory resources are available.

Architectural comparison table

Architectural Comparison: Sparse MoE vs. Dense Models

Characteristic	Dense Model	Sparse MoE Model
Parameter Activation	All parameters are used for every input token.	Only a small fraction (active parameters) are used per token.
Training FLOPs	Proportional to total parameter count.	Proportional to active parameter count (significantly lower for the same total size).
Inference FLOPs	Proportional to total parameter count.	Proportional to active parameter count (significantly lower for the same total size).
Memory Requirement	Proportional to total parameter count.	Proportional to total parameter count (high, as all experts must be loaded).
Communication Overhead (in distributed systems)	Low (primarily for standard model parallelism).	High (requires all-to-all communication for token routing).
Scalability	Scaling total parameters directly and proportionally increases compute costs.	Can scale total parameters massively with a sub-linear increase in compute costs.

Comparison with ensemble methods

Mixture of Experts fundamentally differs from traditional ensemble learning methods through input-dependent, learned routing that enables conditional computation. While MoE is considered a form of ensemble learning, it has important distinctions from traditional ensemble methods:

• Bagging creates diversity through data sampling where each model trains on random bootstrap subsets, combining predictions through averaging. MoE trains all experts on the full dataset while learning to specialize through dynamic gating.

• Boosting builds sequential models where each corrects previous errors, combining through fixed accuracy-based weights. MoE trains experts jointly in parallel with learned input-dependent weights.

• Stacking employs multiple diverse base models combined through a meta-learner, learning a static combination strategy. MoE implements dynamic, input-conditional routing with end-to-end joint training.

Traditional ensemble methods activate all models for every input, whereas MoE's conditional computation activates only selected experts, enabling massive capacity scaling impossible with dense ensembles due to computational constraints.

Recent advances

Research in Mixture of Experts is rapidly evolving, with a focus on improving routing algorithms, developing more efficient training strategies, and extending the architecture to new domains.

Auxiliary-loss-free load balancing

The auxiliary-loss-free load balancing revolution initiated by DeepSeek's research team in 2024 addresses MoE training's most persistent challenge through a fundamentally different approach. Rather than penalizing imbalanced expert usage through auxiliary loss terms, the mechanism applies expert-wise bias terms to routing scores before top-k selection. Instead of adding a term to the main loss function altogether, it directly adjusts the router's logits with a set of learnable biases. These biases are updated based on recent expert utilization, depressing the scores for overloaded experts and elevating them for underloaded ones, thereby enforcing balance without introducing conflicting gradients.

Multi-token prediction

Multi-token prediction (MTP) training emerged as a powerful technique for improving data efficiency and enabling inference acceleration through speculative decoding. Rather than training models to predict only the next token, MTP extends the objective to simultaneously predict the next D tokens, creating denser training signals from each input context.

FP8 mixed precision

FP8 mixed precision training reached production validation for the first time with DeepSeek-V3, demonstrating that extremely large models can train stably in 8-bit floating point precision through careful quantization strategies. Benefits include reduced memory consumption, lower communication overhead, and improved hardware utilization.

Innovations in routing algorithms

The router is increasingly seen not as a simple switch but as a sophisticated control system and a key site of architectural innovation. Recent advancements include:

StableMoE: A two-stage training process that first learns a balanced and cohesive routing strategy, then freezes the router for the remainder of training. This reduces "routing fluctuation" (where a token's assigned expert changes during training), which can improve sample efficiency.

Dynamic Routing: Methods that dynamically adjust the number of activated experts ($k$) based on the perceived difficulty or complexity of the input token, allowing the model to allocate more computation to more challenging inputs.

Layerwise Recurrent Routers (RMoE): Architectures that use a recurrent neural network to pass routing information between consecutive layers. This allows the router at a given layer to make a more informed decision based on which experts were activated for that same token in previous layers.

Mixture of Routers (MoR): An approach that applies the MoE concept to the router itself, using multiple sub-routers and a main router to orchestrate a more robust and fault-tolerant expert selection process.

Future outlook

The MoE paradigm continues to be a fertile ground for research and development. Key future directions include:

Hierarchical MoE: Exploring architectures where experts are themselves composed of MoE layers, a recursive application of the principle that could enable even greater scalability and specialization.

Hardware Co-design: The development of specialized hardware accelerators, compilers, and communication libraries that are explicitly designed to handle the sparse, conditional computation patterns of MoE models, which could significantly mitigate current bottlenecks.

Multimodal Applications: Extending the MoE framework beyond language to create unified models that can process diverse data types like images, audio, and video, with different experts potentially specializing in different modalities.

See also

• Ensemble learning
• Neural architecture search
• Large language model
• Transformer (machine learning model)
• Model parallelism
• Switch Transformers
• Attention mechanism
• Deep learning
• Attention (machine learning)
• Conditional computation
• Geoffrey Hinton

External links

• Hugging Face: Mixture of Experts Explained
• DeepSpeed MoE Tutorial
• Google Research: MoE with Expert Choice Routing
• DeepSeek-V3 GitHub Repository