protologue

Mixture of Experts

Also called MoE, sparse mixture of experts, sparsely-gated mixture of experts.

Mixture of experts (MoE) is a neural network architecture in which a layer contains many parallel sub-networks, called experts, and a learned router sends each token to only a few of them, so a model's total parameter count can grow much faster than the computation it uses per token.

Description

The sparsely gated mixture-of-experts layer placed thousands of experts between the layers of a language model and activated only a small number per input. The Switch Transformer simplified this by routing each token to a single expert, which reduced communication costs and made training at a trillion parameters practical. MoE language models are therefore described by two numbers, total parameters and active parameters per token. Training needs a balancing mechanism so the router does not send most tokens to a few experts, and all experts must still be held in memory at inference time. The name is unrelated to mixture-of-agents, which combines the outputs of separate models.

Sources

  1. Shazeer et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.
  2. Fedus, Zoph and Shazeer (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.

Cite this entry

Protologue. (2026). Mixture of Experts. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0229). https://protologue.com/t/mixture-of-experts/

BibTeX
@misc{protologue_mixture_of_experts,
  title = {Mixture of Experts},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0229},
  url = {https://protologue.com/t/mixture-of-experts/}
}

Markdown JSON