# Mixture of Experts > Mixture of experts (MoE) is a neural network architecture in which a layer contains many parallel sub-networks, called experts, and a learned router sends each token to only a few of them, so a model's total parameter count can grow much faster than the computation it uses per token. - Identifier: PTL-0229 - Category: Foundations - Canonical URL: https://protologue.com/t/mixture-of-experts/ - Also known as: MoE, sparse mixture of experts, sparsely-gated mixture of experts - Introduced: 2017 ## Description The sparsely gated mixture-of-experts layer placed thousands of experts between the layers of a language model and activated only a small number per input. The Switch Transformer simplified this by routing each token to a single expert, which reduced communication costs and made training at a trillion parameters practical. MoE language models are therefore described by two numbers, total parameters and active parameters per token. Training needs a balancing mechanism so the router does not send most tokens to a few experts, and all experts must still be held in memory at inference time. The name is unrelated to mixture-of-agents, which combines the outputs of separate models. ## Related terms - [Large Language Model](https://protologue.com/t/large-language-model/) - [Foundation Model](https://protologue.com/t/foundation-model/) - [Routing](https://protologue.com/t/routing/) - [Mixture-of-Agents](https://protologue.com/t/mixture-of-agents/) ## Sources - Shazeer et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. https://arxiv.org/abs/1701.06538 - Fedus, Zoph and Shazeer (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. https://arxiv.org/abs/2101.03961 ## Cite this entry Protologue. (2026). Mixture of Experts. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0229). https://protologue.com/t/mixture-of-experts/ License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)