The Rise of Sparse Mixtures of Experts: Switch Transformers

A deep-dive into the technology that paved the way for the most capable LLMs in the industry today

Image generated using Dall-E

Sparse Mixtures of Experts (MoE) has become a key technology in the latest generation of LLMs such as OpenAI’s GPT-4, Mistral AI’s Mixtral-8x7, and more. In a nutshell, sparse MoE is an extremely…