In this video we go back to the extremely important Google paper which introduced the Mixture-of-Experts (MoE) layer with authors including Geoffrey Hinton.  
The paper is titled "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer". MoE today is widely used in various top Large Language Models and interestingly, it was published at the beginning of 2017, while the Attention All You Need paper which introduced Transformers was published later that year, also by Google. It this video the purpose is to understand why the Mixture-of-Experts method is important and how it works.

Paper page - https://arxiv.org/abs/1701.06538 
Blog post - https://aipapersacademy.com/mixture-of-experts/

-----------------------------------------------------------------------------------------------
✉️ Join the newsletter - https://aipapersacademy.com/newsletter/

Become a patron - https://www.patreon.com/aipapersacademy

👍 Please like & subscribe if you enjoy this content
-----------------------------------------------------------------------------------------------

Chapters:
0:00 Why MoE is needed?
1:33 Sparse MoE Layer
3:41 MoE Paper's Figure

AI Papers Academy

Mixture-of-Experts (MoE) is a technique that increases model capacity without proportional computational cost by routing each input token to only a subset of specialized sub-networks called experts. Originally introduced in a 2017 Google paper ('Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer') — predating Transformers and co-authored by Geoffrey Hinton — MoE uses a gating/router component to select which experts process each token. Only the chosen experts run for a given input, enabling parallel execution and reduced compute. The outputs of selected experts are combined via a weighted sum determined by the gating network, and all components are trained jointly.

Introduction to Mixture-of-Experts | Original MoE Paper Explained