Transformer vs. Mixture of Experts in LLMs

Mixture of Experts (MoE) is an architecture used to enhance Transformer models by employing different 'experts' to improve performance. Transformers use feed-forward networks, while MoE models select a subset of smaller, specialized networks during inference, making operations faster. MoE faces training challenges such as some experts becoming under-trained. Solutions include adding noise to expert selection and limiting the number of tokens an expert processes. MoE models have more parameters but activate only a few during inference, leading to efficiency improvements.