Thanks to the release of Mixtral, the Mixture of Experts (MoE) architecture has become popular in recent months. This architecture offers an interesting tradeoff: higher performance at the cost of…

Towards Data Science is a community-powered publication that showcases work in data science, machine learning and artificial intelligence. Every day newcomers, seasoned researchers and industry practitioners publish tutorials, research notes and real-world case studies that help the field move forward.

Towards Data Science

This post introduces the Mixture of Experts (MoE) architecture and explains how frankenMoEs can be created using the MergeKit library. It explores the benefits and challenges of MoEs and provides a step-by-step guide for creating a frankenMoE. The post also highlights the performance of a specific frankenMoE model called Beyonder-4x7B-v3.

Create Mixtures of Experts with MergeKit