From MHA and GQA to MLA, sparse attention, and hybrid architectures

Sebastian Raschka's Blog offers insights, tutorials, and research updates on machine learning, deep learning, and artificial intelligence. Covering topics such as neural networks, data science, and Python programming, Sebastian Raschka's Blog provides resources for students, researchers, and practitioners in the field of AI. Developers can learn about  algorithms, research methodologies, and practical applications of machine learning through Raschka's blog posts and publications.

Sebastian Raschka

A comprehensive visual guide covering the major attention variants used in modern open-weight LLMs. Starting from the fundamentals of Multi-Head Attention (MHA), it progresses through Grouped-Query Attention (GQA), Multi-Head Latent Attention (MLA), Sliding Window Attention (SWA), DeepSeek Sparse Attention (DSA), Gated Attention, and hybrid architectures combining linear/state-space modules with full attention. Each variant is explained with its motivation (primarily KV-cache efficiency and long-context scaling), trade-offs, and real-world adoption in models like Llama 3, Gemma 3, DeepSeek V3, Qwen3, and Kimi K2. The guide also discusses emerging hybrid designs (Qwen3-Next, Kimi Linear, Ling 2.5, Nemotron) that replace most attention layers with cheaper recurrent or linear mechanisms while retaining a few full-attention layers for retrieval.

A Visual Guide to Attention Variants in Modern LLMs