LLM’s Billion Dollar Problem

Token consumption in LLMs has exploded with thinking models and AI agents, creating scalability challenges. Standard attention mechanisms scale quadratically with context length, making long contexts prohibitively expensive. Three approaches attempt to solve this: sparse attention (restricts which tokens interact), linear attention (accumulates information in shared memory), and compressed attention (compresses tokens before comparison). While sparse and compressed attention help, only linear attention can theoretically scale past 1M context windows. Recent developments show hybrid approaches combining linear attention with standard or compressed attention achieving promising results, with Google's Gemini 3 Flash demonstrating breakthrough performance at 1M context length.