How TurboQuant compresses the KV cache without losing accuracy, and what that could mean for memory stocks

Hacker News is a community-driven platform for sharing and discussing technology news, startups, and programming-related topics. Through user submissions and comments, Hacker News offers insights into emerging technology trends, industry developments, and entrepreneurial ventures. Readers can participate in discussions, share their insights, and stay informed about the latest advancements in technology and innovation.

Hacker News

Google released TurboQuant, a two-stage KV cache compression algorithm that achieves a 6x reduction in memory usage with no measurable accuracy loss. Stage 1 (PolarQuant) converts vectors from Cartesian to polar coordinates, exploiting the predictable angular distribution in transformer key spaces to compress without calibration data. Stage 2 (QJL) applies a Johnson-Lindenstrauss transform to correct quantization error at zero memory overhead. The result is 3.5 bits per channel with quality neutrality across major models, and up to 8x performance improvement on H100 GPUs. Unlike other quantization methods, TurboQuant is data-oblivious and requires no fine-tuning. Beyond LLMs, it shows promise for vector databases, RAG pipelines, recommendation engines, and on-device inference. The announcement caused memory stock prices (Micron, SanDisk) to drop, raising questions about whether AI's memory demand will grow as linearly as previously assumed.

@adlrocha - What if AI doesn’t need more RAM but better math?