TurboQuant Is Way Too Overhyped

Google's TurboQuant research claims up to 6x KV cache memory reduction and 8x inference speedup, but these numbers are misleading. The 8x speedup compares 4-bit against a 32-bit unquantized baseline that no one uses in practice — modern LLM inference already uses lower precision. The actual technique works by applying random rotations to KV cache vectors to normalize their distribution, then using scalar quantization plus a 1-bit residual correction to preserve dot product accuracy. At ~3.5 bits per value, quality is nearly identical to full precision. However, the paper has methodological issues: unfair comparisons against a similar prior work called RabbitQ (run on CPU vs. GPU), dismissal of that prior work without proper analysis, and cherry-picked baselines. KV cache quantization is not new — every major LLM serving provider already uses it. The claimed 83% memory savings is relative to a theoretical baseline, not current production systems, making the stock market reaction to this announcement largely unwarranted.