1-Bit LLM: The Most Efficient LLM Possible?

BitNet introduces 1-bit quantization for large language models, reducing memory usage by up to 7 times and energy consumption by 12 times compared to full-precision models. The technique uses ternary weights (-1, 0, 1) instead of traditional 16-bit floating point numbers, enabling efficient matrix operations through simple addition and subtraction. Recent advances include BitNet B1.58 with sparsity support and A4.8 with 4-bit activations and 3-bit KV cache, allowing 5x larger context windows. A 2B parameter BitNet model achieves comparable performance to much larger models while requiring only 0.44GB memory footprint and costing around $1.3K to train versus $26K for traditional approaches.