Without quantization, distillation and pruning. 🔥
💡 Key Features:
<ul>
<li>Supports Llama, ChatGLM, QWen, Baichuan, Mistral, InternLM</li>
<li>4-bit/8-bit compression: 3x inference speedup</li>
<li>MacOS support on Apple Silicon</li>
<li>3x speedup with compression</li>
</ul>
⚙️ Technique:
<ul>
<li>Layer-wise model decomposition</li>
<li>Block-wise quantization for compression</li>
<li>Focuses on reducing disk loading bottleneck</li>
</ul>
<hr>
layer-wise inference is essentially the “divide and conquer” approach
📌 LLMs contains many “layers.” A 70B model has as many as 80 layers. But during inference, each layer is independent, relying only on the output of the previous layer.
Therefore, after running a layer, its memory can be released, keeping only the layer’s output. Based on this concept, AirLLM has implemented layered inference.
How ❓
During inference in a Transformer-based LLM, layers are executed sequentially. The output of the previous layer is the input to the next. Only one layer executes at a time.
Therefore, it is unnecessary to keep all layers in GPU memory. We can load whichever layer is needed from disk when executing that layer, do all the calculations, and then completely free the memory after.
This way, the GPU memory required per layer is only about the parameter size of one transformer layer, 1/80 of the full model, around 1.6GB.
📌 Then using flash attention to deeply optimizes cuda memory access to achieve multi-fold speedups
📌 shard model-files by layers.
<hr>
On the con side, latency should increase hugely - accessing data from slower storage (Disk I/O is generally slower, vs having all layers in GPU memory )
📌 You’re always going to be bottlenecked by SSD read speeds, paging doesn’t help. If the SSD reads at 3GB/s and the model has 70Gb, then you’ll wait 23.3 seconds for the generation of just one token. For each token you need a full pass. Only MoE helps here, as you reduce the amount of weights that are needed to be loaded for the computation of one token.
#python #llama3 #llama #llm #machine-learning

Whether you're a beginner or an expert, join us to learn and grow together! 🌱💡
#AI #MachineLearning #Innovation #TechCommunity #LearnTogether

ML & AI

The AirLLM library enables running large language models (LLMs) like Llama3 70B and Llama3.1 405B on GPUs with minimal memory requirements. It supports multiple models and offers 4-bit/8-bit compression to significantly speed up inference. Techniques such as layer-wise model decomposition and block-wise quantization reduce memory usage and disk loading bottlenecks, although there may be increased latency due to reliance on slower disk I/O.

Llama3 70B on 4GB GPU, Llama3.1 405B on 8GB GPU with AirLLM lib.