<p>Certainly one of the most jaw-dropping LLM projects I’ve come across lately! 🌟</p>
<h2>🚀 Introduction</h2>
<p><code>TinyLlama</code> is setting its sights on an awe-inspiring goal: to pretrain a 1.1B Llama model on a staggering 3 trillion tokens – yes, you read that right, 3 trillion! 🚀</p>
<p>The audacious plan? Achieving this monumental feat in just 90 days, powered by 16 A100-40G GPUs 🚀🚀, all finely tuned for optimal performance.</p>
<p>The epic journey commenced on September 1, 2023, and you can watch its live convergence (check out the link below).</p>
<p>Jump right into the code 🚧</p>
<p><a href="https://twitter.com/ItakGol/status/1698770915759178224?s=20" target="_blank" rel="noopener nofollow">TinyLlama 🦙</a></p>
<h2>🏛️ Architecture</h2>
<p>TinyLlama has faithfully embraced the same architecture and tokenizer as Llama 2. This means TinyLlama can seamlessly integrate into a multitude of open-source projects rooted in Llama’s foundation.</p>
<p>What’s more, TinyLlama boasts a sleek profile with only 1.1B parameters. This streamlined design equips it to serve an array of applications with limited computational and memory requirements.</p>
<h2>📚 Dataset</h2>
<p>The dataset is a treasure trove of 3 trillion tokens, carefully curated from a blend of 70% Slimpajama and 30% Starcoderdata, with the GitHub subset of Slimpajama excluded.</p>
<h2>⚙️ Optimizations</h2>
<p>The team has harnessed an array of cutting-edge optimization techniques, including flash attention 2, fused layernorm, fused swiglu, fused cross-entropy loss, and fused rotary positional embedding.</p>
<p>These optimizations yield a blazing throughput of 24k tokens per second per A100-40G GPU, achieving an astonishing 56% model FLOPs utilization (a truly remarkable achievement! 💯).</p>
<p>This could be a game-changer for end devices, as the weight of the 4-bit-quantized TinyLlama-1.1B model occupies a mere 550MB of RAM! 💥</p>
<h2>📊 Status</h2>
<p>Keep an eye out for intermediate checkpoints, as they’ll be released according to the schedule provided below. The first checkpoint already has competition with StableLM-Alpha-3B &amp; Pythia-1B. 🚀🏁</p>
<h3>a proud father 😄</h3>
<p><img src="https://media.daily.dev/image/upload/s--4daXV-cs--/f_auto,q_auto/v1694036505/ugc/content_90494d0c-003d-4439-9be5-3a8e15df2c7c" alt="PhotoReal_Code_LLaMA_is_the_King_0 (1)"></p>
<p>Article Source: <a href="https://twitter.com/ItakGol/status/1698770915759178224?s=20" target="_blank" rel="noopener nofollow">https://twitter.com/ItakGol/status/1698770915759178224?s=20</a></p>
<p><code>:: all photos generated by me</code></p>


The official daily.dev AI community. Led by thought leaders and AI experts. Stay tuned for insights, news and discussions. 

Introducing TinyLlama🦙

<p>Impressive! Blending vast datasets and smart optimizations seems game changing. Curious to see how it’ll shape up against other models</p>
