The post reviews 12 influential research papers on large language models (LLMs) published throughout 2024. It covers significant advancements and methods, including the Mixture of Experts models, improvements in low-rank adaptation techniques, effective pretraining strategies, and the introduction of new scaling laws. The reviews highlight developments in LLM architectures, optimization techniques, and the use of synthetic data, emphasizing their implications for future LLM research and applications.
Table of contents
1. January: Mixtral’s Mixture of Experts Approach2. February: Weight-decomposed LoRA3. March: Tips for Continually Pretraining LLMs4. April: DPO or PPO for LLM alignment, or both?5. May: LoRA learns less and forgets less6. June: The 15 Trillion Token FineWeb Dataset7. July: The Llama 3 Herd of Models8. August: Improving LLMs by scaling inference-time compute9. September: Comparing multimodal LLM paradigms10. October: Replicating OpenAI O1’s reasoning capabilities11. November: LLM scaling laws for precision12. December: Phi-4 and Learning from Synthetic DataConclusion & OutlookSort: