Best of Deep Learning — July 2025
- 1
- 2
Daily Dose of Data Science | Avi Chawla | Substack·44w
4 Stages of Training LLMs from Scratch
Training large language models from scratch involves four key stages: pre-training on massive text corpora to learn language basics, instruction fine-tuning to make models conversational and follow commands, preference fine-tuning using human feedback (RLHF) to align with human preferences, and reasoning fine-tuning for mathematical and logical tasks using correctness as a reward signal. Each stage builds upon the previous one to create increasingly capable and aligned AI systems.
- 3
Prompting vs. RAG vs. Finetuning
A decision framework for choosing between prompt engineering, RAG, and fine-tuning when building LLM applications. The choice depends on two key factors: the amount of external knowledge required and the level of model adaptation needed. RAG works best for custom knowledge bases without behavior changes, fine-tuning modifies model structure and behavior, prompt engineering suffices for basic adjustments, and hybrid approaches combine RAG with fine-tuning for complex requirements.
- 4
C/C++ Community·46w
🚀 Built a Neural Network Library in C++ from Scratch - Here's What I Learned About the Fundamentals Behind ML Frameworks
A developer shares their experience building a neural network library in C++ from scratch over two weeks to understand the fundamentals behind ML frameworks like TensorFlow and PyTorch. The project includes dense layers, various activation functions, SGD optimizer with momentum, batch training pipelines, and dataset support. Key insights include the challenges of gradient debugging, importance of memory management in ML contexts, and how implementing algorithms from scratch provides deeper understanding than high-level tutorials. Future plans include adding tensor datatypes, convolutional layers, and additional optimizers.
- 5
bycloud·44w
POV: Chinese AI Lab Teaching Everyone How To Save Millions of Dollars
ByteDance's AI lab has published research on Pre-trained Model Averaging (PMA), a technique that merges model checkpoints during training to predict final performance while saving 15% of compute budget. The method averages snapshots taken at fixed intervals during the constant learning rate phase, effectively achieving similar results to traditional annealing without the computational cost. Testing on models from 411M to 70B parameters showed 3-7% accuracy gains and potential savings of millions in training costs. The technique also provides crash recovery capabilities and early performance estimates for hyperparameter optimization.
- 6
Community Picks·47w
Train a Convolutional Neural Network from Scratch: PyTorch, Next.js, React, Tailwind, Python (2025)
A comprehensive tutorial covering the complete process of building a convolutional neural network from scratch using PyTorch to classify audio files. The guide starts with neural network fundamentals including neurons, activation functions, and training concepts like forward pass, backpropagation, and loss optimization. It then dives deep into CNN theory, explaining kernels, feature maps, spatial information preservation, and how CNNs extract hierarchical features from images. The practical implementation includes converting audio to spectrograms, training on serverless GPUs with Modal, achieving 83% accuracy, and building a Next.js frontend to visualize the model's convolutional layer outputs and feature extraction process.
