Best of Neural Networks — February 2025

1. 1
   Video

   Artem Kirsanov·1y

   Are There Limits What Brains Can Learn?

   Human brains are exceptional at learning new skills, but there are intrinsic limitations in neural circuits that can make certain patterns and behaviors impossible to master. A recent study reveals that our brain's physical wiring creates preferred pathways for neural activity, indicating fundamental constraints that neither strong motivation nor extensive practice can overcome. Understanding these limits could explain why some skills feel natural while others seem unattainable, emphasizing the biological nature of our learning capabilities.

   103

   1
2. 2
   Video

   Hacker News·1y

   Deep Dive into LLMs like ChatGPT

   Large language models (LLMs) such as ChatGPT are built through a complex pre-training process involving the downloading and processing of large quantities of diverse, high-quality internet texts. Common Crawl data, along with filtering steps like URL filtering, text extraction, and language filtering, are critical components. Tokenization converts these texts into a sequence of symbols for neural networks to process. These networks are trained to model the statistical relationships between tokens to predict the next token in a sequence. Inference is generating new data from the trained model by predicting subsequent tokens based on a given input.

   12
3. 3
   Article

   Daily Dose of Data Science | Avi Chawla | Substack·1y

   Transformer vs. Mixture of Experts in LLMs

   Mixture of Experts (MoE) is an architecture used to enhance Transformer models by employing different 'experts' to improve performance. Transformers use feed-forward networks, while MoE models select a subset of smaller, specialized networks during inference, making operations faster. MoE faces training challenges such as some experts becoming under-trained. Solutions include adding noise to expert selection and limiting the number of tokens an expert processes. MoE models have more parameters but activate only a few during inference, leading to efficiency improvements.

   11