A group of researchers argues that a scientific theory of deep learning is actively emerging, proposing the term 'learning mechanics' to describe it. They identify five converging research directions: solvable idealized settings, tractable limits, simple mathematical laws, theories of hyperparameters, and universal behaviors across systems. The proposed framework emphasizes training dynamics, coarse aggregate statistics, and falsifiable quantitative predictions. The paper also discusses the relationship between learning mechanics and mechanistic interpretability, addresses skepticism about whether fundamental theory is possible, and outlines open research directions for newcomers.
Sort: