Smallest transformer that can add two 10-digit numbers - anadim/AdderBoard

Hacker News is a community-driven platform for sharing and discussing technology news, startups, and programming-related topics. Through user submissions and comments, Hacker News offers insights into emerging technology trends, industry developments, and entrepreneurial ventures. Readers can participate in discussions, share their insights, and stay informed about the latest advancements in technology and innovation.

Hacker News

AdderBoard is a community leaderboard tracking the smallest transformer models capable of adding two 10-digit numbers with ≥99% accuracy. Two categories are tracked: trained models (weights learned via SGD/Adam/etc.) and hand-coded models (analytically set weights as constructive proofs). The current record holders are a 36-parameter hand-coded model using ALiBi positional encoding and a 311-parameter trained model using rank-3 factorization with grokking. Key community findings include a parameter cliff around 800 params, single-layer architectures outperforming two-layer ones at equivalent budgets, and rank-3 factorization as the critical trick for trained models. The challenge originated from an experiment comparing Claude Code and Codex outputs, which the community has since dramatically improved upon.

anadim/AdderBoard: Smallest transformer that can add two 10-digit numbers