Train LLMs better, without using language...?

A new AI research paper proposes pre-training language models on synthetic worlds generated by neural cellular automata before exposing them to natural language. By learning to infer hidden rules from evolving patterns (similar to Conway's Game of Life), models develop fundamental skills like pattern tracking and dependency understanding. Training on just 164 million tokens of this synthetic data improved subsequent language training by ~6% and made it up to 1.6x faster, outperforming models pre-trained on significantly more natural text. The finding suggests that teaching abstract inference before language leads to better and faster language learning.