Researchers propose the Fractured Entangled Representation Hypothesis, arguing that conventional neural networks trained with stochastic gradient descent produce chaotic, entangled internal representations despite good performance. In contrast, compositional pattern producing networks evolved through human-guided selection create elegant, modular representations with clear factorization of concepts. The paper demonstrates that networks can achieve identical outputs through radically different internal structures, suggesting that how knowledge is acquired matters as much as what is learned. This challenges the assumption that good benchmark performance indicates good internal representations, with implications for creativity, generalization, and out-of-distribution performance in AI systems.
Sort: