Towards Data Science is a community-powered publication that showcases work in data science, machine learning and artificial intelligence. Every day newcomers, seasoned researchers and industry practitioners publish tutorials, research notes and real-world case studies that help the field move forward.

Towards Data Science

Attention mechanisms in AI transformers aren't novel inventions but rediscoveries of fundamental optimization principles. The same mathematical pattern—selective amplification combined with normalization—emerges independently across evolution (500+ million years of neural systems), chemistry (autocatalytic reactions), and AI (gradient descent). This convergence suggests attention represents a universal solution to information processing under energy constraints. Reframing attention as amplification rather than selection offers practical insights for improving AI architectures: decoupling amplification from normalization, exploring non-content-based amplification, implementing local normalization pools, and designing systems that operate at critical dynamics for optimal information processing.

We Didn’t Invent Attention — We Just Rediscovered It

Attention as Amplification: Reframing the Mechanism

Chemical Computers and Molecular Amplification

Information-Theoretic Constraints and Universal Optimization

Convergent Mathematics, Not Universal Mechanisms

<p>The author goes searching for commonalities in a wide range of systems.  He gives an example of very crude, common notions of attention–namely organisms being attentive in nature–a chemical reaction that requires output training with a tangent on energy efficiency.</p>
<p>I have no doubt that  there are commonalities across all sorts of varied phenomena.  None of these yet seem to be interesting. These all can be describe by math, optimization occurs, rates of change are essential, they are all ‘things’.  The author comes up with another one: they are all emphasizing certain bits of information as part of the processing (he calls it amplification).  I don’t find this all that interesting.  Isn’t this inherent to almost any optimization process?  LSTMs are also ‘amplifying’ certain neurons even though it’s not doing it in a global sense.  This doesn’t even have to do with attention.  It seems that the notion is so general and ill-defined that it doesn’t really seem to be all that interesting.</p>
