A New Way to Handle Residual Connections in Transformers

Kimi researchers introduced Attention Residuals, a new approach to residual connections in Transformers that addresses the PreNorm dilution problem. Instead of summing all previous layer outputs with equal weight=1, each layer uses softmax attention to selectively weight contributions from prior layers. A practical variant Show more