Robert Youssef

Kimi's research identifies a fundamental inefficiency in transformer architectures: residual connections use fixed uniform weights, causing hidden state magnitudes to grow with depth and making deeper layers progressively less influential. The proposed fix, called Block AttnRes, replaces fixed residual additions with learned softmax attention over preceding layer outputs using a single learned query vector per layer. To manage memory overhead at scale, layers are partitioned into blocks and attention is computed over block-level summaries rather than individual layers. Results on a 48B parameter model trained on 1.4T tokens show equivalent validation loss to a baseline trained with 1.25× more compute, with consistent benchmark improvements across GPQA-Diamond (+7.5 pts), Math (+3.6 pts), and HumanEval (+3.1 pts). Training overhead is under 4% and inference latency overhead under 2%. The finding also shifts the optimal architecture under fixed compute budgets toward deeper, narrower networks.

Holy shit... Kimi just found a design flaw that has been inside every LLM since 2015.

> Residual connections use fixed uniform weights every layer treated identically, no matter what came before. Replace them with learned attention over preceding layers and you get 1.25x more compute for free. Same model. Same data. Different architecture.

> Standard residual connections have been the default building block of every transformer since He et al. 2015. The update is simple: each layer adds its output to the previous hidden state with a fixed weight of 1. Unroll this recurrence and you get the same thing every time a uniform sum of all prior layer outputs, with no mechanism to weight some layers more than others. Early layers and late layers contribute equally regardless of what the model actually needs. The result: hidden state magnitudes grow proportionally with depth, each layer's relative contribution shrinks, and a significant fraction of layers can be pruned with minimal loss. The architecture was built as a gradient highway. Nobody noticed it was also a bottleneck.

> The insight is a direct analogy to sequence modeling. RNNs had the same problem over time compressing all prior information into a single fixed-weight state. The transformer solved it by replacing recurrence with attention, letting each position selectively access all previous positions with learned, input-dependent weights. Kimi applied the exact same logic to depth. Instead of a fixed uniform sum across layers, each layer now computes softmax attention over all preceding layer outputs using a single learned query vector per layer. One d-dimensional vector. That's the entire overhead.

> The practical challenge: at scale, storing all preceding layer outputs for every layer requires O(Ld) memory and creates communication overhead under pipeline parallelism. Kimi's solution is Block AttnRes partition layers into blocks, compress each block to a single representation, attend over block-level summaries instead of individual layers. Memory drops from O(Ld) to O(Nd). With 8 blocks, Block AttnRes recovers almost all the gains of the full version. Infrastructure overhead under pipeline parallelism: less than 4%. Inference latency overhead: less than 2%.

The numbers from the 48B model trained on 1.4T tokens:
→ Block AttnRes reaches the same validation loss as baseline trained with 1.25× more compute
→ GPQA-Diamond: 36.9% → 44.4% (+7.5 points)
→ Math: 53.5% → 57.1% (+3.6 points)
→ HumanEval: 59.1% → 62.2% (+3.1 points)
→ MMLU: 73.5% → 74.6%
→ Improvements consistent across all 15 evaluated benchmarks zero regressions
→ Scaling laws confirm gains hold across every model size tested
→ Training overhead under pipeline parallelism: less than 4%
→ Inference latency overhead: less than 2%
→ Only addition: one RMSNorm and one learned vector per layer

> The training dynamics analysis explains why it works. Standard residuals force deeper layers to produce increasingly large outputs just to maintain influence over the accumulated hidden state the PreNorm dilution problem. Block AttnRes resets this accumulation at block boundaries.

> Output magnitudes stay bounded across depth. Gradient norms distribute more uniformly across layers. The model stops wasting capacity compensating for a structural inefficiency and uses it for actual learning.

> The architecture sweep reveals something deeper. Under a fixed compute budget, the optimal architecture shifts when you add AttnRes. The baseline prefers wider, shallower networks. AttnRes shifts the optimum toward deeper, narrower ones because with selective depth-wise attention, additional layers are no longer diluted by uniform accumulation.

> The inefficiency that made depth expensive to exploit is gone.

> Residual connections have been in every transformer for a decade. Nobody questioned the fixed weights until now. One learned vector per layer changes the math entirely.

<p>Holy shit... Kimi just found a design flaw that has been inside every LLM since 2015.

&gt; Residual connections use fixed uniform weights every layer treated identically, no matter what came before. Replace them with learned attention over preceding layers and you get 1.25x more compute for free. Same model. Same data. Different architecture.

&gt; Standard residual connections have been the default building block of every transformer since He et al. 2015. The update is simple: each layer adds its output to the previous hidden state with a fixed weight of 1. Unroll this recurrence and you get the same thing every time a uniform sum of all prior layer outputs, with no mechanism to weight some layers more than others. Early layers and late layers contribute equally regardless of what the model actually needs. The result: hidden state magnitudes grow proportionally with depth, each layer's relative contribution shrinks, and a significant fraction of layers can be pruned with minimal loss. The architecture was built as a gradient highway. Nobody noticed it was also a bottleneck.

&gt; The insight is a direct analogy to sequence modeling. RNNs had the same problem over time compressing all prior information into a single fixed-weight state. The transformer solved it by replacing recurrence with attention, letting each position selectively access all previous positions with learned, input-dependent weights. Kimi applied the exact same logic to depth. Instead of a fixed uniform sum across layers, each layer now computes softmax attention over all preceding layer outputs using a single learned query vector per layer. One d-dimensional vector. That's the entire overhead.

&gt; The practical challenge: at scale, storing all preceding layer outputs for every layer requires O(Ld) memory and creates communication overhead under pipeline parallelism. Kimi's solution is Block AttnRes partition layers into blocks, compress each block to a single representation, attend over block-level summaries instead of individual layers. Memory drops from O(Ld) to O(Nd). With 8 blocks, Block AttnRes recovers almost all the gains of the full version. Infrastructure overhead under pipeline parallelism: less than 4%. Inference latency overhead: less than 2%.

The numbers from the 48B model trained on 1.4T tokens:
→ Block AttnRes reaches the same validation loss as baseline trained with 1.25× more compute
→ GPQA-Diamond: 36.9% → 44.4% (+7.5 points)
→ Math: 53.5% → 57.1% (+3.6 points)
→ HumanEval: 59.1% → 62.2% (+3.1 points)
→ MMLU: 73.5% → 74.6%
→ Improvements consistent across all 15 evaluated benchmarks zero regressions
→ Scaling laws confirm gains hold across every model size tested
→ Training overhead under pipeline parallelism: less than 4%
→ Inference latency overhead: less than 2%
→ Only addition: one RMSNorm and one learned vector per layer

&gt; The training dynamics analysis explains why it works. Standard residuals force deeper layers to produce increasingly large outputs just to maintain influence over the accumulated hidden state the PreNorm dilution problem. Block AttnRes resets this accumulation at block boundaries.

&gt; Output magnitudes stay bounded across depth. Gradient norms distribute more uniformly across layers. The model stops wasting capacity compensating for a structural inefficiency and uses it for actual learning.

&gt; The architecture sweep reveals something deeper. Under a fixed compute budget, the optimal architecture shifts when you add AttnRes. The baseline prefers wider, shallower networks. AttnRes shifts the optimum toward deeper, narrower ones because with selective depth-wise attention, additional layers are no longer diluted by uniform accumulation.

&gt; The inefficiency that made depth expensive to exploit is gone.

&gt; Residual connections have been in every transformer for a decade. Nobody questioned the fixed weights until now. One learned vector per layer changes the math entirely.</p>

A tweet claims that Kimi (an AI model) discovered a fundamental design flaw in residual connections used in large language models since 2015, specifically related to fixed uniform weighting. The content is extremely thin — it's a retweet fragment with no technical details, explanation, or verification provided.