<p>DeepSeek’s latest paper introduces Engram, a new architectural block that sits alongside attention and feed-forward networks inside a transformer. If that framing sounds significant, it’s because it is — most transformer research tweaks existing components rather than proposing a genuinely new one.</p>
<h2>What Engram actually does</h2>
<p>The core idea is straightforward: instead of recomputing the meaning of a well-known phrase like “Diana, Princess of Wales” from scratch across multiple layers every single time it appears, Engram stores a representation of it and retrieves it on demand.</p>
<p>Think of it as a pantry. Attention and feed-forward networks are the cooking — active, compute-intensive processes. Engram is just grabbing something off the shelf.</p>
<p>Technically, it works as a hashed lookup table. When the model encounters a trigger token, it uses multi-head hashing to retrieve stored n-gram embeddings for matching multi-token patterns. A contextualized gating mechanism then filters out noisy or irrelevant retrievals before the result gets fused into the model’s representation. The lookup is deterministic, which means it can be prefetched — inference overhead ends up being only about 1.9–2.8% slower, which is negligible.</p>
<h2>How it differs from Mixture of Experts</h2>
<p>It’s worth being precise here because the distinction matters. Mixture of Experts (MoE) is conditional <em>compute</em> — the model activates different expert weights depending on the input. Engram is conditional <em>memory</em> — it fetches static stored patterns rather than running any additional computation. Scaling the hash table improves performance without increasing per-token compute at all.</p>
<h2>What the ablation studies show</h2>
<p>This is where things get interesting. The researchers ran ablation tests — removing Engram from a trained model — and found:</p>
<ul>
<li>Factual knowledge and algorithmic task performance dropped by up to 56%</li>
<li>Trivia accuracy specifically fell by 70%</li>
<li>Reading comprehension stayed at 93%</li>
</ul>
<p>That last point is telling. Reading comprehension doesn’t rely on stored facts — it works from context already present in the input. Trivia does rely on stored facts. The pattern suggests Engram is doing exactly what it’s supposed to: handling factual retrieval specifically, rather than general language understanding.</p>
<p>Ablations also found the optimal placement for Engram is at transformer layers 2 and 6, and the best budget split allocates 20–25% of sparse capacity to Engram rather than MoE layers. Replacing that slice of MoE with a simple lookup table not only improves efficiency — it improves accuracy across benchmarks, which is the genuinely surprising result here.</p>
<h2>Why this matters</h2>
<p>The practical implication is that models could become cheaper to run. If a meaningful portion of what large models do is just pattern-matching against known entities — and the ablation numbers suggest it is — then offloading that to a static lookup table rather than burning compute on it is a real efficiency gain. The authors suggest this could make capable models more viable to run locally.</p>
<p>DeepSeek V4 is expected to implement Engram at scale, so we’ll get a clearer picture of how it holds up outside controlled ablation conditions. For now, the paper makes a credible case that the transformer’s two-component architecture has room for a third.</p>


Collections

DeepSeek's Engram paper proposes a third architectural component for transformers, sitting alongside attention and feed-forward networks. It functions as a hashed lookup table that stores and retrieves multi-token pattern embeddings (e.g., named entities like 'Diana, Princess of Wales') rather than recomputing them. Unlike Mixture of Experts, which is conditional compute, Engram is conditional memory — scaling the hash table improves performance without increasing per-token compute. Ablation studies show factual knowledge tasks drop up to 56% and trivia accuracy falls 70% when Engram is removed, while reading comprehension is largely unaffected, confirming it handles factual retrieval specifically. Inference overhead is only 1.9–2.8%. Replacing 20–25% of MoE sparse capacity with Engram improves both efficiency and benchmark accuracy. DeepSeek V4 is expected to implement Engram at scale.

DeepSeek's Engram: a memory lookup table that improves transformer accuracy and efficiency