DeepSeek Just Fixed One Of The Biggest Problems With AI

DeepSeek researchers introduced a technique called Engram that adds a memory retrieval mechanism (like a pantry) to transformer-based AI models. Instead of recomputing facts from scratch every time, the model can look up stored n-gram embeddings via multi-head hashing. Surprisingly, replacing 20-25% of the mixture-of-experts layers with this simple lookup table not only improves efficiency but also improves accuracy across all benchmarks. A context-aware gating mechanism filters out irrelevant retrieved memories. Ablation tests showed that disabling Engram dropped trivia accuracy by 70% while reading comprehension stayed at 93%, suggesting the model uses Engram specifically for factual storage. The authors argue this could lead to cheaper, locally-runnable AI systems.