MemLong introduces a novel memory-augmented retrieval mechanism to tackle the limitations of handling long contexts in Large Language Models (LLMs). By integrating an external retrieval system, MemLong efficiently extends context length without sacrificing model performance. It stores past contexts in a memory bank, retrieves relevant historical data during text generation, and maintains distributional consistency. The approach significantly reduces training costs and has shown superior performance in long-context benchmarks, managing up to 80,000 tokens on a single GPU.
Sort: