MemAlign is an innovative framework designed to optimize large language model (LLM) judges by closely aligning them with human feedback through a dual-memory system. This system utilizes both semantic and episodic memories, allowing MemAlign to learn effectively from as few as 2-10 natural language feedback examples. Compared to traditional methods that require extensive labeled datasets and expensive fine-tuning, MemAlign offers a significant improvement in efficiency, being 10-100× faster and cheaper, while still achieving quality comparable to or better than state-of-the-art prompt optimizers like MIPROv2, SIMBA, and GEPA.
A standout feature of MemAlign is its ability to demonstrate ‘memory scaling,’ wherein the quality of alignment improves as more feedback accumulates, eliminating the need for continuous re-optimization. This framework is currently available in open-source MLflow and on Databricks, allowing for rapid alignment of LLM judges to specific domain standards through interactive feedback loops.
Databricks has integrated MemAlign into MLflow to address the high costs and latency historically associated with training LLM judges. The dual-memory system at the heart of MemAlign captures general evaluation principles within its semantic memory, while episodic memory retains task-specific feedback from subject matter experts. This configuration paves the way for quick adaptation to new domains and evolving business needs, maintaining consistent evaluation standards.
The future of MemAlign looks promising, with potential integration into Databricks’ Agent Bricks interface. This move aims to enhance agent evaluation and governance, pushing the boundaries of cost-effective and efficient LLM evaluation capabilities.

Collections

MemAlign is a framework that optimizes LLM evaluation by aligning model judges with human feedback using a dual-memory system (semantic and episodic). It learns from just 2-10 feedback examples, operates 10-100× faster and cheaper than traditional fine-tuning methods, and demonstrates memory scaling where alignment quality improves with accumulated feedback. Now available in open-source MLflow and Databricks, it enables rapid adaptation to domain-specific evaluation standards without continuous re-optimization.

MemAlign: Enhancing LLM Evaluation with Scalable Memory and Human Feedback