How we built Agent Builder’s memory system

LangSmith Agent Builder uses a filesystem-based memory system to give task-specific agents persistent, evolving knowledge across sessions. Memory is stored as files in Postgres but exposed to the agent as a virtual filesystem, mapping to COALA's memory taxonomy: procedural (AGENTS.md, tools.json), semantic (skill files, knowledge files), with episodic memory planned. Agents update their own memory in-the-hot-path as they work, with human-in-the-loop approval to guard against prompt injection. Key learnings include: prompting is the hardest part, agents need help compacting generalizations, and explicit memory commands are still useful. Future work includes background memory processes, a /remember command, semantic search over memory, and user/org-level memory scopes.