The Drasi team (a CNCF sandbox project at Microsoft Azure) built an AI-powered documentation testing system using GitHub Copilot CLI, Dev Containers, GitHub Actions, and Playwright. The system deploys a 'synthetic user' agent that reads tutorials literally, executes every command, and verifies expected outputs — mimicking a new developer with no prior context. Running weekly across isolated containers, the agent found 18 distinct issues including implicit dependencies, missing verification steps, and format drift. The security model isolates the agent inside ephemeral Dev Containers with minimal PAT permissions. The approach converts documentation maintenance from a manual chore into an automated monitoring problem, effectively giving a four-person team a tireless QA engineer.
Table of contents
The problem: Why does documentation break?The solution: Agents as synthetic usersWhat we found: Bugs that matterAI as a force multiplierSort: