Agents grew up, so did our docs
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
Neon's engineering team shares their iterative journey to make documentation readable by AI agents, not just humans. Starting from hand-maintained text files that drifted out of sync, they progressed through agent detection middleware to a build-time MDX-to-Markdown conversion pipeline. Key findings from scanning 250+ doc sites: 53% serve Markdown via .md URL suffix, 41% honor Accept: text/markdown headers, 93% have llms.txt but placement varies, and only 9% return content-type-aware 404 responses. Lessons include: keeping a single canonical URL with two ways to request Markdown, structuring llms.txt hierarchically rather than as a flat list, detecting HTTP clients like axios and got (used by Claude Code and Cursor), and adding agent analytics early. The current system has four layers: build-time MDX conversion, URL rewrites, agent-detecting middleware, and structured navigation context on every page.
Table of contents
The setupPhase 1: hand-maintained text filesPhase 2: teach the site to recognize agentsPhase 3: converting MDX to MarkdownWhat other sites are doingA few more lessonsWhat the system looks like nowWhat we’d do differentlyWhat’s nextThanksSort: