How AI Agents Revolutionized Documentation Testing for Drasi

For open-source projects, documentation is the first impression. When commands fail or steps are unclear, developers leave—not file bugs. Drasi, a CNCF sandbox project for detecting data changes, faced this challenge. Their small team of four engineers from Microsoft Azure's Office of the CTO moved fast, shipping code faster than they could test tutorials manually. A critical incident in late 2025—when a GitHub Dev Container update broke all tutorials—forced them to rethink. They turned documentation testing into a monitoring problem using AI agents. Below, we explore how they did it and what lessons they learned.

1. What is Drasi, and why is its documentation so important?

Drasi is an open-source tool that detects changes in your data and triggers immediate reactions—like a smart event-driven system. For a project like this, the getting-started guide is the first real interaction most developers have. If a command fails or an output doesn't match, new users don't file bugs; they just leave. That's especially risky for a CNCF sandbox project trying to build a community. The team knew they had comprehensive tutorials, but they were shipping code faster than they could manually test each step. This gap became a crisis when an upstream dependency broke every tutorial, effectively locking out new developers during a critical window.

2. What are the two main reasons documentation breaks?

Documentation breaks for two primary reasons. First is the curse of knowledge: experienced developers write with implicit context. They say “wait for the query to bootstrap,” assuming the reader knows to check the status or run a specific command. A new user—or an AI agent—reads instructions literally and gets stuck. Second is silent drift: unlike code, documentation doesn't fail loudly. If you rename a config file in your codebase, the build fails immediately. But if your docs still reference the old filename, nothing alerts you. The drift accumulates until a user reports confusion—or gives up entirely. For Drasi's tutorials, which spin up Docker, k3d, and sample databases, silent drift is compounded by upstream changes like deprecated flags or version bumps.

3. What incident forced Drasi to change their approach?

In late 2025, GitHub updated its Dev Container infrastructure, bumping the minimum Docker version. This update broke the Docker daemon connection—and every single tutorial stopped working. Because the team relied on manual testing, they didn't immediately know the extent of the damage. Any developer trying Drasi during that window hit a wall. This was a wake-up call. It forced the realization that with advanced AI coding assistants, documentation testing could be converted from a manual, reactive task into a proactive monitoring problem. Instead of waiting for breakage reports, they could detect failures automatically.

4. How did Drasi solve the documentation testing problem?

Drasi built an AI agent that acts as a “synthetic new user.” This agent is designed with three critical characteristics: it is naïve—having no prior knowledge of Drasi beyond what's in the tutorial; it is literal—executing every command exactly as written, failing if a step is missing; and it is unforgiving—verifying every expected output and flagging any discrepancy immediately. By running this agent against each tutorial, they could monitor documentation health continuously. The agent simulates a fresh user experience, catching gaps like ambiguous instructions, missing steps, or outdated outputs before real users encounter them.

5. What tools did Drasi use to build this AI agent?

The solution was built on GitHub Copilot CLI and Dev Containers. GitHub Copilot CLI provides the AI capabilities to interpret and execute commands from the tutorial. Dev Containers offer a reproducible, isolated environment that simulates a fresh user setup. Together, they create a testing sandbox where the agent can run through tutorials without affecting real systems. The team also integrated this into their CI/CD pipeline, so every time documentation or code changes, the agent automatically tests the tutorials. This turns documentation validation from a manual chore into an automated, continuous process.

6. What key lessons can other projects learn from Drasi's approach?

First, documentation testing should be treated like code testing. Use automated agents to simulate real user journeys and catch breakage early. Second, the curse of knowledge is real—assume nothing. Write tutorials so literally that even an AI agent can follow them without confusion. Third, silent drift is dangerous. Monitor upstream dependencies and test regularly to prevent invisible breakage. Finally, leverage AI tools like GitHub Copilot to build synthetic users that are cheaper and faster than manual testing. For small teams shipping fast, this approach transforms documentation from a liability into a reliable asset.