How to catch spec drift when AI agents write the code

A spec-driven feature usually starts clean. Someone writes the spec, the agent builds from it, the tests are generated from the same criteria, and everything lines up. Then a week later somebody asks the agent to tweak the flow. It edits the code. The tests get updated to match. Nobody opens the spec.

Do that a few times and you have spec drift. Hartwig Grabowski's recent paper on spec-anchored development describes it well: the code evolves, the specification doesn't, and the gap stays invisible until it's expensive to fix. Agents make the code side move faster than ever, so the gap opens faster too.

Writing more spec won't close that gap. What helps is making divergence visible, so a mismatch shows up as a failed check instead of a surprise in production. Here's how to set that up.

Know which way it's drifting

"Drift" covers three different problems, and each one needs a different check.

• Code drifts from the spec. The behavior changes, the spec stays put. This is the one people usually mean
• The spec moves and the tests don't. Someone updates a criterion, and the suite keeps checking the old one
• Tests drift toward the code. A test fails, and the expected result gets edited until it passes. Of the three, this one does the most damage, because it turns your check into a recording of whatever the code does now

The steps below cover all three. Most of them are cheap. The hard part is doing them consistently.

Step 1: Give every criterion a stable ID

You can't trace what you can't name. Give each acceptance criterion an ID that won't change, like the Jira key plus a number (PROJ-412-AC3). If a criterion is removed, retire its ID. Never renumber, because every link to the old number breaks silently.

The best time to do this is during the spec review, before anything is built. Our guide on how to review a spec before the agent builds it covers the rest of that review.

Step 2: Tag every test with what it checks

Each test should name the criteria it verifies. In Playwright, tags and annotations do the job:

test('expired coupon shows an error', {
  tag: '@PROJ-412',
  annotation: { type: 'criterion', description: 'PROJ-412-AC3' },
}, async ({ page }) => {
  // ...
})

In qtrl, each test case has a requirement ID field, and tests generated from a Jira issue start from that issue's acceptance criteria. Whatever you use, the rule is the same: an untagged test is a test nobody can trace, and it'll be the first one to drift.

Step 3: Check the links in CI

With IDs on both sides, two checks catch a lot. Run them on every pull request that touches the spec or the tests.

Criteria with no test. Every active criterion in the spec should be referenced by at least one test. A new criterion with nothing pointing at it means a requirement went in untested. A short script can do this: pull the IDs out of your spec files, list your tests with their tags (Playwright's --list flag with the JSON reporter works), and fail when a criterion has no match.

Tests pointing at criteria that no longer exist. These are orphans. Either the criterion was removed and the test should go too, or the criterion was reworded under a new ID and the test is now checking a requirement nobody holds. Both need a decision.

That gives you requirements coverage by criterion instead of by line. Our post on measuring test coverage beyond code coverage shows how to report it.

Step 4: Protect the expected results

This step handles the third kind of drift, and it's mostly a review rule. When a pull request changes an expected result in a test, it should also link to a spec change that justifies it. If the spec didn't change, why did the expectation?

You can back that up with tooling. Put the spec and test folders under CODEOWNERS so QA sees every change. Flag pull requests where an agent edited the code and the assertions that check it in the same commit. That combination isn't always wrong, but it's exactly the shape a drifting test takes.

qtrl builds the same rule into execution. On a test's first, adaptive run, the agent can adjust the steps to fit the real UI, but it isn't allowed to change the expected results, and every change it makes is logged as a deviation you can review. Once a test is approved, runs are strict. If a step can't be done as written, the step fails and says why. Self-healing handles the button that moved. The spec decides what counts as a pass.

Step 5: Record which spec version each run passed against

A green run only means something if you know what it was checked against. Store the spec's version with every run: the commit SHA of the spec file, or the last-updated time of the Jira issue. When something breaks later, the question "what changed since the last pass?" becomes a diff between two versions of the spec instead of an argument.

Version the tests the same way. qtrl keeps a version history for every test case, with an audit log of who changed what, so you can line up which version of a test checked which version of a requirement.

Step 6: When the spec changes, review the tests first

Spec changes should trigger test review, not wait for it. When a criterion changes, find every test tagged with its ID and look at them before the next run. If the expected results no longer match, update the tests in the same pull request as the spec, so both readings move together and the history explains why.

Some SDD tools already help here. OpenSpec's change model, for example, merges a change's spec deltas into the main specs when you archive it, which keeps the spec current as the code changes. That handles one direction. The tags from step 2 handle the other.

Step 7: Run the spec-derived suite on every deploy

Everything so far checks the links between documents. The final check is behavioral: run the tests generated from the spec against the deployed app, on every change that reaches staging. If the code drifts in a way nobody noticed in review, a strict test checking the original criterion goes red.

That's the drift detector that doesn't depend on anyone remembering to update anything. Our guides to agentic testing in CI and fixing staging environment drift cover how to keep those runs fast and trustworthy.

When the test goes red, decide which side moved

A failing spec-derived test means the code and the spec disagree. It doesn't tell you which one is right. Sometimes the code drifted and should be fixed. Sometimes the team changed its mind and the spec never caught up, in which case you update the spec first and the test second. The one move to avoid is editing the expected result just to get back to green.

One thing no drift check can catch is a spec that drifted away from what users need. A perfectly traced suite still only checks what you wrote down. That's what exploratory testing is for, and it gets more useful as more of the scripted checking is generated.

qtrl keeps the thread from requirement to result: requirement IDs on every test, versioned test cases with an audit log, expected results that adaptive runs can't rewrite, and strict runs against each environment with screenshots and video. When the code drifts, you see it in the next run. Set it up on one feature.