Why agentic tests flake and how to make AI runs reliable

Here's a failure pattern teams run into the first month they try agentic testing. An AI agent runs a checkout flow and passes. The next run, against the exact same build, it clicks the wrong "Continue" button, wanders into the address form, and reports a failure. Nobody changed the code. The agent just took a different route.

If you've fought flaky scripted tests, this looks familiar and it isn't. Scripted flakiness comes from a shaky environment underneath a fixed set of steps. Agentic flakiness adds a second source on top: the agent deciding what to do next. This post is about that second source, why it's harder to debug than a stale selector, and how to make agent runs reliable enough to gate a release on.

Two layers of non-determinism, not one

A scripted test is a fixed sequence. Click this, type that, assert the total. When it flakes, the steps didn't change, so the cause is always the environment: a race condition, test data that rotted overnight, a CI runner under load. We wrote up all five of those in how to fix flaky tests, and every one of them still applies to agentic runs. Agents run in the same messy CI, hit the same shared databases, depend on the same third-party sandboxes.

What's new is the layer above the environment. An agent loop reads the page, decides on an action, takes it, checks the result, and repeats. That decide step is a model call, and model calls aren't guaranteed to return the same answer twice. Even at temperature zero the output isn't fully repeatable: Thinking Machines Lab traced this to batch-size variation in inference kernels, not a setting you can flip off. Most of the time the drift doesn't matter because the right move is obvious. The trouble starts where it isn't.

Where agents actually go off the rails

Agent flakiness isn't random. It clusters in three places, and naming them is most of the fix.

The page is ambiguous and the agent guesses

Two buttons both read "Submit." A modal covers the thing the agent was about to click. A toast fires and then vanishes before the next decision. A human resolves these without noticing. An agent has to pick, and when the page gives it two plausible moves, it won't always pick the same one. The app isn't broken. The interface just left room for interpretation, and the agent interpreted.

The assertion is vague, so the grade is soft

"Verify the order went through" is not a checkpoint, it's a mood. If you let the agent decide whether it succeeded, it grades itself on the same fuzzy criteria it used to act, and now two different failure modes hide in one instruction: a run that should have failed gets waved through, and a run that actually worked gets flagged because the confirmation copy changed. This is the single biggest source of agentic flakiness we see, and it's entirely self-inflicted.

The model changes under you

Providers ship new model versions, adjust defaults, and deprecate old snapshots on their own schedule. A suite that was rock-solid in March starts behaving differently in May because the model behind it moved, not because your app did. This is the agentic version of dependency drift, and teams that don't pin versions get surprised by it every time.

Why this is harder to debug than a broken selector

When a scripted test fails, you get a line number. The selector on line 34 didn't match. You fix line 34. The failure is a point.

An agent failure is a trajectory. The run went right for eleven steps, made a questionable call on step twelve, and everything after that was a reasonable response to a bad position. The error message at the end ("expected confirmation, found error page") tells you almost nothing about where it actually went wrong. Without a record of the whole path, you're debugging a story you can only see the last page of.

That's the real cost of the second layer. Not that agents fail more often, but that each failure takes longer to understand if you haven't set the run up to explain itself.

How to make agent runs reliable

You don't make agentic tests reliable by making the agent deterministic. You can't, and chasing it wastes time. You make the runs reliable by constraining where the non-determinism is allowed to matter. Five habits do most of the work.

Write assertions the agent can't wiggle out of

Separate acting from grading. The agent can find its own way to the checkout page, but whether the test passed should come down to concrete checkpoints you control: an order ID that matches the pattern, a total that equals the expected number, a database row that exists. Deterministic checks on a non-deterministic path. That's the whole trick, and it maps directly onto the eval thinking in our QA playbook for AI agents: define what success looks like precisely enough that a machine can rule on it.

Constrain the path where it matters

Not every step deserves the same freedom. Exploratory runs should roam. A release-gating regression should stay on the critical path and not improvise its way into the account settings. Guardrails that scope an agent to the flow it's testing cut the ambiguous guesses off at the source, because a narrower page has fewer plausible wrong turns. This is the practical side of proportional governance: match the agent's freedom to the job in front of it instead of granting it everywhere or nowhere.

Pin the model like any other dependency

Record which model version a suite runs against and change it on purpose, not by surprise. When the provider ships an upgrade, treat it like a library bump: move to the new version in a branch, run the suite, look at what shifted, then adopt it. A model change that lands silently and moves your pass rate is indistinguishable from a regression until you rule it out, so don't let it land silently.

Record the trace, always

Every agent run should leave behind the full path it took: what it saw, what it decided, what it did, step by step. When a run fails, you want to open the trajectory and find step twelve, not reverse-engineer it from a final error. This is also what turns an agent failure into something you can hand to a developer, and it's the same audit trail the EU AI Act pushes teams toward for non-deterministic systems anyway.

Tell "the agent stumbled" apart from "the app is broken"

These are different failures and they need different responses. If the agent took a wrong turn on an ambiguous page, that's a test problem: tighten the assertion, add a guardrail, clarify the goal. If the agent did everything right and the app returned a 500, that's a real bug and the point of the whole exercise. Conflating the two is how teams either lose trust in agentic testing ("it's always crying wolf") or miss real regressions in the noise. Triage every failure into one bucket or the other before you touch anything.

Run it more than once before you trust a green

The flake-rate discipline from scripted suites carries straight over, with a twist. A scripted test that passes once will almost certainly pass again on the same build. An agent test that passes once might have gotten lucky on the ambiguous step. For anything you're about to gate a release on, run it a handful of times and look at the spread. If it's green every time, ship the gate. If it's green four times out of five, you don't have a reliable test yet, you have an assertion that's too loose or a path that's too wide. Fix that before it's in your pipeline, not after it's woken someone up.

Where qtrl fits

This is the problem qtrl is built around. The agents adapt to a changing UI the way self-healing tests promise to, so a moved button doesn't break a run. Guardrails keep an agent scoped to the flow it's meant to test, so it stays on the path where you want reliability. Every run records a full trace you can open and read, and the platform builds knowledge about your product over time so the agent guesses less as it goes. Structured underneath, non-deterministic on top: that's the combination that makes agent runs trustworthy instead of just clever.

Where to start this week

Take one agentic test you don't fully trust and do two things. First, replace whatever soft "verify it worked" assertion it ends on with a concrete check the agent can't talk its way around. Second, run it ten times and count the passes. Those two moves will tell you more about why the test flakes than any amount of staring at a single failed run, and they usually turn a coin-flip test into a dependable one in an afternoon.

Frequently asked questions about agentic test flakiness

Why do my agentic tests fail without any code change? Because the agent's decision step is a model call, and model calls aren't perfectly repeatable. When a page offers more than one plausible action, the agent won't always pick the same one. The fix isn't to make the agent deterministic, it's to constrain where its choices are allowed to matter: tighter assertions, scoped guardrails, and a pinned model version.

Are agentic tests more flaky than scripted tests? They inherit every environmental cause of scripted flakiness and add one more layer: the agent's own choices. But that layer is controllable. With deterministic checkpoints and guardrails, a well-built agentic suite can be more stable than a scripted one, because it doesn't break every time a selector moves.

How do I debug an agent test that failed? Open the run's trace and find the step where it went wrong, rather than reading only the final error. Agent failures are trajectories, not single lines, so the last error usually describes a symptom several steps downstream of the actual mistake. This is why recording a full trace on every run is non-negotiable.

Does temperature zero make agent tests deterministic? No. Setting temperature to zero reduces randomness but doesn't eliminate it. Production LLMs still vary slightly between runs for reasons below your control, a result that's held up in testing across major models. Design your suite to tolerate that variance instead of assuming you can turn it off.

qtrl runs AI agents with guardrails, full run traces, and product knowledge that grows over time, so your agentic tests stay reliable enough to gate real releases on. Start free with qtrl or read the QA playbook for AI agents for the eval side of the same problem.