Agentic testing in CI: record the run, replay the script

Most of what gets written about AI agents running browser tests is a demo. Slack's engineering team went and measured it instead. They published an experiment covering 200+ automated executions across two real Slack flows, three execution models, and three Claude models, with each configuration run 20 times. Cost, runtime, and failure rates attached.

The numbers are worth sitting with, because they're the first public data most teams will see on the question. They also point at a conclusion that we think is only half right, and the half that's missing is the interesting part.

What they found

Agent-driven runs cost $15–30 per execution and burned 3.5M to 7M tokens. They took 5–11 minutes, against roughly 3 minutes for a generated Playwright test. Failure rates climbed with flow length: on the short Thread Reply flow, Playwright MCP failed 0% of the time, while on the longer Search Discovery flow it failed about 12%, the CLI setup about 20%, and generated tests about 48%.

Then the number that says the most about what an agent actually is: only about 20% of runs followed the exact same sequence of actions. Four runs in five took a different route to the same correct outcome.

Slack's conclusion is that agents belong at the apex of the testing pyramid, for exploring UI behavior, debugging flaky workflows, and reproducing production bugs, while deterministic tests keep doing the CI work. Their framing for the split is sharp: "Tests enforce journeys. Agents verify goals."

That's a good distinction and we'd defend it. What we don't think follows is that you have to pick one per test.

The cost lives in the loop, not in the agent

Look at where the money went. Slack traced the cost to context accumulating across turns, not to model reasoning. Every browser interaction returns an accessibility tree snapshot, every snapshot lands in the context, and every turn after that re-sends the whole conversation. Their CLI configuration needed around 85 turns where MCP needed 40, because each browser interaction split across several commands, and every extra turn pays the full system prompt plus all prior context again.

So $15–30 isn't the price of an agent testing your app. It's the price of an agent working out how to test your app from scratch, again, on every run.

The 20% figure has the same shape. It isn't measuring an unreliable agent. It's measuring one that solved the same puzzle 20 times and found several valid routes, which is exactly what you want while you're exploring and exactly what you don't want on run 400 of a suite that gates deploys. We've written before about the second layer of non-determinism agents add on top of ordinary environment flakiness. Slack just put a percentage on it.

Agents make better authors than runtimes

Here's the option the experiment didn't include. Run the agent once. Record the concrete browser actions it took on the run that succeeded. Replay those actions on every run after that, with no model in the action loop at all.

The agent writes the test. It doesn't run it.

That is not the same thing as Slack's "Generated Tests" configuration, and the difference matters because generated tests were the worst performer on the complex flow at roughly 48% failure. A generated test is a model writing Playwright code from a description of a flow it hasn't executed. It's guessing at selectors, at waits, at what the DOM looks like when it gets there. A recorded test is the sequence of Playwright calls that already worked, in a real browser, against a real build, five minutes ago. One is a prediction. The other is a transcript.

Run one still costs what an agent run costs, and it still takes agent time. You pay it once per test instead of once per execution. From run two onward you're paying for assertion checks and browser time, and the runtime lands where a scripted test lands. The action sequence comes out identical every time by construction, because it's the same list of calls, so the variance Slack measured has nowhere left to enter.

Recordings rot, and that used to be the end of the story

Record and replay has been tried. It earned its reputation in the Selenium IDE era, when a recording broke the moment anyone renamed a button and you were left maintaining a blob of coordinates and XPaths you couldn't read. Teams abandoned it for hand-written code for good reasons.

What's different now is that you have something to fall back to. When a replayed step fails, the run doesn't have to fail with it. Hand it back to the agent at that exact step, with the context of everything that already passed, let it work out the new path, and save the corrected sequence for next time. The suite repairs itself, and it only pays for model reasoning when the app actually changed.

It's the same idea behind self-healing tests, moved up a level. Instead of patching one selector, the agent re-solves the part of the journey that moved.

Where an agent is still worth the full price

Replay is the wrong tool for plenty of work, and Slack's recommendations hold up here. Exploring a feature nobody has touched yet, or chasing a bug from a vague support ticket, is work where you want the model reasoning on every step. There's no recording to replay, and the whole point is that nobody knows the path.

Which makes the useful question per test how much freedom the run should have, rather than whether an agent is involved at all. That's proportional governance in practice. An exploratory run should roam. A regression that blocks a merge should do the same thing it did yesterday.

How qtrl does this

You write test steps in plain language. On the first execution, the agent drives a real browser through Playwright MCP, works out the path, and records the raw Playwright code behind every action it takes, tagged to the step it belongs to. If the run passes, that recording becomes the active one for the test.

Every run after that replays it. The recorded actions execute in order with no model involved in deciding what to do next. The only model calls left are the assertion checks, one per step, asking whether the expected result is on screen. Same screenshots, same live logs, same step-by-step trace in the UI as an agent run, because the trace is worth as much when a test passes as when it fails.

When a step does break, replay stops there and the agent picks up from that step number with the context of the steps that already passed. It finishes the run, and the actions from the replayed portion get merged with the agent's new ones into a fresh recording. The old one is marked outdated. Next run replays the repaired path.

You turn replay on per project, so exploratory work can stay fully agentic while the suites that run on every merge get the deterministic version. Variables and secrets are injected per environment, and secrets are never exposed to the agent in either mode.

What to look at this week

Take the flow you most want in CI and least trust to an agent. Run it with an agent once and read the trace end to end. Then go through the steps and mark which ones genuinely needed judgment, and which ones are a fixed sequence of clicks that only needed a model to discover the first time.

For most regression flows, almost everything falls in the second bucket. Those steps don't need to be re-derived on every run, and once you stop re-deriving them, Slack's cost and variance numbers stop being an argument against putting agents anywhere near your pipeline.

Frequently asked questions

Can you actually run AI agents in CI? Running the full agent loop on every CI execution is hard to justify at Slack's measured $15–30 and 5–11 minutes per run. Running a recording the agent produced is a different economic question, because the reasoning happens once instead of on every pipeline trigger.

How is this different from AI-generated Playwright tests? A generated test is a model predicting the code from a description. A recorded test is the transcript of calls that already worked in a real browser. In Slack's data, generated tests failed roughly 48% of the time on the complex flow, which is the cost of guessing at selectors you haven't seen.

What happens when the UI changes? The replayed step fails, the agent takes over from that step, finishes the run, and the recording is rewritten with the corrected path. You pay for model reasoning on the runs where something moved, not on the ones where nothing did.

Do I still need deterministic tests? Yes, and that's the point. A replayed recording is a deterministic test. What changed is who writes it and who fixes it when it breaks.

qtrl runs AI agents that record what they did, then replay it deterministically until your app changes. Start free with qtrl or read up on how agentic testing works first.