Insights7 min read

Do you still need to convert manual test cases into scripts?

By qtrl Team · Engineering

Open almost any test management tool and you'll find a pile of manual test cases nobody got around to automating. They're written in plain language: open the settings page, change the billing email, check the confirmation arrives. Turning them into Playwright or Selenium code was always planned, and it always lost to whatever shipped this sprint.

On November 16, Katalon launched an AI Test Runner built for exactly that pile. Its announcement says AI agents "run the natural-language test cases teams already built", with "no scripts to write". Katalon isn't alone. Autify, testRigor and TestMu's KaneAI all describe some version of running plain-language tests with AI. So the question is fair: if an agent can run the case as written, do you still need to turn it into a script at all?

Our answer: you don't need to write the script by hand anymore. But you still want something deterministic to run on every merge, and the work you used to spend on scripting moves into the test cases themselves.

Three ways to automate a manual case

"Automating a manual test with AI" covers three quite different things, and vendors don't always say which one they're doing.

Three ways to automate a plain-language test caseWhat you keepEach run after the firstWatch forGenerate a scriptCase → model writes code→ code runs
Test code you own
Script speed, no model
You maintain the code as the app changes
Agent runs the caseCase → agent drives the appon every run
The case and the evidence
Model cost and time on every run
Route can differ run to run
Record, then replayCase → one agent run→ recorded actions replay
The case plus a recording that worked
Replayed actions, no model choosing steps
Recording needs repair when the UI changes

Generate a script. A model reads the case and writes test code, which you then run like any other test. Playwright's test agents work this way: a planner writes a Markdown plan, a generator turns it into .spec.ts files, and a healer patches tests that fail. Testsigma describes turning an accepted test into an automation script with one click. You end up owning code, which is great if your team reads code and less great if the people who wrote the manual cases don't.

Have an agent run the case. No code in between. The agent works from the plain-language steps and drives the app itself, which is the closest thing to a human tester working through the case. Katalon's runner and Autify's Aximo are presented this way, though neither public page goes into whether steps get cached between runs. Katalon says its runner checks the evidence for each step before it gives a verdict, and attaches video and screenshots to every result. That evidence matters a lot when the person reviewing results isn't the person who wrote the case.

Record, then replay. An agent runs the case once, the actions it took on a passing run get recorded, and later runs replay the recording with no model choosing the next step. Momentic caches resolved steps, Stagehand documents replaying cached actions "deterministically with self-healing turned off," and KaneAI's CLI replays unchanged steps from a recording. qtrl works this way too, which we'll come back to.

What running the case directly gets right

The case stays the source of truth. That's the big one. With generated scripts, the moment someone edits the code, the manual case and the automated test start to drift apart, and six months later nobody knows which one describes the product. When the agent runs the case itself, there's only one thing to keep up to date.

It also lets the people who know the product best keep writing tests. Manual testers usually understand the edge cases, the weird customer configurations and the flows that broke last quarter. Converting their cases to code has historically meant handing that knowledge to someone who writes Playwright, and some of it gets lost in the handoff.

Most teams are still early with this. TestMu AI's State of AI in Quality Engineering 2026 report (a vendor survey, so read it with that in mind) found only 10% of teams let an agent write, run and decide on tests. Running plain-language cases with an agent is new territory for most QA teams, which is a good reason to be deliberate about how you do it.

Where it costs you

An agent that works out each run from scratch pays for that every time, unless the tool caches or records what it did. The most detailed public numbers we know of come from Slack's engineering team, who measured agent-driven runs at $15 to $30 each and found only about 20% of runs took the exact same sequence of actions. We went through that experiment in agentic testing in CI. Those numbers come from Slack's own setup, and commercial runners are tuned differently, but the shape holds: a model in the loop on every run costs more and varies more than replaying known steps.

For exploratory work, variation is useful. An agent that finds a second route to the same goal is telling you something. For a regression suite gating deploys, it's noise, and it adds a layer on top of the ordinary flakiness your suite already has. Run 400 of a checkout test should do the same thing as run 399.

That's why we'd point most regression suites at the third option. The agent turns the plain-language case into a run without anyone writing code, and the recording turns that run into something repeatable. You keep the case as the source of truth and still get a repeatable run on every merge, with no model choosing the steps.

The conversion work that's left

The "no scripts" pitch is accurate. What it doesn't cover is the cases themselves. Manual test cases were written for people, and people fill in a lot of gaps without noticing. An agent fills those gaps too, but it fills them with guesses. So the effort you used to spend writing scripts goes into making the cases precise enough that a run means something.

A typical manual case looks like this:

Title: Change billing email
1. Log in as an admin
2. Go to billing settings
3. Change the email
4. Verify it works

A human tester knows which admin account to use, which email to type, and what "works" means. An agent has to guess all three, and "verify it works" will pass as long as nothing obviously breaks. Rewritten for an agent:

Title: Change billing email
Preconditions: Logged in as {{ADMIN_USERNAME}} on an account with an active plan
1. Open Settings, then Billing
2. Replace the billing email with {{NEW_BILLING_EMAIL}} and save
   Expected: a success message confirms the change
3. Reload the page
   Expected: the billing email field shows {{NEW_BILLING_EMAIL}}
4. Check the inbox for {{NEW_BILLING_EMAIL}}
   Expected: a confirmation email arrives

Three changes did most of the work. Test data moved into variables (the curly braces are just placeholders, every tool has its own syntax), so the case runs the same in every environment and no credentials sit in the text. Preconditions say what state the app is in before step one. And every step that matters has an expected result a reviewer can check against a screenshot. Vague expectations are a common reason AI-executed tests pass when they shouldn't. Our guide on writing test cases from requirements goes further into what a checkable expected result looks like.

Do this for your highest-value cases first. Rewriting a case is far cheaper than scripting it, but it's not free, and a lot of old manual cases turn out to describe flows that no longer exist. Rewriting doubles as the cleanup pass the suite needed anyway.

When you still want code

Plain-language cases aren't the right home for every test. Keep writing code for:

  • Unit and integration tests. They belong next to the code they test, and the people changing that code should own them.
  • API tests with precise assertions on response bodies, status codes and headers. An exact comparison is easier to read in code than in a sentence.
  • Tests with heavy setup, like seeding a specific dataset or faking a third-party webhook. Do that in code or fixtures and let the plain-language case start from the state it creates.
  • Anything you already have as a stable, readable script. If a Playwright test works and people understand it, there's no prize for rewriting it in English.

The manual backlog is mostly end-to-end user journeys, which is exactly where plain-language cases fit best. That's where to start.

Review doesn't go away

Whichever path you pick, someone still has to look at what the AI did, at least the first time. In the TestMu survey, respondents estimated that AI-generated tests were "almost right, but not quite" a median 65% of the time, and "almost right" is the hardest kind of wrong to spot. Check the first run of each case step by step against its expected results before you trust it in CI. We wrote a checklist for that in how to review AI-generated tests.


That's how qtrl is built. You write test cases in plain language, with variables and secrets per environment. On the first run, an AI agent works through the case in a real browser and records what it did. Once the run passes and replay is turned on for the project, later runs replay the recording, and the agent only steps back in when the app changes and a step breaks. Every run keeps screenshots and a step-by-step trace, and the cases stay organized in structured test management with review states, so your manual and automated coverage live in one place. See how it works.

Have more questions about AI testing and QA? Check out our FAQ

Take a break

Some bugs got loose. Catch them.

Bugs caught: 0