Human-in-the-loop testing: where people fit in AI QA

An AI agent can draft a batch of test cases in the time it takes to read one. So the drafts pile up, and whoever owns the review queue starts clicking approve a lot faster than they can read. Nobody decided to stop reviewing. It just stopped being possible to keep up.

That's the real problem human-in-the-loop testing has to solve. Everyone agrees a person should stay involved when agents write and run tests. Far fewer teams have decided which decisions that person owns, and which ones they should stop pretending to check.

What human-in-the-loop testing actually means

Human-in-the-loop (HITL) testing is a QA setup where AI agents do the high-volume work (exploring the app, drafting test cases, running them, collecting evidence) and people make the calls that decide whether a result counts. A test isn't part of the suite until a person approves it, and a failure isn't a bug until someone confirms it. Shipping is always a person's call.

Nobody watches every agent run in real time. The human sits at a handful of specific checkpoints, and everything between them runs without waiting.

The interesting part is choosing those checkpoints. Put them in the wrong place and you get one of two failures.

Two ways the loop breaks

Everything goes through a human. Every generated test, every run, every failure waits for someone to look at it. The queue grows faster than anyone can clear it, and review quietly turns into approval. There's a name for this: automation bias, the tendency to trust an automated system's output over your own judgment. A gate that approves everything is worse than no gate, because the approval still shows up in the audit log as if someone checked.

Nothing goes through a human. The agent writes the tests, runs them, reviews its own results and reports green. That's a closed loop. When the same system defines what "correct" means and then checks against it, mistakes get confirmed instead of caught. We covered the code version of this in how to review AI-generated tests: tests derived from the code they check will happily pass on buggy code.

Both failures show up as the same symptom. The suite says one thing, production says another, and nobody can explain the gap. Closing that gap is a big part of what we called the verification tax.

Where the human belongs

Our rule of thumb: if a decision defines what "correct" means, or can't be undone, a person makes it. If it's about how to get there, the agent can have it.

The expected result is the line we care most about. An agent that can rewrite steps to cope with a redesigned page is useful. An agent that can rewrite the expected result to match what it found has stopped testing and started describing. Keep that line fixed and most of the risk in letting an agent adapt goes away.

Note what's missing from the human column: watching runs, re-running flaky tests, fixing selectors. If your people spend their review time there, the loop is in the wrong place.

How to keep review from becoming the bottleneck

A checkpoint is only useful if people think at it. These habits keep review honest as the volume of agent output goes up.

1. Review evidence, not just text. A test case written in plain language reads fine almost every time. The screenshot of step four showing the wrong account page doesn't. Reviewers should see what the agent saw.
2. Review the diff. A short change gets read; a long test that's mostly unchanged gets skimmed.
3. Match review depth to risk. A test on the settings page and a test on checkout don't deserve the same scrutiny. Use the same tiers you use for risk-based test prioritization: deep review where failure is expensive, a quick look everywhere else.
4. Unreviewed means not in the suite. Drafts that nobody has approved stay drafts. They don't gate a release and they don't count toward coverage. Without that rule, a stalled review queue turns into untrusted tests running in CI by default.
5. Watch your rejection rate. If reviewers almost never reject or edit anything, either the agent is very good or nobody is reading. Spot-check a few approved tests now and then to find out which.

Loosening the loop over time

Human-in-the-loop isn't a fixed setting. A suite that has run cleanly for months on stable flows doesn't need a person confirming every green result. A brand-new agent exploring a part of the app nobody has tested before needs a lot more attention.

The pattern that works is to start agents with narrow permissions and widen them on evidence. Suggestions first, then drafts that need approval, then approved suites running unattended. If something starts behaving oddly, you move it back a step. We went into the levels in detail in AI agent governance for testing teams. The short version: an agent earns autonomy by being right, and the humans are the ones who decide when it has. The definition of correct and the decision to ship never get delegated, however good the agent gets.

qtrl's take: humans own intent, agents do the legwork

We built qtrl around one belief: AI testing is only useful if you can trust the results, and the results you can trust are the ones you can trace back to a person's decision. The review loop is how tests move through the product, from the first draft onward.

When an agent generates test cases, they land in your test library as drafts. The agent then runs each draft once against the real app. On that first run it can adjust steps that don't match the UI, but it can't touch the expected results, and every step it inserts, deletes or edits is recorded with a reason. The test's version history shows what changed, so whoever reviews it checks the diff instead of re-reading the whole test.

Next, an AI reviewer does a first pass: it checks the test for vague steps and gaps, and sets it to Approved or Rejected with its reasoning on record. That first pass is a starting point. Anyone on your team can change a test's status, and the audit trail shows whether a person or an agent made each call. For critical flows like checkout or billing, we'd have a person confirm every new test before it counts.

Once a test is approved, it runs as written. If a step no longer works, the agent reports it as a failure instead of quietly rewriting the test, so drift reaches a person. Every run comes with screenshots, video and step-by-step logs: a failure shows you where and why, and a pass shows you what was checked.

Some flows need a person to run them, not just review them: a visual check, a payment with a real card, a screen reader pass. In a qtrl test run, each test can be assigned to the agent or to a teammate, and both show up in the same run and report.

How much the agents do on their own is up to you. You can start with a structured suite your team runs by hand, point agents at the tests you trust, and have agents draft tests for the areas you haven't covered yet. Underneath all of it, encrypted secrets never reach the agent, and the audit trail records who changed which test, when, and what was approved.

The loop shouldn't mean a person babysitting an agent. The agent should do everything mechanical well enough that review takes minutes, not an afternoon. The human should spend those minutes on the one question an agent can't answer for you: is this what the product is supposed to do?

Human-in-the-loop testing: FAQ

Isn't human-in-the-loop just manual testing? No. In manual testing, a person does the clicking. In human-in-the-loop testing, an agent does the clicking and the person decides what the results mean. The human effort moves from execution to judgment.

Can another AI do the review instead? An AI reviewer is useful for a first pass. It catches vague steps, missing preconditions and tests that don't match their title. It shouldn't be the only check on anything that matters, for the same reason the agent shouldn't grade its own work: if both share the same blind spots, the review confirms them. In qtrl the AI reviewer's verdict is recorded with its reasoning, and anyone on the team can overturn it.

Who should do the reviewing? Whoever knows what the feature is supposed to do. On most teams that's QA, but a product manager is often the better reviewer for expected results on a new flow, and a developer for a test that touches an edge case they wrote.

Does this count as human oversight under the EU AI Act? Not by itself. The Act's human oversight requirement is about the high-risk AI system you ship, not about how your QA tools work. But if you are shipping one, an approval record for every test and an audit trail of who signed off are useful evidence that your testing process has real human checkpoints. More on that in what the EU AI Act means for QA.

When can a test run with no human involved at all? Once it's been approved and has a track record of stable runs on a flow that isn't changing much. Even then, a person still decides what a failure means and whether to ship.

qtrl gives you AI agents that explore, draft and run tests in real browsers, and a review workflow that keeps your team in charge of what counts. If you're new to the approach, start with what agentic testing is or see how the agentic testing tools compare. Try qtrl free and see the review loop for yourself.