Insights10 min read

AI test generation: unit vs integration vs E2E in 2026

By qtrl Team · Engineering

The newest coding models write tests that run. In a benchmark published in October, 94.3% of the tests GPT-6 Astra wrote passed against the working program. Those same suites caught 50.1% of the bugs planted in it.

The gap between a test that runs and a test that catches something is the most useful number in AI testing right now. It also changes depending on where in the test pyramid you point the model. This post pulls together what the latest research says about AI test generation at the unit, integration and end-to-end level, and compares only models that are current today.

The current models, and which ones have been tested

Model releases move faster than research papers. As of November 2026, the flagships are Claude Opus 5.5 (released September 22), OpenAI's GPT-6 Astra, and Google's Gemini 4 Argon (released in October). Google's Gemini 4 Carbon is out too, but only to a restricted group so far. Here's which current models show up in any published benchmark for writing tests:

ModelIn a published test-writing benchmark?
Claude Opus 5.5Not yet
Claude Sonnet 5.5Yes, Req2Test-Bench
GPT-6 AstraYes, TestSuiteBench
GPT-6 Sol and GPT-6 LunaYes, Req2Test-Bench
Gemini 4 ArgonNot yet

So two of the three flagships have no published test-writing results at all. Where a paper tested a model that has since been replaced, we report what it found about the test level and leave the model scores out.

Unit, integration and E2E in one minute

If you want the long version, we covered unit testing vs integration testing on its own. The short version, with the part that matters for AI:

LevelWhat it checksWhat the author needs to knowWhere AI-written tests tend to slip
UnitOne function or class on its ownThe code and its contractAsserting what the code does instead of what it should do
IntegrationModules, APIs and the database working togetherSetup, fixtures, data contractsSkipping the project's own fixtures, checking only part of a response
End-to-endThe whole app, the way a user drives itThe running app, its state, timing, and what the product is supposed to doNoticing something is off without pinning down what

The pattern in that last column is the theme of this post. The higher you go, the more of the expected result lives outside the code. A unit test's correct answer is often readable from the function itself. An end-to-end test's correct answer lives in a spec, a product decision, or a person's head. Models are very good at reading code. Knowing intent is harder.

What the research says, level by level

Four recent benchmarks cover different parts of the pyramid. None of them covers all of it, which is worth knowing before you read any single headline number.

LevelMost recent benchmarkCurrent models testedHeadline
UnitTestEvo-Bench (Jul 2026)NoneAgent tests that passed caught as many mutants as the developers' own tests
Integration and system, no browserReq2Test-Bench (Oct 2026)Claude Sonnet 5.5, GPT-6 Sol, GPT-6 LunaAround 80% of tests pass, about a third catch everything the developer's test catches
Black-box system tests from a specTestSuiteBench (Oct 2026)GPT-6 AstraGPT-6 Astra's suites caught 50.1% of planted bugs
End-to-end in a browserCATTest (Aug 2026)NoneNo model found even half the known bugs

Unit tests: no current-generation numbers yet

This surprised us. Unit test generation is the most studied area in AI testing, but we couldn't find a published benchmark that runs the current generation of models on it. The most recent one, TestEvo-Bench from the University of Waterloo, ran 746 test-writing tasks drawn from real Java commit histories on models that have all been replaced since. So we're leaving its model scores out.

One finding is still worth carrying over, because it's about the level, not the model. When the agents' tests passed, they killed about as many mutants (deliberately planted bugs) as the tests the project's own developers wrote for the same change. At unit level, a passing AI-written test is roughly as sharp as a human one. The weak spot is everything around it: which tests get written at all, and which ones quietly fail to.

Integration and system tests: Req2Test-Bench

Req2Test-Bench, published on October 27 by researchers at the National University of Singapore and others, gives agents a behavior-driven scenario (Given, When, Then) and asks them to write the test behind it. The code being tested is hidden from the agent, the way it would be if you wrote the test before the feature. It has 5,086 instances across 8 projects and 5 languages, labelled as component, integration or system tests. Scenarios that needed a browser were excluded.

Two numbers matter. Pass@1 is the share of tests that pass on the real code. KillComplete@1 is the share that also catch every mutant the developer's own test catches.

Agent and model (high reasoning)Tests that passTests that catch everything the developer's test doesCost per test
Claude Code, Claude Sonnet 5.580.9%35.9%$0.68
Codex CLI, GPT-6 Sol73.5%31.3%$1.16
Codex CLI, GPT-6 Luna66.2%28.3%$0.04

Every current model loses more than half its tests between "passes" and "catches what a developer's test catches." The authors looked at why. Of the tests that fell short in one configuration, 77.9% actually ran the buggy line and passed anyway. They reached the right code and then checked too little: some of the response fields instead of all of them, or a shortcut around the project's fixtures and entry points.

The paper didn't include Opus 5.5 or any Gemini model, so treat this as a comparison of the models it did test, not a ranking of the field.

Longer scenarios scored worse on both measures. The paper reports results across all three test levels together, so it doesn't tell us whether integration tests fared better or worse than system tests.

Black-box tests from a spec: TestSuiteBench

TestSuiteBench, from ETH Zurich and LogicStar.ai, published October 9, is the closest thing we have to an end-to-end benchmark with current models. It covers 165 command-line programs written in Rust, Go and C/C++. Agents get only a written specification (median length around 5,000 words) and have to write a full test suite that checks the program's input and output behavior. For a command-line tool, that is end-to-end testing. Each model ran in its vendor's own agent.

GPT-6 Astra is the one current flagship in the paper. 94.3% of its tests were valid (they passed on the correct program) and its suites caught 50.1% of the planted bugs. It cost $2.18 per program and took about 12 minutes. The other models in the paper, including Claude Opus 5 and Gemini 3.8 Flash, are no longer the current flagships: Anthropic has since released Opus 5.5, and Google's top model is now Gemini 4 Argon. So we're not comparing them here.

Credit where it's due. A suite where 94 out of 100 tests are valid, written from a spec alone, is a real achievement. But the paper's own conclusion is blunt: even frontier coding agents struggle to write strong test suites. The suites the agents wrote were also consistently smaller than the reference suites written by people.

End-to-end in a browser: CATTest

This is the level with the least data. CATTest, accepted at EMNLP 2026, has agents write Playwright code to explore 102 AI-generated web apps and report bugs, then scores them against a list of bugs that people confirmed by hand. It was published at the end of August, before Opus 5.5, GPT-6 and Gemini 4 came out, so none of the models it tested are current. We aren't comparing them here. Three findings still hold regardless of model:

  • No model got to half. The best model found 42.57% of the known bugs, and the best precision on reported bugs was 48.56%. In the authors' words, no model can reliably uncover even half of the bugs while keeping its reports trustworthy.
  • Knowing something is wrong isn't the same as finding it. One model flagged 98.65% of buggy apps as failing, yet identified only 37.68% of the actual bugs. The model could tell the app was broken. It couldn't say how.
  • Reading the code isn't enough. When the strongest model reviewed the source without driving the browser, the share of its reported bugs that were real fell from 38.30% to 10.60%. Without seeing the app run, it reported a lot of bugs that weren't there.

The authors also found that this task rewards newer models far more than older ones within the same family. That's the strongest argument for re-running browser E2E with the current generation, which nobody has published yet.

Five patterns that hold at every level

1. Passing is cheap, catching is hard

94.3% valid against 50.1% of bugs caught. 80.9% passing against 35.9% catching everything the developer's test catches. Whatever the level, a green AI-written test tells you far less than it seems to. Pass rate is the number every tool shows you, and it's the least informative one.

2. The agent matters as much as the model

Req2Test-Bench also ran one open model, DeepSeek V4.1 Flash, inside different agents. With Aider and reasoning off, 9.7% of its tests passed. With Codex CLI and reasoning on, 82.8%. Same model, more than an eightfold difference. Claude Code with the same model landed at 80.5%. How the agent explores the repo, runs tests and retries is a big part of what you're buying. That's why model leaderboards alone don't tell you which coding agent to hand your testing to.

3. The input beats the model

The most striking number in TestSuiteBench isn't in the main table. When the authors replaced their carefully written specs with the projects' official developer documentation, the mutation score of the model they tried it on dropped from 25.5% to 4.0%. That model was an earlier OpenAI release, but the lesson is about the input, not the model. If you want stronger AI-written tests, a clear spec with acceptance criteria is the cheapest lever you have. We wrote about that in what is spec-driven testing.

4. Effort helps, money only up to a point

In TestSuiteBench, turning reasoning effort from low to the highest setting raised one earlier OpenAI model's mutation score from 31.3% to 46.5%. But Req2Test-Bench found that spending more per test gave little or no further gain in catching bugs. Its cheapest current setup, GPT-6 Luna at about four cents per test, wrote bug-catching tests 28.3% of the time, against 31.3% for GPT-6 Sol at $1.16. Turn reasoning up for test writing. Don't assume the most expensive option is the right one.

5. Ask twice

In Req2Test-Bench, taking the best of three attempts instead of one raised the share of bug-catching tests by 8.6 to 10.2 percentage points. For tests that guard something important, generating a few candidates and keeping the strongest is a cheap upgrade.

What nobody has measured yet

Put the four benchmarks side by side and the gaps are obvious:

  • No study runs the same agents on unit, integration and E2E tests for the same app, so nobody can say how much the gap widens as you move up the pyramid.
  • No published benchmark includes Claude Opus 5.5 or Gemini 4 Argon, and no study tests any current model on browser E2E.
  • Nobody has measured how flaky AI-written E2E tests are. The evidence so far is practitioner reports about brittle selectors and hard-coded waits.
  • Earlier research such as EvilGenie caught coding agents editing tests or hard-coding answers to get a green run. Nobody has checked whether that happens more at one level than another.

That's what we're testing next. We're running Claude Code, Codex, Cursor, GitHub Copilot and Gemini CLI on each vendor's current models against the same app, with bugs planted at every level, and measuring bugs caught, flakiness and test tampering instead of pass rate. We'll publish the method, the raw data and the results, whichever way they go.

What to do with this today

  • Measure bugs caught, not tests passed. Run a mutation testing tool (Stryker for JavaScript and TypeScript, PIT for Java, mutmut for Python) over your AI-written unit and integration tests every so often. It's the closest thing you have to the metric these benchmarks use. More in how to measure test coverage.
  • Give the agent a spec, not just the code. Acceptance criteria tell it what correct means. Without them, it can only describe what the code already does. See how to turn acceptance criteria into E2E tests.
  • Review the assertions, not the steps. The common miss is a test that reaches the right code and checks too little. Our checklist for reviewing AI-generated tests starts there.
  • Tell the agent to use your fixtures. Put your test setup conventions in the instructions file your agent reads (AGENTS.md, CLAUDE.md or similar). Shortcuts around project fixtures were one of the main reasons tests fell short in Req2Test-Bench.
  • For E2E, let the agent see the app run. Reviewing code alone produced mostly false bug reports in CATTest. And keep a person on the question of whether a failure is a real bug, which we covered in human-in-the-loop testing.

qtrl's take

The research lines up with what we built qtrl around. AI is getting genuinely good at writing tests, and at the unit level a passing AI test already holds up next to a human's. The hard part is knowing what correct means, and that gets harder the closer you get to the user.

That's why qtrl works at the end-to-end level, in real browsers, from test cases with expected results your team sets. Agents explore the app, draft tests and run them. They can't change the expected results, every run comes with screenshots, video and logs, and a person decides what a failure means. Unit and integration tests stay with your developers and their coding agents, ideally with mutation testing keeping them honest.

AI test generation: FAQ

Which model writes the best tests right now? Nobody can say yet. Claude Opus 5.5 and Gemini 4 Argon haven't appeared in a published test-writing benchmark. Among current models that have, Claude Code with Claude Sonnet 5.5 scored highest on catching bugs in Req2Test-Bench, ahead of GPT-6 Sol and GPT-6 Luna, and GPT-6 Astra's suites caught half the planted bugs in TestSuiteBench. The gap between "passes" and "catches bugs" is bigger than the gap between any of them, and the agent around the model moves results a lot. Pick on your own codebase, not on a leaderboard.

Does code coverage tell me whether AI-written tests are any good? Not on its own. Coverage tells you which lines ran, not what was checked. In Req2Test-Bench, most of the tests that fell short ran the buggy line and passed anyway. Mutation score is a much better signal.

What is mutation testing? A tool makes small deliberate changes to your code, like flipping a comparison or changing a number, and reruns your tests. If no test fails, that change survived, and you've found a gap. The mutation score is the share of changes your tests catch.

Can AI write end-to-end tests yet? It can write them, and they usually run. Finding the bugs that matter is where it falls short. The latest browser benchmark found no model reliably uncovering even half the known bugs, and that was with an agent driving the app, not just reading code.

Should I still write unit tests if AI can write them? Yes, and let the AI help. Unit level is where AI-written tests are closest to human quality. Just measure them with mutation testing instead of trusting a green run.


qtrl gives you AI agents that explore, draft and run end-to-end tests in real browsers, with expected results your team owns and evidence for every run. If you're new to the approach, start with what is agentic testing.

Have more questions about AI testing and QA? Check out our FAQ

Take a break

Some bugs got loose. Catch them.

Bugs caught: 0