How to prune a regression suite without losing coverage

Regression suites only ever grow. Every bug adds a test, every feature adds a few more, and nothing ever gets deleted because deleting a test feels like removing a safety net. Five years in you have 4,000 tests, a 90-minute pipeline, and no idea which of them have caught a real bug since the last reorg.

Pruning is the fix and almost nobody does it, because the downside is vivid (you delete a test, a bug ships) and the upside is diffuse (the pipeline is a bit faster). Here's how to do it with evidence instead of nerve.

Get the data first

You need four numbers per test, and most CI systems can give you all four from run history. If yours can't, instrument it and wait a month. A month of waiting beats a year of arguing.

• Total runtime across all runs. The tail is usually shocking
• How often the test went red
• How often a red led to an actual code change
• When it last caught something real

The third one is the number that matters and the one nobody tracks. It also takes real effort to collect, because it means correlating failures with fixes. A rough version is enough: tag failures that preceded a revert or a bugfix commit touching the same area. Precision isn't the point, the distribution is.

The four categories

Once you have the data, most tests sort themselves.

Never failed, ever. A test that has run 900 times and passed 900 times is either covering something stable or asserting nothing. Check which. A surprising number of long-lived green tests turn out to assert that a page loaded, which any other test on that page would also catch.

Fails often, never finds anything. Pure flake. These are worse than useless: they consume runtime, they consume attention, and they train the team to treat red as noise. Fix them or delete them. Do not leave them with a retry wrapper and a TODO, which is the industry standard approach and the reason flaky suites persist for years.

Fails rarely, always meaningfully. Your best tests. Protect them, keep them fast, and if one is slow, invest in making it faster rather than considering it for the chop.

Redundant. Twelve tests covering the same login flow with trivial variations. Each one individually defensible, collectively a waste. This is usually the largest category in a mature suite and the biggest source of savings.

Find redundancy without reading 4,000 tests

Coverage overlap is the mechanical approach: run tests individually with coverage instrumentation, then look for tests whose covered lines are a subset of another test's. Slow to compute, but you run it once.

Cheaper heuristics that work nearly as well:

• Group by the page or component under test, then read titles. Redundancy is usually visible from the names alone
• Look for tests that always fail together. Tests that fail in lockstep across a year of history are testing the same thing
• Search for repeated setup blocks. Identical setup usually means overlapping assertions

When you find a cluster, don't delete the extras outright. Merge them. One test walking the flow with several assertions along the way is usually better than six tests each walking the same flow to assert one thing, and it's faster by roughly the cost of five setups.

Mutation testing for the ones you're unsure about

For the tests you can't classify from history, mutation testing gives you a direct answer. Introduce deliberate bugs into the code a test covers and see whether the test fails. If it doesn't, it isn't protecting that code, whatever the coverage report says.

It's too slow to run across a large codebase routinely. Run it once, during the prune, on the modules where you have the most tests and the least confidence. The results are usually uncomfortable and always clarifying.

Delete, don't disable

Skipped tests are debt that looks like an asset. They sit in the repo with a skip annotation, they show up in the count, and they protect nothing. A year later nobody remembers whether the skip was temporary, and re-enabling it produces a failure nobody can interpret.

Git remembers. If you need the test back, it's in the history. Write the deletion commit message properly, explaining what the test covered and why it went, and you've preserved everything that mattered.

One reasonable exception: move borderline cases to a nightly suite before removing them entirely. You keep the signal, you lose the pipeline cost, and after a quarter of nightly runs you'll know whether the test ever mattered.

What to do with the time you get back

A pruned suite has a gap in it, because the tests you removed were covering something even if they were covering it badly. Take the runtime you saved and spend some of it on the journeys that never had coverage at all.

Almost every mature suite has the same shape: extremely dense coverage of whatever was being built three years ago, when someone was enthusiastic about testing, and nothing at all on the features shipped in the last eighteen months. Pruning the first half funds the second. Risk-based prioritization tells you which gaps to fill first.

Make it a habit, not a project

A one-time prune buys you eighteen months and then you're back where you started. The version that lasts is a standing rule: a quarterly pass where the slowest and flakiest 5% get reviewed, and a policy that new tests state what failure they're meant to catch.

That second one does most of the long-term work. A test whose author can't articulate the bug it would catch is a test that probably doesn't catch one, and asking the question at review time is much cheaper than discovering the answer three years later with a coverage instrumentation run.

qtrl keeps coverage as structured cases mapped to the journeys they verify, which is what makes redundancy visible. Six cases covering the same path look like six cases covering the same path, rather than six files in a directory.

Run history sits alongside the cases, so the pruning decision has the failure and outcome data attached to it instead of requiring a separate archaeology project against your CI logs. See how it works.