How to run performance tests in CI without flaky gates

Performance regressions almost never arrive as one bad commit. They arrive as forty commits over six months, each adding twenty milliseconds that nobody could reasonably have objected to, until somebody notices the dashboard takes four seconds to load and there's no single change to revert.

Catching that in CI is possible. The reason most teams don't is that the obvious approach, assert the page loads in under two seconds, produces a gate that fails randomly and gets switched off within a month. Here's how to build one that survives.

Why the obvious threshold fails

CI runners are shared, noisy, and variable. The same commit measured twice on the same runner can differ by a wide margin depending on what else is on the machine. Set a hard threshold and you get a gate that fails on a build where nothing changed, which teaches everyone to re-run it.

That's the same failure pattern as any other flaky check, and it has the same consequence: once red means "probably nothing," the check has stopped working even while it keeps running.

Measure relative, not absolute

The fix that makes everything else workable: compare against the baseline branch measured on the same runner in the same job, not against a fixed number.

Environment noise affects both measurements roughly equally, so the delta is far more stable than either absolute value. A 30% regression against the branch you're merging into means something regardless of how loaded the runner was. "Under 2000ms" means whatever the runner felt like that morning.

Practically, this means each performance job builds and measures both revisions. Twice the runtime, and worth it, because the alternative is a signal nobody believes.

Pick metrics that correspond to something

Total page load time is a poor gate. It bundles a dozen independent things together, so a regression tells you something got slower without telling you what, and improvements in one area mask regressions in another.

Better to gate on a small set of specific measures:

Bundle size is the best value on that list. It's completely deterministic, so there's no noise problem at all, it's trivial to measure, and it catches one of the most common real-world regressions. If you do one thing from this post, gate on bundle size.

Use percentiles, and use the right one

Averages hide exactly the behavior you care about. A p50 that looks fine with a p95 that doubled is a real regression affecting a real slice of users, and the mean will barely move.

Run each measurement several times and compare p95 across runs. Discard the first run, which includes cold caches and JIT warmup and is consistently unrepresentative. Five runs is usually enough to be stable; below three you're measuring noise.

Separate the fast gate from the deep check

Two tiers, with different jobs:

On every PR: bundle size, request count, and a lightweight browser measurement of two or three key pages. Under two minutes. Fails the build on a clear regression against the base branch.

Nightly on the main branch: full load tests with real concurrency, longer soak runs, and the scenarios that need a proper environment. Doesn't block anything, reports to a dashboard, and alerts on trend.

Trying to do the second tier on every PR is how teams end up with a 45-minute pipeline and a performance suite that gets disabled during the next crunch.

Watch the trend, not just the gate

A per-PR gate catches the 30% regression. It structurally cannot catch the 2% regression that happens forty times, because each individual change passes.

That's what the nightly trend line is for. Record the numbers over time and look at the shape monthly. Slow accumulation is the most common way products get slow, and the only way to see it is to look at a graph that spans longer than one merge.

When a gate fails, make it obvious why

A performance check that reports "LCP regressed 22%" and nothing else will get overridden, because the author has no path from that message to a cause and a deadline to hit.

Attach the evidence. A trace, the network waterfall, the bundle diff showing which module grew. Modern browser tooling exposes all of it, and an agent with DevTools access can pull a trace and summarize what changed. The difference between a gate people fix and a gate people bypass is almost entirely whether the failure message points at the cause.

Set the budget where users are

Relative gates catch regressions but they don't tell you whether the current state is acceptable. If your p95 was already four seconds, a gate holding it at four seconds is working exactly as designed and your users are still waiting four seconds.

So set absolute budgets too, informed by real user monitoring rather than a round number someone liked. Use them as targets on the nightly dashboard rather than as PR gates, where the noise problem comes back. Relative gates stop things getting worse. Absolute budgets are how they get better.

qtrl runs tests in real browsers, so the journeys you already cover for correctness are the same ones you can measure for speed, on the paths that matter to users rather than on a synthetic page load.

Each run is recorded with its evidence attached, which is what turns a red performance gate into a fixable ticket instead of an argument about whether the runner was busy. See how it works.