How to manage test data without breaking your suite

A test suite that was green in March and is amber in September usually hasn't found a bug. Something about the data changed. An account got deleted during a cleanup, a subscription expired, a record somebody seeded eighteen months ago drifted out of the state the test assumed.

Test data is the least glamorous part of test infrastructure and one of the most common reasons suites stop being trusted. It also gets worse rather than better as a product matures, because the data model gets richer and the number of things that can be subtly wrong grows with it.

The four ways teams do it

Nearly every setup is one of these, or a mix that drifted together over time.

Shared fixture accounts. A handful of long-lived accounts everyone's tests use. Cheap to start, and it works until two tests run in parallel and one changes something the other asserted on. The failure mode is intermittent and order-dependent, which makes it expensive to diagnose.

Database seeding. A script that loads a known dataset before the run. Deterministic and fast, and it decays: the seed file drifts from the current schema, someone adds a column with a NOT NULL constraint, and now the seed represents a state your application can no longer produce.

Factories and per-test creation. Each test creates what it needs through the application's own API. Slower per test and much harder to break, because the data is created the way production creates it. This is the right default for most teams.

Production clones. A sanitized copy of real data. The best fidelity and the worst compliance profile. Sanitization that misses one column is a GDPR incident, not a test failure.

Make each test own its data

The single change that fixes the most test data pain: every test creates what it needs and depends on nothing another test created.

That means a fresh user per test, or per file at minimum. Yes, it costs setup time. It buys parallel execution without cross-talk, which is usually a bigger speed win than the setup cost, and it removes an entire category of flake where the failure depends on execution order. If you're working through a flaky suite, shared mutable data is worth checking before you go looking at timing.

Create through the API rather than the UI where you can. A test about checkout shouldn't fail because the registration form changed. Drive the thing you're testing through the interface, set everything else up through the fastest path available.

Give every record a run identifier

Tag test-created data with the run that made it. An email pattern works fine: test+{runId}-{caseId}@yourdomain.test, or a metadata field if your schema has room for one.

Three things get easier. Cleanup becomes a query instead of a guess. Debugging a failure means finding the exact records that run touched. And when someone spots odd data in staging, you can tell immediately whether a test made it or something real did.

Pair it with time-based cleanup. Delete test-tagged records older than a few days on a schedule, rather than relying on teardown. Teardown doesn't run when a test crashes, when CI is cancelled, or when the runner dies, and those are exactly the runs that leave the most mess.

Synthetic data, and where it falls short

Generated data is the right answer for volume and for anything touching personal information. Faker-style libraries handle names, addresses, and card numbers well enough for most purposes, and nothing you generate can leak a real customer.

What it misses is distribution. Real data is lumpy in ways generators aren't. Real users have names with apostrophes and characters outside Latin-1. Real accounts have been migrated through three schema versions. Real orders include the one from 2019 with a null field that was optional back then. Generated data is uniformly reasonable, and uniformly reasonable data never finds the bug.

Practical middle ground: generate the bulk, then maintain a small set of deliberately awkward fixtures. The user with a 200-character name. The account with no orders. The one with 10,000. The record with every optional field empty. Keep that set in version control and treat it as documentation of the edge cases you know about.

Don't clone production without doing the work

If you copy production data into a lower environment, the sanitization is the whole job and it's never finished. Names and emails are the obvious columns. The ones that catch people: free-text fields where users pasted personal details, uploaded file contents, audit logs, notification history, support ticket bodies, and anything in a JSON blob column where the schema is whatever the application felt like writing.

If you do it, automate the sanitization and test it. A check that runs after every refresh and fails loudly if it finds something matching a real-customer pattern. Manual sanitization is a control that works until the one time someone is in a hurry.

For most teams the honest answer is that a well-maintained factory setup plus a curated edge case set gets you most of the fidelity at a fraction of the risk.

Time is data too

The tests that break on a specific day of the year are almost always about dates. A trial that expires in 14 days. A report defaulting to last month. A subscription renewing on the 31st in a month with 30 days.

Never hardcode an absolute date in a fixture, because it becomes a past date and the test starts meaning something different. Always test relative to a controllable clock, so you can assert what happens at the boundary without waiting fourteen days to find out.

The reason this keeps getting deprioritized

Test data work never shows up as a feature and rarely shows up as a bug. It shows up as a suite that's slowly getting less reliable, which nobody files a ticket about until it's bad enough that people are re-running failures as a matter of routine.

By that point the fix is a project. Caught earlier, when the first shared fixture account starts causing order-dependent failures, it's an afternoon of moving to per-test creation. The gap between those two moments is where most of the cost lives.

qtrl runs agents against real browsers, which means the data question is the same one your existing suite has. What changes is visibility: each run records what it exercised, so when a case starts failing you can see whether the behavior changed or the state it depended on did.

Cases are managed records rather than scattered scripts, so the preconditions a test needs are written down in the same place as the test itself. That's the part that usually lives in one person's head until they go on holiday. See how it works.