How to build LLM eval suites: testing beyond pass/fail

Ship a feature that calls an LLM and testing looks different than it did last sprint, whether your team has noticed yet or not. Nobody writes expect(response).toBe(exactString) against a generated summary or a chatbot reply. Someone reads ten or twenty outputs, decides they look reasonable, and ships. That's not a test suite. It's a vibe check with a deadline.

Evals are the discipline built to replace that vibe check. Not agent testing, which checks whether an AI agent clicked the right button or completed a multi-step workflow in a browser, but the older, narrower problem of grading whether what a model actually said or generated is any good. The two get lumped together constantly, and the mix-up is a big part of why teams either skip this work entirely or bolt a single "does this look right?" review onto the end of a sprint and call it done.

Evals grade output, agent tests grade actions

If your product has an AI agent that books a flight, fills out a form, or clicks through your app, an agent test checks whether it filled in the right fields, hit submit, and landed on a confirmation page. That's behavior verification against a real interface, and it's the territory covered in our playbook for testing AI agents.

An eval asks a different question: not "did the agent do the right thing," but "was the thing it said or wrote actually correct, complete, and safe to show a user." A support bot that clicks every right button but gives a wrong answer about your refund policy passes an agent test and fails an eval. Both checks matter. Most teams building anything with an LLM in it need both, run separately, because they catch different failures.

What actually goes into a golden dataset

A golden dataset is a curated set of real inputs paired with what a correct output looks like, or at least the properties a correct output has to have. The mistake teams make is inventing cases in a whiteboard session. Better material already exists: production logs, support tickets where a customer flagged a bad answer, edge cases a teammate stumbled into while testing something else entirely.

Anthropic's engineering guidance on building evals makes a point worth borrowing directly: you don't need hundreds of cases to start. Twenty to fifty real examples pulled from actual failures gets you further than a large dataset of invented scenarios, because real failures tell you what actually goes wrong instead of what you assumed would go wrong. Their write-up is worth reading in full if you're starting from zero.

Build the dataset out from there. Include the boring cases (a straightforward question with one clearly correct answer), the ambiguous ones (a question that could reasonably be answered two different ways), and the adversarial ones (a user trying to get the model to say something it shouldn't). Skew the mix toward whatever your product actually gets asked, not whatever's easy to write a test for.

Three ways to grade a response

Once you have inputs, you need a grader: something that looks at what the model produced and decides whether it passed. Three approaches cover almost every real eval suite, and they trade speed and cost against nuance.

LLM-as-judge is the most flexible of the three, and the easiest to reach for by default, which is exactly why it needs the most scrutiny. Anthropic's guidance recommends calibrating the judge against real human ratings before trusting its scores at scale, and giving the judge an explicit way to say "unknown" instead of forcing a verdict when the input doesn't give it enough to go on. There's also a growing body of research on how gameable these judges are: a 2026 robustness study found that both pointwise and pairwise LLM judges stay vulnerable to adversarial manipulation of their inputs, and that how well they hold up depends heavily on prompt design and which model you pick as the judge. The paper is a good reality check before you wire an LLM judge into a merge-blocking CI gate and stop looking at it.

You're grading a distribution, not a single run

Run the same input through the same model twice and you can get two different outputs, even at low temperature. That breaks the pass/fail mental model most engineers bring from unit testing, where a test either passes or it doesn't, full stop.

The fix is to stop treating each eval case as a single trial. Run each case multiple times (three to five runs is a reasonable default) and grade the distribution: what fraction of runs passed. A case that passes five out of five is solid. A case that passes two out of five is telling you something, even if it's not an outright failure. Set your suite-level threshold on the aggregate pass rate across the dataset, not on any individual run, and you stop chasing ghosts every time one sample comes back flaky.

Catching a regression when you swap a model or edit a prompt

This is where an eval suite earns its keep day to day. You tweak a system prompt to fix one bad response, or you swap from one model version to a newer one because it's faster or cheaper, and there's no way to know what else moved without running the whole thing again.

Model swaps are the sneaky one. A newer model in the same family can be measurably better on public benchmarks and still regress on your specific task, because it formats answers differently, follows instructions with different strictness, or defaults to a different level of verbosity. None of that shows up if your QA process is a few people skimming outputs before a release. It shows up immediately if you have a baseline pass rate and rerun the same dataset against the new model or prompt before shipping it.

Treat the eval run the way you'd treat a performance benchmark: snapshot the pass rate before the change, run the full suite after, and look at the delta. A five-point drop on your golden dataset is a real signal. A single "this response looks weird" from someone testing it by hand isn't, not because it's wrong, but because you can't tell if it's representative until you've run enough cases to know.

The rough shape of an eval suite

Strip away the tooling differences between vendors and every eval suite has the same four parts.

In practice that's a dataset file, a grading function, a threshold, and a CI step that reads the score and decides whether to let the change through. Here's roughly what one eval case and its gate configuration look like, stripped down to the essentials:

{
  "case_id": "refund_policy_003",
  "input": "Can I get a refund if I bought the annual plan two months ago?",
  "reference": {
    "must_mention": ["30-day refund window", "prorated"],
    "must_not_mention": ["full refund after 60 days"]
  },
  "grader": "rubric",
  "runs_per_case": 5,
  "pass_condition": "4-of-5 runs satisfy all rubric items"
}

{
  "dataset": "refund_policy_evals.jsonl",
  "model": "current-production-model",
  "aggregate_threshold": 0.92,
  "on_below_threshold": "block_merge",
  "on_pass": "post_score_to_pr"
}

OpenAI's evals framework documents essentially the same shape under the hood: a JSONL file of inputs and reference answers, a grading template (exact match or model-graded), and a runner that produces a pass rate you can act on. Their build-eval guide shows what a production implementation of that same shape looks like.

The threshold is a judgment call specific to your product, not a universal constant. A customer-facing refund answer probably needs a threshold in the high nineties. An internal draft-generation tool that a human reviews before it goes anywhere can tolerate a lot more slack.

Where this sits next to the rest of your QA stack

Evals don't replace the rest of your testing. They cover one slice: is the model's output good. Everything else about the feature still needs the same testing it always did: the UI rendering the response, retry logic when the API times out, the loading state under a slow connection. The broader set of generative AI use cases worth knowing in QA covers several of the adjacent categories, and where generative AI fits in a testing strategy overall is a good next stop if evals are the first piece you're adding rather than the last.

The honest version: evals are the newest addition to a QA stack, not a replacement for the rest of it. Unit and integration tests still catch bugs in the code that calls the model and the plumbing around it. What's new is the one check nothing else was built to run: whether the model's actual output holds up.

Evals grade the model in isolation, one input at a time, against a dataset. Most LLM features don't ship that way. They live inside a product: a chat widget, a generated summary rendered on a page, a recommendation a user clicks through. qtrl tests that surface, running real browser flows against your actual UI and keeping an audit trail of every run, so the feature wrapping the model gets the same governed testing as the rest of your app.

Pair it with an eval suite that grades the model's output directly, and you've got coverage on both layers instead of just one. See how it works.