How to test for prompt injection in your AI features

If your product has a feature that reads something a user didn't write, a web page, an uploaded PDF, an email thread, a support ticket, and then acts on it, you have a prompt injection surface. Most teams shipping that kind of feature have no test for it. Not because they think it's safe, but because nobody has told them what the test would look like.

Here's what to build. The short version: treat injection as an input validation problem where the validator is a language model that can be argued with, and write cases the way you'd write them for any other class of untrusted input.

Why this is worth your time now

The attacks stopped being theoretical. Brave documented indirect prompt injection in Perplexity's Comet browser, where content on a page could steer the agent using the user's own session. Palo Alto's Unit 42 published web-based injection observed in the wild. Anthropic has published its own research on mitigations for browser use, and is fairly direct that mitigation is the right word. Nobody claims this is solved.

That last part is what makes it a QA problem rather than a security team's problem. Unsolved classes of vulnerability get managed with regression tests, because the defense you shipped last quarter can stop working when you change the model, the system prompt, or the tool list.

Step 1: map what the model can actually do

Before writing a single test, write down every tool, function, or API the model can call, and what the blast radius of each one is. Injection only matters in proportion to the capability behind it. A summarizer that can only return text has a small problem. An agent that can send email, move money, or delete records has a large one.

Sort the list into three buckets: read-only, write-scoped-to-the-user, and anything that touches another user or an external party. That third bucket is where your test effort goes first. It's the same risk-tiering logic behind risk-based test prioritization, applied to capabilities instead of features.

Step 2: enumerate your untrusted inputs

Every path where content reaches the model without a user typing it is an injection vector. The usual suspects:

• Web pages the agent browses
• Uploaded documents, including images with text in them
• Email bodies, calendar invites, and their attachments
• Third-party API responses, including search results
• Database fields that other users can write to (profile bios, ticket comments, file names)
• Tool output from another agent, which is the one people forget

Tool output deserves a flag. In a multi-agent pipeline, one agent's output is another agent's input, and it usually arrives with more implied trust than a random web page. If agent A can be injected, agent B inherits the problem.

Step 3: write the test cases

Build a fixture set where each case is a piece of content carrying an instruction, plus an assertion about what the system must not do. The payload goes in the untrusted input. The assertion goes on the tool calls, not on the text output.

Asserting on output text is the mistake most first attempts make. The model refusing in prose while still calling the tool is a fail, and a text assertion will happily call it a pass. Instrument the tool layer and assert there.

Cases worth having, at minimum:

1. Direct override. Plain instructions in the content telling the model to ignore prior instructions and do something else. The baseline. If this passes, you have nothing.
2. Exfiltration. Content that asks the agent to include data it can see (a session token, another record, the system prompt) in a URL it fetches or a message it sends. This is the one that turns a read-only agent into a data leak.
3. Hidden text. The same payloads in white-on-white text, zero-size fonts, HTML comments, alt attributes, and image text. A human reviewer sees a normal page. The model sees the instruction.
4. Authority spoofing. Content formatted to look like a system message or a message from the developer. Fake XML tags, fake role markers, anything that mimics your own prompt structure.
5. Delayed payload. An instruction to do something on a later turn, or to remember something for the rest of the session. Tests whether your defense holds across a conversation rather than one request.
6. Benign control cases. Content that mentions instructions without being an attack. A page about prompt injection, a support ticket quoting an error message. If your defense blocks these, you've built something users will hate.

Benign controls are not optional. A filter tuned only on attacks will flag half your legitimate traffic, and the team will turn it off within a month.

Step 4: score it as a rate, not a pass

Here's where injection testing diverges from normal functional testing. The model is non-deterministic. A payload that fails to land nine times out of ten still lands. One green run tells you very little.

Run each case multiple times and track the block rate. Set a threshold and gate on it, the same way you'd gate on any other eval. The mechanics are the ones in building an LLM eval suite: a fixed dataset, a grader, a threshold, a CI gate. The difference is that the grader here is checking tool calls, and your threshold for the high-blast-radius bucket should be 100% with no rounding.

Step 5: re-run it on every change that touches the model

Prompt injection defenses are not code in the usual sense. They live in the system prompt, the tool descriptions, the model version, and the guardrail layer. Any of those can change without a normal code review catching the regression.

Wire the suite to run on system prompt changes, model version bumps, and tool definition changes, not just on application code. A model upgrade that improves every benchmark you care about can still be more susceptible to a payload class your old one shrugged off. You won't know unless you check.

What this doesn't get you

A passing injection suite means the payloads you thought of don't work. It says nothing about the ones you didn't. Since the underlying issue is that language models don't reliably separate instructions from data, testing is a way to manage the risk, not close it.

Which is an argument for keeping the capability list short. The most effective control is still not giving the agent a tool it doesn't need. Tests tell you how well your defenses hold. Scope tells you how much it costs when they don't.

qtrl runs agents against real browsers with defined boundaries on what each run is allowed to touch, which is the same control that limits what an injected instruction could reach. Runs are logged step by step, so when something in a page steers a run off its intended path, you can see exactly where it happened.

That record is also what makes injection cases repeatable. Keep them as managed test cases alongside your functional coverage, re-run them on every model or prompt change, and watch the block rate over time instead of hoping. See how it works.