Security testing for AI features beyond prompt injection

Security testing for an AI feature usually starts and ends with prompt injection, because that's the risk everyone has heard of. It's the right place to start. It's a bad place to stop, because several of the other failure modes are easier to exploit and considerably easier to test for.

OWASP maintains a Top 10 for LLM applications that maps the territory. What follows is the testing angle: which of those categories you can write a concrete test for, and what the test looks like.

Insecure output handling

The one that gets underrated. Model output is untrusted input to whatever consumes it, and teams treat it as trusted because it came from their own system.

If model output is rendered as HTML, it's an XSS vector. If it goes into a SQL query, a shell command, a template, or a file path, it's an injection vector of the ordinary, well-understood kind. The novelty is that the payload arrives from your own API rather than a form field, which is exactly why it skips the validation everyone built for form fields.

Test it the way you'd test any injection sink. Craft an input that makes the model emit a script tag, a SQL fragment, a path traversal sequence. Assert the consuming layer neutralizes it. If your product renders markdown from a model, assert that raw HTML in the output doesn't execute.

Excessive agency

An agent with more permissions than its task needs. This is the category that turns every other vulnerability into a bigger one, and it's mostly a design issue you can nonetheless write tests against.

Three tests worth having:

• Permission boundary. Ask the agent, in the normal way a user would, to do something outside its remit. Delete a record it should only read. Access another tenant's data. Assert the call is refused at the authorization layer, not by the model declining.
• Tool scope. Enumerate the tools available in each agent configuration and assert the list against an expected set. Tool lists grow quietly, and a tool added for one workflow stays available in all of them.
• Human-in-the-loop. For any irreversible action, assert that confirmation is required and that the confirmation can't be produced by the model itself.

That first test matters because a model refusing is not a security control. It's a preference, and preferences can be argued with. The control has to be in the code path.

Sensitive information disclosure

It goes both ways, and both are testable.

Outbound: does the model ever emit data the current user shouldn't see? In a retrieval-augmented setup, this is the tenant isolation question. Build a fixture with two tenants and distinctive data in each, then query as tenant A with prompts designed to surface tenant B's content. Assert on what comes back. Do it for every retrieval path, because isolation tends to be enforced in one place and forgotten in the second one somebody added.

Inbound: what leaves your system when a request goes to a model provider? If users paste customer data into a support summarizer, that data is now in a request body going somewhere. Test that your redaction layer catches the patterns it claims to catch, and check the logs while you're there. Prompt logging is where sensitive data accumulates without anyone deciding it should.

System prompt leakage

Assume your system prompt will be extracted, because it will be. The test isn't whether it can be extracted, it's whether extraction matters.

So the check is a review rather than an assertion: does the system prompt contain anything that's damaging in public? API keys and internal endpoints are the obvious fails. Less obvious: business rules that reveal pricing logic, the names of internal systems, or a description of your moderation thresholds precise enough to route around.

Add a test that attempts extraction and asserts no high-sensitivity string appears in output. It won't catch every paraphrase. It will catch the day someone puts a credential in there.

Unbounded consumption

Inference costs money per request, which makes an unrated endpoint a financial denial-of-service target rather than just an availability one. An attacker doesn't need to take you down, they just need to run up the bill.

Test that rate limits exist per user and per organization, that input length is capped before it reaches the model, that output tokens are bounded, and that an agent loop has a maximum step count. That last one also catches an honest bug: an agent that gets stuck retrying the same action will happily do so until something stops it, and the natural stopping point is your quota.

Supply chain and data poisoning

Harder to test directly, which doesn't make them less real. The practical controls are pinning model versions rather than tracking a floating alias, and knowing the provenance of anything in your retrieval corpus.

The testable part is what happens when the model changes. A model version bump can shift behavior across every category above, including ones that passed last month. Wire your security suite to run on model version changes and system prompt changes, not just on application code, for the reason laid out in testing for prompt injection: the defense lives partly outside your repository.

How to run these

Same mechanics as any eval. Fixed dataset, programmatic grader, threshold, CI gate, multiple runs per case because the system under test is non-deterministic. The scaffolding in building an LLM eval suite carries over directly.

One difference in how you set thresholds. A quality eval can reasonably pass at 95%. A tenant isolation test cannot. For anything where a single failure is a breach, the threshold is every run, and a single failure in a hundred is a finding rather than noise.

What to do first

If you're starting from nothing, do them in this order: tenant isolation, output handling, permission boundaries, rate limits, then injection. That's roughly descending order of how bad the worst case is and ascending order of how hard the test is to write, which is an unusually convenient alignment.

Injection last isn't a demotion. It's the hardest to test well and the one where a passing suite gives you the least certainty, so it's worth having the cheaper controls solid before you spend a sprint there.

qtrl runs agents against real browsers within defined boundaries, so permission and scope tests exercise the same path a user would rather than a mocked approximation of it. Runs are logged step by step, which is what makes a boundary violation visible instead of inferred.

Keeping these as managed test cases alongside functional coverage is the part that makes them last. They need to re-run on every model change, and a security check nobody can find is a security check that quietly stops running. See how it works.