How-To10 min read

How to test WebMCP tools before AI agents use your site

By qtrl Team · Engineering

When an AI agent uses your site today, it reads the page and clicks things. It finds the search box from the accessibility tree, types into it, and hopes the results page looks the way it expects. It works, more or less, and it breaks in all the ways browser automation has always broken.

WebMCP offers another way. Your page registers tools, small JavaScript functions with a name, a description and a JSON schema, and the agent calls them directly instead of driving the UI. That's good for reliability. It also means your site now has an API whose caller is a language model, and that API needs testing like any other. The tooling for that is young but usable: Chrome DevTools, Playwright 1.64 and later, and the MCP servers agents already use.

What WebMCP is (and how settled it is)

Chrome's docs call WebMCP "a proposed web standard to help you build and expose structured tools for AI agents". The spec is a Draft Community Group Report from the W3C Web Machine Learning Community Group, edited by people at Google and Microsoft. It is not a W3C Standard and isn't on the Standards Track. Chrome offers an origin trial from Chrome 149, and you can turn it on locally with chrome://flags/#enable-webmcp-testing.

There are two ways to register a tool. The imperative API is a call to document.modelContext.registerTool(). The declarative API adds attributes like toolname and tooldescription to an HTML form. Chrome documents both, but the spec repo lists the declarative form as future exploration rather than part of the spec, so treat it as Chrome-specific for now.

A registration looks roughly like this (our example, based on the documented shape):

const controller = new AbortController()

await document.modelContext.registerTool({
  name: 'get_cart',
  description: 'Read the current shopping cart. Makes no changes.',
  inputSchema: { type: 'object', properties: {} },
  annotations: { readOnlyHint: true },
  execute: async () => ({
    content: [{ type: 'text', text: JSON.stringify(readCart()) }],
  }),
}, { signal: controller.signal })

// Later, to unregister:
controller.abort()

The API is still moving. Chrome's own notes list changes across recent versions: Chrome 155 deprecated JSON-stringified input arguments, and outputSchema arrived in Chrome 159. Even the docs disagree on return values: Chrome's imperative API examples return a plain string from execute, while Playwright's release notes read the result as content[0].text. That instability is the first reason to test: the browser underneath your tools will change more often than the tools do.

What to test

A WebMCP tool sits in an odd spot. It runs in the page, in the user's session, like UI code. It has a schema and a return value, like an API. And its caller picks it based on a sentence of English, like a prompt. Testing it well means covering all three.

Five things to test once your site registers WebMCP toolsLayerThe questionWhere to check itRegistrationRight tools, right pages, right user?page.webmcp.tools()ContractDo inputs and outputs match your schemas?DevTools, then AjvBehaviorSame end state as the UI path?page.webmcp.callTool()SafetyHonest annotations, server still checking?Request log + API testsAgent fitDoes a model pick the right tool and arguments?Playwright MCPFirst four: deterministic, run on every commit. Last one: model in the loop, run as evals.

The top four layers are deterministic and belong in your normal suite. The last one has a model in the loop, so it behaves more like an LLM eval than a test.

Registration

Tools are registered by page code, so they come and go with page state. Check that each page registers the tools you expect and nothing else. Then check the transitions: a tool that only makes sense for a logged-in user shouldn't be there for a logged-out one, and a tool tied to a component should disappear when that component unmounts and its signal is aborted. Stale tools are the WebMCP version of a button that's still on screen after the feature behind it was removed.

Contract

Your inputSchema is a promise to the agent about what it can send. Test the edges: missing optional fields, wrong types, empty strings, values just outside an enum. If you declare an outputSchema, check that real results match it. Chrome DevTools flags schema violations for both parameters and return values, which makes this layer easy to check by hand. To automate it, validate callTool() results in Playwright against the schema object you import from your own source, with a validator like Ajv. Playwright's tools() returns the input schema but not the output schema, so the copy in your code is the one to test against.

Behavior

The most useful WebMCP test is a parity test: does calling the tool leave the app in the same state as doing the same thing through the UI? If add_to_cart via the tool skips a validation the form enforces, an agent can now do something a human user can't. Run the tool, then assert on the page the way a user would see it.

Safety

Annotations are claims you make about your own tools. readOnlyHint says a tool changes nothing. consequentialHint lets the browser or agent ask the user before running something like a purchase. untrustedContentHint marks output that contains content you don't control, like reviews or messages. Agents and browsers use these to decide how careful to be, so a wrong annotation is a real bug. The good news is that "read only" is testable: call the tool and assert that nothing tried to write.

Two more things belong here. First, a tool runs with the user's session, so the server has to keep enforcing authorization exactly as it does for UI requests. Your API tests should already cover that, and WebMCP is a good reason to check they do. Second, the spec names tool descriptions and tool outputs as prompt injection vectors. Any tool that returns user-generated text deserves the same tests we describe in how to test for prompt injection.

Agent fit

A tool can pass every test above and still go unused, because the model reading its description doesn't understand when to call it. Chrome's DevTools docs list "agent rejection" as a troubleshooting case: check the tool's invocation counter to see whether the agent ever picked it for a given prompt. Descriptions are the part of WebMCP that behaves least like code, and the only way to test them is to put a model in front of them.

Step 1: inspect your tools in DevTools

Start by hand. Enable #enable-webmcp-testing and #devtools-webmcp-support in chrome://flags, relaunch, open DevTools, and select WebMCP in the Application panel sidebar.

The pane has two sections. Available Tools is a live list of what's registered, with a counter per tool. Invoked Tools is a log of every call with its status, input and output, filterable by name, status, and whether the tool is declarative or imperative. You can select a tool, edit its parameters and run it, which is the fastest way to see whether a schema and an execute function agree with each other.

Step 2: automate the deterministic layers with Playwright

Playwright 1.64 added page.webmcp and frame.webmcp. In Chromium you enable it with a launch argument. Playwright's Firefox supports it behind the dom.modelcontext.enabled and dom.modelcontext.testing.enabled preferences, and WebKit doesn't implement it. Keep it simple and put the flag on your Chromium project:

// playwright.config.ts
projects: [
  {
    name: 'chromium',
    use: {
      ...devices['Desktop Chrome'],
      launchOptions: { args: ['--enable-features=WebMCP'] },
    },
  },
  // firefox, webkit...
],

Then skip the WebMCP specs everywhere else:

import { test, expect } from '@playwright/test'

test.skip(({ browserName }) => browserName !== 'chromium', 'WebMCP runs in Chromium only here')

test('cart page registers the expected tools', async ({ page }) => {
  await page.goto('/cart')
  const tools = await page.webmcp.tools()

  expect(tools.map((t) => t.name).sort()).toEqual(['apply_coupon', 'get_cart'])
  expect(tools.find((t) => t.name === 'get_cart')?.annotations?.readOnly).toBe(true)
})

test('get_cart matches what the user sees', async ({ page }) => {
  await page.goto('/cart')
  const result = await page.webmcp.callTool('get_cart', {})
  const cart = JSON.parse(result.content[0].text)

  await expect(page.getByTestId('cart-count')).toHaveText(String(cart.items.length))
})

The second test is the parity check from above. The tool and the page should agree, and when they don't, one of them is wrong. It assumes the content array shape from our registration example. If your tool returns a plain string, assert on the result directly.

For the read-only claim, record what the page sends to your API while the tool runs:

test('get_cart really is read only', async ({ page }) => {
  await page.goto('/cart')
  const apiOrigin = 'https://api.example.com'
  const writes: string[] = []
  page.on('request', (req) => {
    if (new URL(req.url()).origin === apiOrigin && req.method() !== 'GET') {
      writes.push(`${req.method()} ${req.url()}`)
    }
  })

  await page.webmcp.callTool('get_cart', {})
  await page.waitForLoadState('networkidle')
  expect(writes).toEqual([])
})

Filtering to your API origin keeps analytics beacons from failing the test, and waiting for the network to settle catches requests the tool fires without awaiting. This only catches server writes, though. A tool that quietly changes local storage or in-memory state needs a second check, like reading the cart through the UI before and after.

A few details from the Playwright docs are worth knowing. tools() throws if the browser launched without WebMCP support, so a missing flag fails loudly instead of returning an empty list. callTool() throws if the tool isn't registered or its execute throws, but a result with isError: true comes back as a normal result, so assert on it explicitly. And the docs say to treat tool names, descriptions, schemas and results as untrusted input, since the page provides them.

One inconsistency to watch: Playwright's WebMCP class page refers to navigator.modelContext, while Chrome's docs and the spec use document.modelContext. Go with the Chrome docs and the spec.

Step 3: put an agent in front of it

For the agent fit layer you need a model choosing tools. There are a few ways to set that up.

  • Playwright MCP picks up WebMCP tools by default and offers them to the agent as webmcp_<tool> (pass --no-webmcp to turn that off). If your team already runs agents through Playwright MCP, you get this for free.
  • Chrome DevTools MCP has two experimental tools, list_webmcp_tools and execute_webmcp_tool, off by default behind --categoryExperimentalWebmcp. We compared the two servers in Chrome DevTools MCP vs Playwright MCP.
  • Google's example extensions. Chrome's docs point to an example agentic extension and a Model Context Tool Inspector extension for imitating an agent against your page. Note that the example extension trusts annotations, so it won't catch a dishonest one.

Write the scenarios as tasks, not tool calls: "add the cheapest blue t-shirt in size M to the cart", then check whether the agent called the right tools with the right arguments and ended in the right state. Run each one several times. A description that works four times out of five isn't done, and you won't know it's four out of five from a single run. We wrote more about why agent runs vary and how to make them repeatable.

Step 4: wire it into CI

The Playwright tests from step 2 are ordinary tests and run wherever your suite runs, as long as the Chromium project has the launch flag. Keep the agent scenarios from step 3 in a separate job. They're slower, they cost model calls, and their results are a pass rate rather than a pass or fail, so they fit better as a nightly run or a gate on changes to tool definitions.

Pin your browser version, and read the WebMCP notes when you bump it. With an API changing this often, a browser upgrade is the most likely thing to break your tools, and you want that to show up as a failed test rather than an agent quietly giving up on your site.

Is it worth testing yet?

If you've shipped WebMCP tools, yes, even in an origin trial. Chrome's docs say WebMCP is designed mainly for local browser workflows with a person in the loop, which means the agents calling your tools are acting on behalf of real users in real sessions. The registration and behavior tests are cheap, and they'll catch the bugs that hurt: tools that do something the UI wouldn't, and tools that say they're safe when they aren't.

If you haven't shipped any yet, write the parity test first. It forces the question of which UI flow each tool stands in for, and that's the design question that matters most anyway.


qtrl's AI agents test your app the way users reach it today: through the page, in real Chromium browsers. As agents start calling tools instead of clicking buttons, both paths need coverage, and both need to end in the same place. qtrl covers the UI path and keeps the user journeys in structured test management, so you can see which flows are covered and what each run actually checked. See how it works.

Have more questions about AI testing and QA? Check out our FAQ

Take a break

Some bugs got loose. Catch them.

Bugs caught: 0