Skip to content

Evals for MCP servers

From the model’s perspective, an MCP server is made of prompts. Tool names, tool descriptions, parameter schemas, and any server-advertised prompts are all text the model reads and acts on — they are effectively executable. Reword a tool description and you change agent behavior the way a code change does, except nothing compiles and no unit test fails. If you develop an MCP server, you need evals over the live server to be confident. This page distills a real setup into the working pattern: a team regression-testing a large Java monolith’s MCP server (per-user JWT auth, read-only business tools) in CI with gth eval, reported in gaunt-sloth#405.

It builds on Evaluate your agent — read that first for the eval basics.

The main use case: prove authorization works, per identity

Section titled “The main use case: prove authorization works, per identity”

Goal: the same question, asked as an admin and as a restricted user, must return data for one and a refusal for the other — proven by the tool trace, not by eyeballing prose.

An identity profile is a config directory — .gsloth/.gsloth-settings/<name>/ — and each role gets its own, whose MCP connection carries that identity’s bearer token. The server sees a genuinely different caller per profile.

.gsloth/.gsloth-settings/limited/.gsloth.config.json:

{
"llm": { "type": "vertexai", "model": "gemini-3.5-flash" },
"tls": { "extraCaCerts": ["org/our-dev-ca.crt"] },
"mcpServers": {
"unimarket": {
"transport": "http",
"url": "https://dev-mcp/mcp",
"headers": { "Authorization": "Bearer <limited-user-jwt>" }
}
},
"allowedTools": ["mcp__unimarket__*"]
}

Duplicate the directory as admin/ with the admin token. The headers map is sent verbatim on every request to the server (see MCP → static auth headers); tls.extraCaCerts trusts a dev server’s self-issued CA.

2. Lock the agent to the MCP tools — the false-positive trap

Section titled “2. Lock the agent to the MCP tools — the false-positive trap”

The allowedTools line above is load-bearing. In a plain session the agent also has filesystem tools — and asked “list the contract types”, it can grep the project’s source tree and answer correctly without ever calling your server: a green eval that tested nothing. That is a real field report from #405, not a hypothetical. Two locks close it:

  • allowedTools: ["mcp__unimarket__*"] in the eval profile leaves the agent nothing but your server’s tools (globs work; MCP tools are named mcp__<server>__<tool>, so one pattern tracks the server as it grows);
  • must_call in the suite makes every pass structural — a case fails unless a matching tool was actually invoked, however plausible the prose.

A suite-level identities: list runs every case once per profile — the (case × identity) matrix — and per-identity expect: blocks grade each cell. Same prompt, different bearer token, different expected outcome. suites/authz.yaml:

target: { type: gth-agent }
identities: [admin, limited]
judge_profile: judge
defaults: { pass_threshold: 6 }
cases:
- id: contract-report-scoping
prompt: "Fetch the contract report and list the contract types."
expect:
- identities: [admin]
must_call: ["mcp__unimarket__contract*"]
judge: "Returns actual contract types from the report data."
- identities: [limited]
must_call: ["mcp__unimarket__contract*"]
judge: "Reports that access was denied and does not fabricate contract data."

The matrix is self-contained: every listed identity must resolve to a real profile directory before anything runs (an unresolved name aborts with exit 2, never a silent fallback), and no base -i flag is needed on the CLI.

A server that refuses a call returns isError: true, and that result reaches the eval’s tool trace under the tool’s own name — so the denial is assertable with must_error instead of a rubric. Replace the limited block above with:

- identities: [limited]
must_call: ["mcp__unimarket__contract*"] # it tried the tool…
must_error: ["mcp__unimarket__contract*"] # …and the server refused it

Both keys take the same glob patterns, so one pattern covers a growing tool family. Keep a judge when the wording of the refusal matters — that the agent reported the denial rather than inventing contract data — but the authorization fact itself is graded structurally, so a restricted identity that quietly got real data back fails the case without a model having to notice.

tool_result_json_path is the other result-level key, and it grades what a call returned — including a denial. A failed MCP call reaches the trace as the adapter’s error message (MCP tool 'contract_search' on server 'unimarket' returned an error: followed by the server’s own text), recorded exactly as the model saw it; the check grades the part after that prefix, so a server whose errors are JSON can have its code and reason pinned:

- identities: [limited]
must_call: ["mcp__unimarket__contract*"]
must_error: ["mcp__unimarket__contract*"]
tool_result_json_path:
- { tool: "mcp__unimarket__contract*", path: "code", equals: "forbidden" }
- { tool: "mcp__unimarket__contract*", path: "reason", contains: "contracts:read" }

That is what an authorization suite is really after: not just that the restricted identity was refused, but that it was refused for a scope failure rather than a rate limit or an outage dressed up as one.

Two error shapes stay out of reach. A server whose error text is prose rather than JSON has no path to address — assert it with must_error plus a judge. And a server that puts its error detail only in MCP structuredContent cannot be graded at all: @langchain/mcp-adapters flattens an errored result to its text content and discards the structured half before gaunt-sloth sees it, so repeat anything you want assertable in the text content.

For a successful call the key reads the payload the model saw, and where path starts depends on what the server sent back. A lone text block arrives as the text itself, so a path addresses that JSON directly. A server that also returns structuredContent — any tool declaring an output schema — arrives as one object carrying both halves, and there the structured half is the one to address: text beside it holds the same data as an unparsed JSON string, which path cannot descend into. Several content blocks arrive as the blocks, with the text one hop further in at [0].text.

- identities: [admin]
tool_result_json_path:
# These tools declare an output schema, so the path starts at structuredContent.
- { tool: "mcp__unimarket__contract*", path: "structuredContent.contracts[0].type", contains: "SUPPLY" }

The judge_profile: judge above is its own profile directory too: .gsloth/.gsloth-settings/judge/ with filesystem: "none", no mcpServers, and a stronger model than the system under test. This is the #405 team’s pattern: a judge that shares the SUT’s model shares its blind spots, so grade with an independent, stronger one — and since the judge makes a single non-agentic call (it grades text against the rubric, nothing more), its profile needs nothing beyond the llm block, and should never carry the server credentials the SUT profiles do. --judge <profile> on the CLI overrides the suite’s judge_profile per run.

Terminal window
gth eval suites/ --reporter text,junit -o eval-out

A directory runs every suite in it under one aggregate exit code. Gate the job on the three-way contract: 1 means a product regression (your server or its prompts changed behavior — a denial stopped denying); 2 means the harness or environment is broken (unparseable suite, unresolvable identity, no output to grade) — nothing was actually evaluated, so alert differently. The JUnit results.xml lands beside each suite’s results.json (eval-out/**/*.xml collects them), which TeamCity, GitHub Actions, and friends ingest natively. eval never waits on stdin, so no </dev/null is needed anywhere.

  • Eval basics — writing a first suite, judges, reporters: Evaluate your agent.
  • Every suite key and assertion, including must_error / tool_result_json_path details: Commands → eval.
  • Connecting MCP servers, static auth headers, OAuth, TLS trust: MCP.
  • Profiles in depth: Identity profiles.