MCP evals
Know how models actually use your MCP server.
Protocol checks prove a tool responds. Evals prove a real model picks the right tool, sends the right arguments, and completes the job across many realistic requests, scored and repeatable. Generate a suite, run it, and read the failure patterns.
npx @mcpjam/inspector@latestWhat are MCP evals?
MCP evals are repeatable evaluations that run a real language model against your MCP server and score whether it selects the right tool, sends the right arguments, and completes the task across many realistic requests. MCPJam generates the suite from your server's tools, reports accuracy, latency, tokens, and tool calls, and diagnoses the failure patterns, so you catch behavior problems a protocol check never sees.
Why evaluate an MCP server at all?
An MCP server passes its unit tests and still fails in production, because the caller is a language model. A reworded tool description, an ambiguous argument, an over-eager tool: none of these break a protocol check, but all of them change whether the model uses your server correctly. Evals are how you catch that class of failure before your users do.
MCPJam generates an eval suite from your server's tools, runs each case through a real model, and reports accuracy, latency, tokens, and tool calls across the whole suite, then diagnoses the failure patterns and lets you replay any failing case step by step.
Generate, run, diagnose
Generated suites
MCPJam derives test cases from your server's tools, from a single call to multi-step requests. In hands-on testing against a Google Calendar server it produced eight cases automatically.
Real-model runs
Each case runs through a real model. The report aggregates accuracy, latency, tokens, and tool calls across the suite so you can compare runs over time.
Run Insights
Failure patterns are diagnosed for you (over-eager tool use, incorrect date grounding, repeated retries), and every failing case is replayable.
MCP evals vs agent evals: which do you need?
They measure opposite sides of the handshake. Agent evals score the end-to-end agent you built, in a system you control. MCP evals score how a model uses the server you shipped, when a client you don't control calls it. If you're shipping an MCP server, that's the side that decides whether it works in the wild.
Open source, used in the open
Frequently asked questions
Repeatable evaluations that run a real model against your MCP server and score whether it selects the right tool, sends the right arguments, and completes the job, plus accuracy, latency, tokens, and tool-call metrics.
Run npx @mcpjam/inspector@latest, select your server, and Generate a suite. MCPJam derives cases from your tools, runs them with a real model, and diagnoses failure patterns. Free and open source.
Yes. The CLI and SDK wire into GitHub Actions or any pipeline to run evals on every PR with accuracy tracked over time. See MCP CI/CD testing.
Observability watches the agent you built in production. MCP evals test, before production, how external agents use the server you shipped. They're complementary.
Find out how models really use your server.
Generate a suite, run it with a real model, and see the failure patterns. Free, in about two minutes.
Open source core · no credit card · ~2 min
Keep exploring
Generate, run, and diagnose eval suites in detail.
Two eval categories that measure opposite sides of the handshake.
Run evals on every PR and gate releases on model behavior.
Where evals sit in the full testing workflow.