MCP CI/CD
Gate every release on how models use your server.
A green unit test says a tool still responds. It can't tell you a reworded tool description quietly broke how models call it. MCPJam evals run on every PR through GitHub Actions, with accuracy tracked over time, so behavior regressions fail the gate instead of reaching users.
npm test # your @mcpjam/sdk eval suitesHow do I test an MCP server in CI/CD?
To test an MCP server in CI/CD, wire MCPJam's CLI or SDK into GitHub Actions and run an eval suite on every pull request. The evals score how a real model uses your tools and return pass/fail exit codes, so a behavior regression fails the build, the kind a unit test misses when a reworded tool description quietly changes how models call the tool.
Why MCP servers need a behavior gate in CI
The changes that break an MCP server rarely break its tests. A tweaked tool description, a renamed argument, or a new optional field all pass unit tests and still change whether a model selects the tool or fills it correctly. Without a behavior check in the pipeline, that regression ships, and you find out from users.
MCPJam closes the gap: the same evals you run locally run in CI on every pull request, scoring tool selection and task completion with pass/fail exit codes. A regression fails the build, and accuracy is tracked across runs so you can see behavior trend over time.
Evals in your pipeline
Runs on every PR
Wire the CLI and SDK into GitHub Actions or any pipeline so eval suites run automatically when a change lands.
Pass/fail exit codes
Evals return standard exit codes, so a behavior regression fails the gate exactly like a failing test, no dashboards required.
Accuracy over time
Results are tracked across runs, so you can see whether a change improved or degraded how models use your server.
Add the gate to GitHub Actions
Run your @mcpjam/sdk eval suites in CI through your test runner; a non-zero exit fails the job. (Setup: see the CI/CD eval gates page.)
- name: MCP eval gate
run: npm test # @mcpjam/sdk eval suitesOpen source, used in the open
Frequently asked questions
Wire the MCPJam CLI or SDK into GitHub Actions or your pipeline to run an eval suite on every PR. Evals return pass/fail exit codes, so a behavior regression fails the build.
Behavior regressions: a reworded tool description or renamed argument that still passes unit tests but changes whether a model calls the tool correctly.
Yes. The CLI runs in GitHub Actions or any CI system; the exact workflow YAML is on the CI/CD eval gates page and is marked illustrative until you wire your suite.
Yes. Eval results are tracked across runs so you can see whether a change improved or degraded how models use your server.
Stop shipping behavior regressions.
Run your MCP evals on every PR and fail the gate before users hit the change. Open source.
Open source core · no credit card