MCP CI/CD actions
Catch the change that silently breaks how clients use your server.
A green unit test says a tool still responds. It can't tell you a reworded tool description quietly broke how clients call it. MCPJam evals run on every PR through GitHub Actions, with accuracy tracked over time, so behavior regressions fail the gate instead of reaching users.
Behavior regressions, caught in CI
Runs on every PR
The CLI and SDK wire into GitHub Actions or any pipeline, so eval suites run automatically on each pull request.
Accuracy over time
Accuracy is tracked across runs, so you can see a score drift down before it becomes a user-facing bug.
Gate the merge
Pass/fail exit codes mean a failing suite blocks the merge, the same way a failing test would.
An example GitHub Actions job
Illustrative: runs your @mcpjam/sdk eval suites through your test runner in a standard workflow. Check the SDK docs for setup, and the CLI docs for the conformance-suite gates.
name: MCP eval gate
on: [pull_request]
jobs:
evals:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
- run: npm ci
# @mcpjam/sdk eval suites run in your test runner; a non-zero exit fails the job
- run: npm test
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}Frequently asked questions
The @mcpjam/sdk runs eval suites in your test runner (Jest, Vitest, or any runner) on every PR, returning a pass/fail exit code so a regression blocks the merge. The CLI adds health-check and conformance-suite gates, and hosted eval runs track accuracy over time.
Unit tests confirm a tool still responds. Eval gates confirm a real model still uses it correctly, catching things like a tool-description change that silently breaks tool selection.
It's illustrative: it shows how @mcpjam/sdk eval suites fit a standard GitHub Actions job through your test runner. Check the MCPJam SDK and CLI docs for setup and the exact conformance-gate commands.
Gate releases on real model behavior.
Add an eval suite to your pipeline and stop behavior regressions before they ship.
Open source core · runs in any CI