MCPBench is Witnora’s pre-release benchmark engine for MCP servers and agent-exposed tools. It answers: are this server’s tools safe, observable, reliable, and explainable enough to expose to agents?
MCPBench runs fully offline by default. It does not require OpenAI, Anthropic, local model, network, or production credentials.
uv pip install -e ".[dev]"
mcpbench doctor
Run a passing MCP/tool eval:
mcpbench eval --server-command "python examples/servers/github_like_server.py" --suite basic-tool-use --agent scripted --script passing --output-dir .mcpbench/passing
Run a failing behavior-chain eval:
mcpbench eval --server-command "python examples/servers/github_like_server.py" --suite untrusted-output --agent scripted --script failing-untrusted-to-sink --output-dir .mcpbench/failing
Outputs:
events.jsonlresults.jsonreport.mdbadge.svgMCPBench: 35/100
Assurance tier: No assurance tier
Critical violations: 1
High violations: 1
Key finding: sensitive synthetic canary reached a public/external sink.
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- run: pip install -e ".[dev]"
- run: mcpbench eval --suite basic-tool-use --agent scripted --script passing --output-dir .mcpbench/run
MCPBench results.json files are one of the inputs the unified Witnora CLI
accepts:
node packages/witnora-cli/dist/cli.js run --mcpbench .mcpbench/latest/results.json --out .agentcert/latest --subject my-mcp-server
ruff format --check .
ruff check .
mypy src/mcpbench
pytest