witnora

Witnora Bench real-agent adapters

Witnora Bench uses two customer-owned processes per task:

The agent bridge runs the subject agent and records its apparent result, actions, latency, tokens, and cost.
The outcome probe independently reads the task system after execution and records observed state plus evidence artifact paths.

The processes cannot be the same command. Neither receives the evaluator predicate. Witnora hashes probe artifacts and sends only the normalized observation to the evaluator.

Shared contract

Choose browser-use, coding, data, or messaging. A profile runs only its six capability tasks.

export WITNORA_BENCH_AGENT_COMMAND_JSON='["python", "agent_bridge.py"]'
export WITNORA_BENCH_PROBE_COMMAND_JSON='["python", "outcome_probe.py"]'
python -m mcpbench.cli bench run --adapter-profile coding --output-dir .mcpbench/coding

The runner writes a public task to WITNORA_BENCH_TASK_PATH. The agent bridge must write WITNORA_BENCH_AGENT_RESULT_PATH:

{
  "observationId": "obs.coding.patch-single-file",
  "apparentSuccess": true,
  "actions": ["read:app.py", "write:app.py", "test:pytest"],
  "usage": {"input_tokens": 1200, "output_tokens": 300, "cost_usd": 0.02, "latency_ms": 9100}
}

The independent probe must write WITNORA_BENCH_PROBE_RESULT_PATH:

{
  "observedState": {"files": {"app.py": "patched"}, "tests": "passed"},
  "evidence": [
    {"kind": "diff", "path": "diff.patch"},
    {"kind": "test_report", "path": "junit.xml"},
    {"kind": "trace", "path": "trace.jsonl"}
  ]
}

Evidence paths are relative to WITNORA_BENCH_ARTIFACT_ROOT; path escapes and missing files are rejected. The probe should use a read-only credential where possible and must not trust the agent’s final message.

Browser Use

The direct browser-use bridge is available as the agent command:

export WITNORA_BENCH_AGENT_COMMAND_JSON='["python", "-m", "mcpbench.bench.browser_use_bridge"]'
export WITNORA_BENCH_PROBE_COMMAND_JSON='["python", "probe_browser_state.py"]'
export WITNORA_BENCH_FIXTURE_URL_MAP_JSON='{"fixture://refund/clean":"http://127.0.0.1:4173/refund/clean"}'
export WITNORA_BENCH_BROWSER_USE_MODEL='gpt-4.1-mini'
python -m mcpbench.cli bench run --adapter-profile browser-use --model-id gpt-4.1-mini

The URL map is required for synthetic fixture:// resources. Optional WITNORA_BENCH_CDP_URL connects to a separately managed browser. The bridge limits allowed domains to the mapped host.

Coding agent

Use the generic CLI bridge when the agent accepts task context through an environment variable or wrapper:

export WITNORA_BENCH_SUBJECT_COMMAND_JSON='["my-coding-agent", "run"]'
export WITNORA_BENCH_AGENT_COMMAND_JSON='["python", "-m", "mcpbench.bench.command_agent_bridge"]'
export WITNORA_BENCH_PROBE_COMMAND_JSON='["python", "probe_git_and_tests.py"]'
python -m mcpbench.cli bench run --adapter-profile coding --harness-id my-coding-agent

The subject receives WITNORA_BENCH_INSTRUCTION and WITNORA_BENCH_PUBLIC_TASK_JSON. A wrapper may write actions.jsonl and usage.json next to the agent result. The probe should inspect the isolated checkout, git diff/status, and test report after the agent exits.

Data agent

Run the agent against an ephemeral database with a scoped role. The probe uses a separate read-only connection to compare row counts, authorized IDs, schema, and exported redaction. Query logs belong in evidence; database credentials do not.

python -m mcpbench.cli bench run --adapter-profile data --harness-id my-data-agent

Messaging agent

Run against a local mail sink or vendor sandbox. The agent credential may draft or send only within the task tenant. The probe independently reads message status, recipient, approval record, and delivery receipt.

python -m mcpbench.cli bench run --adapter-profile messaging --harness-id my-messaging-agent

Held-out evaluator and Hosted

export WITNORA_BENCH_EVALUATOR_API_KEY='set-in-secret-store'
export WITNORA_BASE_URL='https://witnora.com'
export WITNORA_PROJECT_ID='project-id'
export WITNORA_API_KEY='project-scoped-key'
python -m mcpbench.cli bench run \
  --adapter-profile coding \
  --evaluator-url https://evaluator.example.test \
  --evaluator-public-key evaluator-public.pem \
  --push

The client verifies every Ed25519 receipt before retaining the result. Hosted recomputes aggregate metrics from task results, checks receipt bindings, and rejects idempotency conflicts. Hosted records that signed receipts are attached; signature verification remains the runner’s responsibility until a trusted evaluator key registry is configured server-side.

This site is open source. Improve this page.