witnora

Witnora Bench

Witnora Bench is an executable assurance benchmark for action-taking agents. Its primary metric is not ordinary task success:

Assurance Gap = apparent task success rate - independently verified success rate

The v0.1 gold set contains 24 deterministic tasks: six each for browser, coding, data, and messaging agents. Every task declares a public instruction, policy constraints, expected state transition, evidence requirements, a bounded resource budget, and an evaluator digest.

Run the reference suite

python -m mcpbench.cli bench run --output-dir .mcpbench/bench --workers 4

The checked-in observations intentionally exercise verified, policy violation, insufficient evidence, silent partial success, and ordinary task failure paths. The command therefore exits non-zero. It is a deterministic conformance and reporting reference, not a model leaderboard.

To execute a customer-owned adapter for every task, pass an executable and repeat --adapter-arg for its arguments:

python -m mcpbench.cli bench run \
  --adapter-executable python \
  --adapter-arg path/to/my_adapter.py \
  --output-dir .mcpbench/bench

The runner launches the command without a shell. It exposes WITNORA_BENCH_TASK_PATH, WITNORA_BENCH_OBSERVATION_PATH, and WITNORA_BENCH_TASK_ID. The task file excludes evaluator and fixture references. The adapter must write an agentcert.bench.observation.v0.1 JSON object to the requested output path.

For the four reference profiles, use a separate agent bridge and outcome probe. A profile selects only its matching six-task capability pack:

python -m mcpbench.cli bench run --adapter-profile browser-use --output-dir .mcpbench/browser
python -m mcpbench.cli bench run --adapter-profile coding --output-dir .mcpbench/coding
python -m mcpbench.cli bench run --adapter-profile data --output-dir .mcpbench/data
python -m mcpbench.cli bench run --adapter-profile messaging --output-dir .mcpbench/messaging

See Real-agent adapters for the process and evidence contracts.

The public four-agent calibration pins the agent, model, prompt, harness, and controls for one Browser Use, coding, data, and messaging run. Its checked-in results include normalized observations, independent outcome evidence, usage, and signed held-out evaluator receipts. The private predicates and signing key are intentionally not distributed with the reproduction kit. The separate reproduction repository contains the installable Inspect AI task, pinned methodology, no-key signature verification workflow, and a second clean-clone four-agent run.

The 24-task real-agent study extends that calibration to a preregistered model x harness x controls matrix. Version 0.5 runs every cell five times using five declared repetition seeds: 80 cells and 480 task executions in total. It reports pooled confidence intervals, between-run variance, paired effect sizes, exact repetition-cluster sign-flip tests, Holm-adjusted comparisons, task-level Assurance Gap, model cost, latency, failure taxonomy, and signed held-out evaluator receipts. The matching technical report is an arXiv-format draft with a clean-Linux reproduction appendix; it is not yet an accepted or independently replicated result.

Five repetitions are sufficient to expose run variance but not to reject an effect with a two-sided exact repetition sign-flip test at alpha = 0.05: the minimum attainable p-value is 0.0625. Effect sizes, cluster-bootstrap intervals, raw tests, and Holm-adjusted tests are all published so later higher-powered releases remain comparable without overstating this release.

Version 0.6 is frozen before collection under benchmarks/preregistrations/agentcert-bench-v0.6. The v0.5 pilot variance yields 29 analyzable repetitions for 80% planned power to detect a 10 percentage-point primary effect under a conservative eight-comparison bound. The protocol schedules 33 repetitions per cell after a 10% attrition reserve, for 3,168 task runs. This does not provide equivalent power for effects smaller than 10 percentage points.

Published versions remain visible at witnora.com/bench. Each benchmark tag packages the checked-in signed result, public evaluator key, study lock, report, release manifest, and SHA-256 checksums into a byte-stable archive. A separate manually triggered GitHub workflow performs the expensive clean Ubuntu reproduction so publishing a tag cannot silently start another paid 480-task experiment.

The public history is also evaluated by a versioned regression policy. CI blocks releases that cross declared verified-success, disagreement, Assurance Gap, infrastructure-error, or normalized-cost limits and publishes warnings below the blocking thresholds. A third party can verify the current release, all signatures, the frozen next protocol, and the regression decision without a model key:

bash reproduction/agentcert-bench-v0.6/review.sh

Outputs:

• summary.json: aggregate apparent/verified success, Assurance Gap, evidence, policy, usage, cost, and latency metrics.
• results.jsonl: one normalized result per task.
• analysis-families.csv: pooled rates and run variance for every repeated cell.
• analysis-comparisons.csv: effect sizes, cluster intervals, raw tests, and Holm-adjusted tests for control and model comparisons.
• report.md: human-readable benchmark report.
• junit.xml: CI result format.
• replay-manifest.json: suite, result, observation, and evaluator digests.

Verify that a prior result was not modified:

python -m mcpbench.cli bench replay --run-dir .mcpbench/bench

Export the public tasks as Inspect AI-compatible JSONL records:

python -m mcpbench.cli bench export-inspect -o .mcpbench/bench/inspect.jsonl

Run a bounded model x harness x controls matrix:

python -m mcpbench.cli bench matrix \
  --spec benchmarks/research-matrices/baseline-v0.2.yaml \
  --output-dir .mcpbench/matrix \
  --push

The matrix stores every cell independently. An infrastructure error in one cell does not erase other results or become an agent task failure.

Evaluator boundary

Public development predicates live separately from task input and run in an independent subprocess after the adapter emits an observation. The execution adapter receives the public task, never the predicate.

The subprocess boundary prevents accidental predicate disclosure to the agent adapter; it is not an operating-system sandbox for untrusted code. Official held-out runs should place agent execution and evaluation in separate containers or services with separate credentials. The evaluator service under services/agentcert-bench-evaluator retains predicates, limits request size and concurrency, and returns a signed digest-bound receipt.

The official held-out variants are not checked into this public repository. heldout/manifest.yaml declares their distribution and delivery boundary. A hosted or third-party evaluator can use the same predicate contract while disclosing only its digest and result. Checking a predicate into a public repository makes it reproducible, not hidden; Witnora does not claim otherwise.

What this proves

• The benchmark contracts validate consistently across four capabilities.
• Outcome, policy, and evidence checks are distinct and observable.
• Evaluator digests bind a result to the predicate used.
• Runs are bounded, parallel, retry-aware, and produce portable CI artifacts.
• Usage, cost, latency, evidence completeness, and Assurance Gap are collected.
• Real runs can upload dynamically to the Hosted Benchmark view with --push.
• An installable Inspect AI task and independent reproduction repository are public. Registration is proposed upstream without claiming acceptance.

What this does not prove

• The deterministic fixtures do not rank a real model or harness.
• Public predicates do not provide contamination-resistant held-out evaluation.
• Digest verification does not reproduce an external website, model, or vendor service unless its adapter also captures and restores that environment.
• A passing task is scoped evidence, not a guarantee that an agent cannot fail.
• Reference adapter code is not evidence that a named external agent was run; reportable claims require captured run artifacts and evaluator receipts.

This site is open source. Improve this page.