witnora

Witnora Bench 24-task study

This study runs 24 executable tasks through four real open-source agent harnesses and two pinned model snapshots. Each model/harness pair is tested with policy text alone (baseline) and with deterministic enforcement at the tool or action boundary (enforced).

The complete v0.5 matrix is 2 models x 4 harnesses x 2 controls x 5 repetitions x 6 tasks: 480 task runs. It measures apparent success, independently verified success, Assurance Gap, policy violations, evidence completeness, token usage, list-price model cost, latency, and failure taxonomy. It is a bounded study, not a general model leaderboard.

The frozen v0.6 preregistration uses the v0.5 pilot variance to schedule 33 repetitions per cell and 3,168 task runs. The manual formal workflow reads the frozen seeds directly from that protocol; CLI flags cannot silently reduce or replace its repetition set.

Pinned study

• Browser: Browser Use 0.13.7
• Coding and data: smolagents 1.26.0 tool agents
• Messaging: LangGraph 1.2.10
• Models: gpt-4.1-mini-2025-04-14 and gpt-5-mini-2025-08-07
• Tasks: six each for browser, coding, data, and messaging
• Controls: baseline and enforced
• Repetitions: five locked, independent invocations per cell
• Repetition seeds: five preregistered values in study.lock.json

Exact model prices and all study dimensions are recorded in study.lock.json. Prices use the OpenAI model pages for GPT-4.1 mini and GPT-5 mini.

Evaluator boundary

Public task manifests do not contain the held-out predicates. The Agent bridge, outcome probe, and evaluator are separate processes. The evaluator service owns the private predicates and Ed25519 signing key, binds every result to the observation and predicate digests, and returns a signed receipt. The public key is checked in; the private key, API token, fixture seed, and predicates are not. The orchestrator removes the seed before launching the Agent and probe and passes only the task-specific fixture marker required to initialize the local synthetic system.

Run

Create isolated Python environments for the four harnesses, start the held-out evaluator, and set these variables without committing their values:

$env:OPENAI_API_KEY = '<provider key>'
$env:WITNORA_BENCH_EVALUATOR_API_KEY = '<evaluator token>'
$env:WITNORA_BENCH_HELDOUT_VARIANT_SEED = '<fixture seed>'
$env:WITNORA_BROWSER_PYTHON = '<browser-use environment python>'
$env:WITNORA_TOOL_AGENT_PYTHON = '<smolagents environment python>'
$env:WITNORA_MESSAGING_PYTHON = '<LangGraph environment python>'

python examples/agentcert-bench-24-task-study/run_study.py `
  --evaluator-url http://127.0.0.1:8091 `
  --repetitions 5 `
  --output D:\agentcert-results\study-v0.5 `
  --temp-root D:\agentcert-results\tmp

The scratch directory is explicit because browser frameworks can allocate substantial temporary state. Infrastructure failures are reported separately and excluded from behavioral-rate denominators.

Publish and verify

The publication command copies only portable summaries, task-level results, reports, replay manifests, and signed evaluator receipts. It excludes raw work directories, prompts, screenshots, private predicates, keys, and tokens.

python examples/agentcert-bench-24-task-study/publish_results.py `
  --source D:\agentcert-results\study-v0.5 `
  --destination examples/agentcert-bench-24-task-study/results-v0.5

python examples/agentcert-bench-24-task-study/verify_results.py `
  --results examples/agentcert-bench-24-task-study/results-v0.5 `
  --public-key examples/agentcert-bench-24-task-study/evaluator-public-v0.5.pem

The Witnora Bench reproduction workflow verifies every checked-in receipt on pull requests without a model credential. A maintainer can manually dispatch the formal Ubuntu job to build isolated harness environments and execute all 480 task runs. Tags matching agentcert-bench-v* package the already verified, checked-in result and publish an immutable GitHub release; tags do not start a second paid experiment.

Derive the public trend entry directly from the signed analysis rather than copying metrics by hand:

python examples/agentcert-bench-24-task-study/update_public_history.py `
  --analysis examples/agentcert-bench-24-task-study/results-v0.5/analysis.json `
  --history packages/agentcert-dashboard/src/benchmark-history.json `
  --version v0.5 `
  --published-at 2026-08-03 `
  --artifact-url https://github.com/Kakarottoooo/agentcert/releases/tag/agentcert-bench-v0.5.1 `
  --paper-url https://github.com/Kakarottoooo/agentcert/blob/main/papers/agentcert-bench-v0.5/main.pdf

Interpretation

This study proves that the named framework versions executed the declared synthetic tasks and that the published observations were scored by the signed held-out evaluator. It does not establish production safety, rank whole agent products, or constitute independent third-party certification. With six tasks per repetition, task coverage remains bounded. Five repetitions expose run-to-run variance but do not turn this synthetic fixture suite into a general leaderboard.

For GPT-4.1 mini, the pinned harness requests the preregistered provider seed for each repetition. The pinned GPT-5 mini harness does not expose provider seed support, so its five repetitions are independent API invocations labeled with the same preregistered run seeds rather than deterministic provider-seeded samples. The analysis reports this distinction explicitly.

The reference adapter remains part of the trusted computing base. A signed evaluator receipt proves that a predicate scored a particular observation; it does not prove that a malicious adapter truthfully reported the source system.

The arXiv-format reports and reproduction appendices live under papers.

This site is open source. Improve this page.