Witnora evidence is the portable review artifact for the project. It is meant to be generated by CI, local demos, or runtime gateways, then inspected by a developer, customer, auditor, insurer, or reviewer.
Witnora does not claim official certification status. The schema is a stable technical evidence format that can be mapped to standards and review language.
Witnora uses two version fields:
| Field | Meaning |
|---|---|
schemaVersion |
Compatibility family. Current value is agentcert.evidence.v0.1. Consumers can use this for broad parsing compatibility. |
schemaSemver |
Human-visible schema release. Current evidence-bundle value is 0.1.0. Consumers can use this for documentation and changelog references. |
The bundle also includes:
| Field | Value |
|---|---|
schemaName |
agentcert.evidence_bundle |
kind |
agentcert.evidence_bundle |
The top-level agentcert-evidence.json bundle requires:
| Field | Purpose |
|---|---|
schemaName |
Stable schema identifier for external consumers. |
schemaVersion |
Compatibility family. |
schemaSemver |
Schema release. |
kind |
Artifact kind. |
runId |
Unique Witnora bundle run id. |
generatedAt |
Bundle creation timestamp. |
subject |
Agent, MCP server, tool, or application under review. |
verdict |
Overall pass/fail, score, and level. |
summary |
Product coverage and evidence counts. |
results |
Normalized MCPBench, Tripwire CI, Onegent Runtime, or Witnora CLI results. |
evidence |
Flattened findings across all results. |
artifacts |
Named trace, report, screenshot, DOM, audit, or output paths. |
artifactManifest |
Optional v0.1 path, SHA-256, byte-size, and kind declarations for hosted reconciliation. |
standards |
Explicit standards mappings and non-certification notes. |
Each normalized product result requires:
| Field | Purpose |
|---|---|
schemaVersion |
Result compatibility family. |
product |
Producing engine: mcpbench, tripwire-ci, onegent-runtime, or agentcert-cli. |
runId |
Source run id. |
timestamp |
Source run timestamp. |
phase |
pre-release or runtime. |
score |
Normalized score from 0 to 100. |
passed |
Product-level pass/fail. |
artifacts |
Product artifact paths. |
evidence |
Findings produced by that product. |
Optional result fields include certLevel and summary.
Each evidence finding requires:
| Field | Purpose |
|---|---|
id |
Finding id. |
kind |
Finding category, such as policy_violation, assertion_result, trace_diff, runtime_warning, or audit_event. |
severity |
critical, high, medium, low, or info. |
message |
Human-readable finding. |
Optional finding fields include source, artifactPath, suggestedFix, and
metadata. Metadata is intentionally open so engines can preserve details such
as scenario name, fault name, policy id, verification result, or first
divergence point without breaking the stable base contract.
Corpus records are the data-flywheel layer. They normalize evidence runs into rows that can be appended to JSONL, SQLite, or Postgres, then summarized into a monitor snapshot.
Each corpus record now includes:
| Field | Purpose |
|---|---|
agentName |
Agent or adapter name for filtering and comparison. |
agentVersion |
Version label for longitudinal analysis; unversioned when unavailable. |
failurePatterns[].type |
Stable failure taxonomy bucket. |
failurePatterns[].suggestedType |
Automatic classifier suggestion before review. |
failurePatterns[].reviewStatus |
unreviewed, confirmed, or corrected. |
failurePatterns[].reviewConfidence |
Optional human confidence score from 0 to 1. |
failurePatterns[].reviewEvidenceContext |
Optional first-divergence snippet plus screenshot, trace, and step pointers. |
failurePatterns[].taxonomyRationale |
Optional structured explanation for why the final taxonomy label is correct. |
The dashboard does not read the database directly. It reads
agentcert.monitor_snapshot, which contains filters for agents, faults,
versions, products, failure types, and review status.
Human review records use agentcert.failure_review and are stored as JSONL.
They can target one exact record or a reusable pattern key. This is the
mechanism that turns the taxonomy from a rules-only classifier into a growing
failure dataset.
Required review fields:
| Field | Purpose |
|---|---|
schemaVersion |
Review compatibility family. |
kind |
agentcert.failure_review. |
id |
Stable review id. |
reviewedAt |
Review timestamp. |
reviewer |
Human or process that made the decision. |
status |
confirmed or corrected. |
target.patternKey |
Failure pattern being reviewed. |
type |
Effective taxonomy label after review. |
Optional fields include target.recordId, target.runId, target.product,
target.scenarioName, target.faultName, suggestedType, note,
confidence, evidenceContext, and taxonomyRationale.
confidence is normalized from 0 to 1. evidenceContext can include
firstDivergenceSnippet, screenshotPath, screenshotUrl, tracePath, and
stepIndex. taxonomyRationale stores primaryReason, optional
supportingSignals, optional contradictingSignals, and an optional
classifierLimitation. These fields make reviewed labels usable as a
training and evaluation dataset instead of only a UI correction ledger.
agentcert.failure_classifier_evaluation compares automatic failure taxonomy
suggestions against human-reviewed labels. It reports:
| Field | Purpose |
|---|---|
reviewedRows |
Count of failure patterns with human-reviewed labels. |
correctRows |
Count where suggestedType matched the reviewed type. |
incorrectRows |
Count where review corrected the automatic suggestion. |
precision |
correctRows / reviewedRows; 0 when there are no reviewed rows. |
coverage |
Reviewed failure patterns divided by all failure patterns. |
byType |
Per-reviewed-label precision. |
confusion |
Suggested label to reviewed label correction counts. |
This is deliberately evaluation output, not a trained classifier. It lets the project measure taxonomy quality as the reviewed dataset grows.
Witnora includes a local structural validator for checked-in examples, generated bundles, corpus records, failure reviews, monitor snapshots, and lab snapshots:
node packages/witnora-cli/dist/cli.js schema validate --schema evidence-bundle --file examples/agentcert/evidence-bundle.example.json
node packages/witnora-cli/dist/cli.js schema validate --schema corpus-record --file examples/agentcert/corpus-record.example.json
node packages/witnora-cli/dist/cli.js schema validate --schema failure-review --file examples/agentcert/failure-review.example.json
node packages/witnora-cli/dist/cli.js schema validate --schema classifier-eval --file examples/agentcert/classifier-eval.example.json
node packages/witnora-cli/dist/cli.js schema validate --schema monitor-snapshot --file public-demo/agentcert-monitor/data/monitor.json
The validator checks the stable contract fields and common enum values. It is not a cryptographic signature, certification authority, or proof that the underlying test run was honestly produced.
Consumers should treat schemaVersion as the compatibility family and
schemaSemver as the documentation/changelog release. Witnora v0.1 consumers
should ignore unknown optional metadata fields and fail closed only when a
required field, required enum, or required artifact kind is missing.
| Failure type | Meaning |
|---|---|
prompt_injection |
The agent followed or leaked adversarial page/model instructions. |
wrong_click |
The agent clicked the wrong action target, such as a cancel or misleading button. |
timeout |
The agent or workflow exceeded the allowed step/time budget. |
verification_gap |
Execution happened, but observed state did not match expected state. |
silent_partial_success |
The run exited successfully while required state, URL, text, or result was missing. |
network_failure |
HTTP or network behavior prevented the expected task completion. |
ui_drift |
UI mutation such as renamed buttons, overlays, layout shifts, or disabled controls caused failure. |
policy_or_approval |
Runtime policy, approval, rejection, or governance behavior caused or explained the stop. |
agent_connection |
The harness could not observe a usable agent/browser connection. |
console_error |
Browser console errors violated the run assertions. |
assertion_failure |
Deterministic assertion failed without a narrower taxonomy match. |
unknown_failure |
Evidence did not contain enough detail for a narrower bucket. |
Witnora evidence can support these review concerns:
| Review concern | Witnora evidence language |
|---|---|
| AIUC-1 style independent review | Portable evidence bundle with repeatable scenarios, results, artifacts, and audit notes. |
| NIST AI agent standards direction | Traceability, testability, risk management, lifecycle controls, and human oversight evidence. |
| Audit and procurement review | Subject, run id, timestamp, evidence, artifacts, verdict, approval, verification, and action timeline. |
Recommended wording:
Witnora produces independent, reproducible evidence that a reviewer can map to its own agent assurance standard.
Avoid claiming:
schemas/agentcert-evidence-bundle.schema.json
schemas/agentcert-result.schema.json
schemas/agentcert-evidence.schema.json
schemas/agentcert-corpus-record.schema.json
schemas/agentcert-failure-review.schema.json
schemas/agentcert-failure-classifier-evaluation.schema.json
schemas/agentcert-monitor-snapshot.schema.json
schemas/agentcert-release-gate.schema.json
schemas/agentcert-evidence-signature.schema.json