Agent Harness Benchmark: An Experimental Protocol Beyond Model Scores

A harness upgrade adds retries. Task completion rises. Did reliability improve?

Perhaps. Or the system performed the same payment twice, hid the first timeout, and counted the second successful response as a win. A benchmark that inspects only the agent’s final message cannot distinguish those outcomes.

An Agent Harness Benchmark should measure how the execution system changes completion, authority enforcement, recovery, and resource use. This post specifies an experimental protocol; it does not announce an implemented benchmark, measured results, or a leaderboard. All numerical examples below are illustrative. Sources were checked on September 6, 2026.

Build on existing evaluation work

τ-bench couples a conversational user simulator with domain tools and policy, then compares final database state with annotated goals. Its pass^k metric asks about success across all repeated trials, rather than whether one of several attempts succeeds. It offers a useful foundation for repeatability, although a simulator is not a production user population.

AgentDojo pairs agent tasks with prompt-injection attacks and evaluates utility alongside attack success. Its important methodological lesson is to measure defenses against both malicious and benign inputs. A defense can prevent attacks by preventing useful work.

The ML Test Score also treats production readiness as a collection of data, model, infrastructure, and monitoring tests. Borrow that systems perspective, without assuming an ML checklist already covers agent authorization and side effects.

Our earlier analysis of harness-confounded benchmarks explained the attribution problem. The contribution here is a proposed protocol with experimental units, fault semantics, denominators, and release artifacts.

State the causal question

Represent one tested system as:

X = (task corpus, model profile, harness revision, tool contract,
     environment snapshot, user simulator, evaluator, budgets)

The harness revision includes prompts, parser behavior, context assembly, memory policy, planning, retries, approval handling, termination, and recovery. Model identity includes decoding settings and any available provider revision. If the provider exposes only a mutable alias, record that limitation and run date.

The primary comparison is the mean paired difference between harness B and harness A over the same tasks, model profiles, and externally defined scenarios. That estimates improvement over the sampled distribution. It does not identify a universal harness quality independent of models or tasks.

Use three distinct tracks:

Recorded-output replay is appropriate for the first track. Feeding harness B the exact tool sequence generated by A would suppress B’s adaptive behavior, so it is not a valid substitute for the interactive track.

Use a crossed, paired design

A practical initial design could use two harness versions, three model profiles, 60 held-out scenarios, five operational conditions, and ten repeats: 18,000 interactive runs. This is a planning example, not a power calculation or a completed experiment. Determine required sample size from the smallest meaningful regression and pilot variance.

Cross both harnesses with all model profiles. Pair runs by task, scenario, initial state, and externally specified fault seed. Randomize execution order to reduce time-of-day and provider-load confounding. A common seed does not make two LLM transcripts identical; record actual transcripts and distinguish controlled randomness from provider nondeterminism.

Report the harness difference per model as well as the aggregate. A harness may improve one model and degrade another. The interaction is useful evidence, not statistical noise to average away.

Split by task families, entity templates, and failure mechanisms where possible. Paraphrases of one scenario should not leak across development and final evaluation. Freeze the held-out manifest before optimization, and record every evaluated candidate to reveal selection effects.

Make faults correspond to semantic events

“Fail tool call number three” is not a stable treatment when harnesses choose different call sequences. Prefer triggers such as “the first attempted order creation for this logical operation.” If a harness never reaches that boundary, report the fault as unexposed rather than claiming it recovered.

The following is a proposed fixture format, not a ContextOS runtime schema:

scenario_id: order_commit_response_lost_v1
initial_state_ref: fixtures/orders/eligible-order-v1
user_goal: complete_one_authorized_order
fault:
  trigger: first_create_for_logical_operation
  provider_behavior: commit_then_drop_response
  repeat_behavior: return_original_if_contract_allows
oracle:
  required_effects: [one_matching_order]
  forbidden_effects: [duplicate_order, unauthorized_charge]
  unresolved_at_deadline: incomplete

All referenced fixtures, deadlines, and provider behaviors must exist before this becomes executable. A local deduplication map must not simulate provider guarantees that the real integration lacks.

A minimum coverage matrix

Each condition also needs a benign counterpart. For example, include an approved destination that should be allowed as well as an injected destination that should be blocked. Otherwise a deny-all harness can appear excellent.

Separate the task oracle from the trace oracle

The task oracle checks independently observed outcomes: the right order exists, the expected refund total holds, or the requested document was delivered to the approved recipient. The trace oracle checks prohibited transitions, missing approvals, duplicate effects, and budget breaches throughout the run.

Define a trusted completion indicator for run r:

T(r) = goal_satisfied(r)
       AND no_forbidden_transition(r)
       AND required_receipts_present(r)
       AND within_declared_budget(r)

Final state alone can conceal a forbidden intermediate disclosure or a duplicate charge later refunded. The trace oracle therefore needs an independent provider event log or an instrumented environment. The harness cannot certify its own completeness by omitting a bad event.

Protect evaluator fixtures and expected outputs from the agent. Natural-language judges can assess communication quality, but should not decide whether a payment occurred when an authoritative state query is available. Review ambiguous oracle failures separately, preserving the original score and any later corrected corpus version.

Publish a vector with explicit denominators

Publish counts with rates. If there are no trusted completions, cost per completion is undefined; do not report zero. Separate first-attempt completion from eventual completion after retries. A safe stop is valuable but does not fulfill the user’s task.

Distinguish infrastructure-invalid trials using a rule fixed before evaluation. A timeout caused by the tested harness is a result. An unrelated evaluator outage may invalidate a trial, but must remain visible in the run inventory. Report scheduled, executed, valid, invalid, and retried counts.

Measure consistency correctly

For a task with c successful trials among n exchangeable independent repetitions, and n >= k, the usual combinatorial estimators are:

all-k success:       C(c, k) / C(n, k)
at-least-one-of-k:   1 - C(n - c, k) / C(n, k)

Use the convention C(a,k)=0 when a<k. The first corresponds to pass^k; the second to pass@k, following τ-bench’s definitions. Apply the first to the trusted-completion indicator if you want policy-aware consistency, and name that adaptation explicitly.

With 8 successes in 10 trials, all-3 is 56/120 = 46.7%, while at-least-one-of-3 is 100%. Both describe the same illustrative sample. One asks whether repeated use is dependable; the other rewards having a successful attempt available.

Calculate per-task estimates, then macro-average over tasks with declared weights. Do not cube the overall success rate: task difficulty is heterogeneous. Persistent memory shared across repetitions also breaks the independence assumption; reset it or evaluate adaptation in a separate sequential track.

For uncertainty, bootstrap paired differences at the independent task-family level, preserving nested repeats and harness pairing. Publish the procedure, seed, and intervals. If shared outages or overlapping templates create larger clusters, use those clusters. Ten thousand correlated episodes do not provide ten thousand independent observations.

Attribute improvements through ablations

After the primary comparison, disable one changed mechanism at a time: receipt binding, conflict detection, retry classification, checkpoint recovery, or memory filtering. Keep these experiments in isolated environments; disabling a control is an experimental treatment, not a production recommendation.

An ablation can reveal that a gain disappears without a mechanism. It does not automatically establish that the mechanism is the sole cause: components interact. Predeclare the main comparisons and treat exploratory findings as hypotheses for another held-out test.

Use a release gate with separate constraints: no observed forbidden effects in the specified safety suite, acceptable benign overblocking, a justified completion non-inferiority margin, and cost and latency limits. Zero observed violations is a finite-sample observation, not proof of zero risk. Set thresholds from the workflow’s consequences rather than borrowing an arbitrary universal percentage.

What an honest benchmark release contains

Release the corpus and split manifests, harness revisions, model profiles, environment images, tool contracts, fault schedules, evaluator version, budgets, complete run inventory, redacted traces, raw counts, uncertainty calculation, and known limitations. Record which artifacts cannot be shared and why their absence limits reproduction.

The protocol is ready to implement; its empirical claims are not yet earned. Its value is that it makes an improvement claim falsifiable: another evaluator should be able to identify the tested change, reproduce the conditions, and challenge the score without trusting the agent’s narrative.

What to read next

• Scorecards over vibes covers the broader agent evaluation framework.
• Proof-carrying agent runtime defines the effect-boundary claims this protocol can test.
• From feature stores to agent memory develops the temporal-memory cases in the coverage matrix.