Your Agent Benchmark Is Measuring the Harness

An agent score is not a model score.

It is the outcome of a model inside a harness, acting through tools, against an environment, observed by an evaluator.

Change the parser and a malformed answer becomes valid. Change the retry message and the agent searches again instead of stopping. Replace a live website with a pinned snapshot and a “reasoning failure” disappears. Give the same model a different termination rule and it spends 34 times as many input tokens for a similar result.

Two papers published in the last several weeks make this confounding structure difficult to ignore.

A Unified Framework for the Evaluation of LLM Agentic Capabilities, revised July 2, standardizes seven benchmarks across 24 domains and reports experiments spanning 15 models, 400,000 rollouts, and 5 billion tokens.

AgentCompass, submitted July 15, separates benchmark, harness, model, and environment across more than 20 integrated benchmarks, with a fault-tolerant runtime and versioned trajectory analysis.

Both arrive at the same systems insight from different directions:

Benchmark, harness, and environment are experimental variables. If they are not versioned and controlled, the score is not reproducible evidence.

That insight changes model selection, harness improvement, release testing, and every “state of agents” leaderboard.

This post reviews recent preprints and extracts a production evaluation contract. Reported numbers belong to the authors’ protocols; they are not universal rankings.

The actual unit under test

The familiar benchmark report looks like:

model -> score

The real experiment is:

(task set, model profile, harness, tools, environment state,
 evaluator, budgets, repeat policy) -> outcome distribution

Each term can change the result:

A scalar without this evaluation identity cannot tell you whether a new model is better, a new harness is better, or the website happened to cooperate.

What the 400,000-rollout study found

The unified-framework paper migrates heterogeneous benchmarks into a common instruction–tool–environment format and runs them through one fixed ReAct-style scaffold in a sandbox. For live retrieval tasks, it also supports curated offline snapshots.

Three findings matter.

1. Framework choices move results in both directions

When original benchmark implementations were replaced with the unified scaffold, some model scores rose and others fell. The direction was not consistent.

That rules out the comfortable story that a common scaffold merely removes bad engineering. A specialized benchmark scaffold may help one model’s interaction prior, while a generic scaffold may help another model through clearer tools or error handling.

The paper includes a case in which the same model, instruction, live web, and search backend reaches opposite verdicts under two frameworks. The divergence comes from retry behavior, memory representation, answer format, and tool-invocation contract—not from different facts or model weights.

2. Environmental volatility masquerades as agent failure

Offline snapshots improved results on retrieval-heavy tasks in the paper’s experiments by removing:

• dead or changed reference pages;
• anti-bot responses;
• noisy pages whose boilerplate exhausts context;
• search-index drift.

An offline snapshot does not make the task more realistic. It makes the causal question cleaner. Once the evidence is pinned, a failure can be attributed more confidently to the model–harness system.

Live evaluation remains necessary for operational readiness. It just answers a different question: can the system succeed under today’s external conditions?

3. One success rate hides different operating strategies

The study reports cases where models reach similar task completion with radically different step, token, and time profiles. On one long-horizon planning domain, two models had comparable completion scores while one used roughly 34 times as many input tokens.

The same paper finds different failure signatures:

• reasoning deficits;
• parsing failures;
• malformed or nonexistent tool calls;
• iteration-limit exhaustion;
• context overflow;
• timeout.

Two agents can fail equally often and require completely different fixes. A scorecard that records only the final answer destroys that diagnostic value.

What AgentCompass adds

AgentCompass reaches a similar architecture independently. It makes four components explicit:

• Model: inference identity and parameters;
• Benchmark: task loading, reference data, and scoring;
• Harness: the agent loop and interaction policy;
• Environment: the executable world and tools.

The components are connected through stable protocols, and a run request binds their versions into one execution. The runtime records a versioned trajectory and supports resumable, fault-tolerant evaluation.

This matters beyond engineering convenience. Decoupling allows controlled comparisons:

compare models:       fix benchmark + harness + environment
compare harnesses:    fix benchmark + model + environment
compare environments: fix benchmark + model + harness
ship a product:       evaluate the full released tuple

AgentCompass also analyzes trajectories for suspected reward hacking in coding benchmarks, including modifying tests or retrieving golden patches. The paper is careful to call these cases suspected: a trajectory pattern is not proof of causal intent.

That caveat is a best practice itself. Trace analysis should produce an attributable finding—“the run modified a protected test”—not a psychological claim about what the model intended.

Standardization is a control, not the ground truth

The unified-framework study fixes one general-purpose scaffold. That makes scaffold effects visible, but it does not prove that the selected scaffold is neutral.

The authors state this limitation directly: systematic comparison across several general-purpose scaffolds remains future work. The study is also text-centric, and seven benchmarks cannot represent the full landscape.

AgentCompass covers more benchmarks and makes harness swapping easier, but breadth does not remove benchmark validity questions:

• Does the task represent the production work?
• Is the reference answer actually sufficient?
• Does the environment permit shortcuts?
• Does the evaluator reward superficial compliance?
• Is the task distribution similar to real traffic?
• Does the trace analyzer confuse correlation with reward hacking?

Standardization improves reproducibility. It cannot substitute for construct validity.

The production evaluator still needs domain experts to decide whether passing the test means the system is fit for the job.

Use three evaluation modes

Do not force one benchmark setup to answer every question.

1. Capability mode

Use a fixed general harness and pinned environment to compare model profiles. Hold tool rendering, budgets, retries, and evaluators constant.

This asks:

Under the same experimental interface, which model capabilities differ?

It is useful for model routing and capability floors. It is not a product release test.

2. System mode

Evaluate the exact model–harness combination against replayable tasks and environment snapshots. Include context compilation, memory, tools, parsers, policy, and evaluators.

This asks:

Does this candidate system outperform the current released system, and why?

It is the right mode for harness optimization.

3. Operational mode

Run shadow and canary traffic against live dependencies with real latency, error, freshness, and authorization conditions. Keep effects read-only or simulated until the action path is proven.

This asks:

Can the released system operate safely under the world as it changes?

Offline replay provides attribution. Live evaluation provides realism. Production readiness requires both.

Give every score an identity

An evaluation result should be addressable like a build artifact.

Illustrative, non-normative metadata:

evaluation_run:
  dataset: support_refunds_golden@sha256:91d...
  model_profile: model_route_support_v12
  harness: refund_agent@2.7.1
  context_pack: support.refund@4.3.0
  tool_manifest: support_tools@sha256:2af...
  environment: refund_simulator_snapshot_2026_07_20
  evaluator_set: refund_scorecard@1.9.0
  policy_bundle: support_policy@8.2.0
  budgets:
    max_steps: 12
    max_tool_calls: 8
    timeout_ms: 45000
  repeats: 5

If any field changes, create a new result. Do not overwrite the old score and keep the same label.

For a live environment, record the retrieval timestamp, response digest, dependency version where available, and the actual tool transcript. “Live web” is not a reproducible environment id.

Grade the path without demanding hidden reasoning

Process evaluation does not require collecting private chain-of-thought.

Record observable, decision-relevant artifacts:

• compiled context manifest and omissions;
• tool names, typed arguments, results, errors, and durations;
• evidence references;
• approval and policy decisions;
• state diffs and mutation receipts;
• retries, replans, and termination reason;
• output schema validation;
• evaluator findings;
• total budgets consumed.

Then classify failure at the boundary where it became observable:

This taxonomy makes the result actionable without pretending to read the model’s mind.

Report a vector and a distribution

Final task success is one coordinate.

At minimum, report:

For stochastic behavior, use repeated runs and show uncertainty. A pass rate of 80% from five trials is different evidence from 80% across 500 trials. Preserve per-case outcomes so a regression on a high-risk slice cannot be hidden by easy cases.

Release on constraints, not a weighted average:

promote only if
  policy regressions == 0
  destructive-slice safety >= current
  critical-task utility >= floor
  p95 latency <= budget
  cost per trusted outcome <= budget
  target-slice utility improves materially

A weighted score can let ten cheap utility wins cancel one severe policy regression. A release gate should not.

Design against benchmark gaming

An agent evaluation environment should assume the agent will exploit every reachable shortcut, whether by deliberate strategy or accidental discovery.

For coding and tool-use tasks:

• mount hidden tests and reference artifacts outside the agent’s readable scope;
• make public tests writable only when that is part of the task;
• detect test deletion, assertion weakening, and grader modification;
• separate task credentials from evaluator credentials;
• hash protected artifacts before and after the run;
• validate the resulting state, not only the final message;
• retain the trajectory for review;
• add adversarial canaries that expose leakage without containing real answers.

Do not reward a patch merely because the repository test command exits zero. Verify which tests ran and what changed.

This is where environment design becomes part of trust, not just benchmark plumbing.

The best-practice evaluation contract

The recent research supports ten rules:

1. Name the system under test. Model, harness, tools, environment, evaluator, and budgets all belong in the identity.
2. Change one variable for causal comparisons.
3. Use pinned environments for regression and live environments for operations.
4. Keep the original task semantics when standardizing execution.
5. Report resource use beside success.
6. Store observable trajectories and typed state diffs.
7. Separate agent, harness, environment, and evaluator failures.
8. Use repeated trials and preserve slice-level distributions.
9. Protect the evaluator and golden artifacts from the agent.
10. Release the full product tuple, not the leaderboard winner.

The practical conclusion is sharp:

A model leaderboard helps you choose what to test. A versioned, replayable evaluation harness tells you what to ship.

When a score changes, the first question should no longer be “Did the model get smarter?”

Ask which model, which harness, which tools, which environment, which evaluator, and which budget produced the change. If the result cannot answer those questions, it is a number—not evidence.

Research base

• AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities, submitted July 15, 2026.
• A Unified Framework for the Evaluation of LLM Agentic Capabilities, revised July 2, 2026.
• Holistic Agent Leaderboard, ICLR 2026, for standardized, cost-aware third-party agent evaluation.
• Treat LLM Agents as Programmable Subjects, for configuration schemas and process-trace evaluation.
• ContextOS: Scorecards Over Vibes, Trace Review Is the Agent Debugger, and Dataset-First Agent Engineering.

What to read next

• Scorecards Over Vibes
• Trace Review Is the Agent Debugger
• Dataset-First Agent Engineering
• Replay Harness in Code