FrontierHarness Eval: Auditing the 17.5× Cost Gap

The September FrontierHarness release offers a useful opportunity to examine agent economics with the model held constant. Its most widely discussed result is a large cost spread. Before turning that spread into an architecture decision, inspect what the dollars divide by.

Our recomputation finds that the prominent cost figures match reported task spend divided by successful tasks. They are not the median cost of successful tasks. Some failed tasks also have missing cost values. These distinctions change how the comparison should be interpreted.

This article audits the public data available on September 10, 2026. The calculations were run against an immutable repository revision. It does not reproduce the original agent executions, validate the underlying task verifiers, or independently verify provider invoices. The analysis script and computed results make the arithmetic inspectable.

The experiment and its scope

Runta’s September 1 report describes nine harnesses in twelve configurations, using Kimi K3 on thirty software and terminal tasks. There is one canonical attempt per task–configuration pair: 360 cells, not 360 independent tasks per harness. The report describes common restored environments, a shared inference backend, and first-turn cache repricing. It also discusses model–harness–gateway compatibility. (Zhu and Mei, launch report.)

This is evidence about those configurations on that workload. A model-neutral experimental intention does not eliminate model–harness interaction. The integration adapter, protocol translation, stopping behavior, and cache semantics remain part of the deployed system being measured.

An engineer can use such a comparison to form a shortlist. Selecting a production default requires another experiment on the intended workload, including the model and provider path that will actually be used.

The denominator discrepancy

The launch table labels its cost column “Median cost per pass.” In the pinned JSON, the matching field is effective_cost_per_pass. Recomputing it from non-null task values of cost_first_cold_usd gives:

reported cost per success = sum(available task costs) / successful tasks

The two quantities are not synonyms:

successful-task median = median(cost of each successful task)

The first charges reported failed work to the delivered successes. The second describes a typical successful task and excludes the cost of failures. An especially expensive failure can move the first sharply while leaving the second unchanged.

Here are selected rows from our recomputation of the pinned task-level dataset:

Both cost columns use the source’s cost_first_cold_usd, preserving its normalization. The unrounded ratio of reported cost per success between Claude Code and Exo is approximately 17.54. The corresponding medians have a much smaller ratio. This is a metric-label discrepancy, not evidence that the underlying executions or costs were fabricated. Including failed work in an economic measure is often exactly what an operator needs.

Across the complete dataset, seventeen failed cells have null costs. Their actual costs are not established by the public task rows. The script preserves those missing values as a coverage limitation; it does not declare them zero. Assuming nonnegative costs, incomplete reported sums are lower bounds. Consequently, a ratio of reported sums is not a verified ratio of complete bills. Successful-task medians have complete cost coverage in this snapshot, but omit failures by definition.

Do not substitute these figures for current token prices or an all-in operating budget. Infrastructure, engineering, review, support, and incident costs are separate quantities.

A concrete accounting example

Consider an illustrative five-task workload with costs of $1, $1, $2, $8, and $20. Suppose the first three tasks pass and the last two fail.

The successful-task median is $1. The total bill is $32. Cost per success is $32 divided by three, or $10.67. Neither number is wrong. Reporting only $1 would conceal the economic impact of failures; calling $10.67 a median would conceal the distribution.

For production planning, publish at least the total bill, the success count, cost per success, and the distribution of costs for successes and failures separately. Also report completion time and review effort. A low inference bill can still produce expensive work if humans must reconstruct evidence or repair partial effects.

The business denominator should be an accepted outcome. A test pass, an agent’s completion message, a merged change, and a customer issue resolved without reopening are different events. Decide which event funds the system before choosing the metric.

What the pass-rate ordering establishes

Thirty tasks provide coarse resolution: one outcome changes the rate by approximately 3.33 percentage points. Rankings should therefore be read alongside task-level overlap.

Our paired recomputation compares success on the same task identities. For Codex versus Claude Code, three tasks pass only for Codex and two only for Claude Code. An exact two-sided McNemar test on those five discordant pairs returns an unadjusted p-value of 1.0. For Codex versus Hermes, the split is five versus zero, with p = 0.0625. These are exploratory calculations from the published outcomes, not new agent trials.

Across the eleven comparisons of Codex with the other configurations, none is below 0.05 even before adjustment for multiple comparisons. This does not demonstrate equal capability. Failure to reject a difference is not an equivalence result, and the curated tasks are not established as a random sample of production work. The tests make the limited evidence visible; they do not repair the sampling design.

Repeated trials would answer another question: how stable is each configuration on the same task? The single-attempt design cannot estimate that variability from within-cell repetition. A second dataset with deliberately repeated tasks should preserve every attempt and analyze dependence by task, rather than treating retries as additional unrelated observations.

Reproduce the arithmetic

The downloadable script uses only Python’s standard library. It validates the source hash, checks task identities and explicit success values, rejects invalid numeric costs, records missing-cost task IDs, verifies the recomputed aggregate, and emits all twelve configurations plus the paired comparisons.

curl -fL \
  https://raw.githubusercontent.com/frontier-harness-eval/eval/e837a70bd6beb4e72eeeda62dd06e3bd34f6cb63/results/eval-data.json \
  -o eval-data.json
curl -fL \
  https://contextosai.com/research/harness-september-2026/frontierharness_audit.py \
  -o frontierharness_audit.py
python3 frontierharness_audit.py eval-data.json > audit.json

The exact input SHA-256 is:

ffd18213a165e985fd7d876395a3f53e00961253a89bbc88420a41374bc20c4c

The source JSON reports an August 22 generation timestamp; the launch report is dated September 1. Generation, publication, and the September 10 audit are separate dates. No inference about individual trial dates is made from that metadata alone.

The core calculation is deliberately simple:

tasks = harness["task_details"]
successful = [task for task in tasks if task["success"]]
reported_cost_per_success = sum(
    task["cost_first_cold_usd"] for task in tasks
    if task["cost_first_cold_usd"] is not None
) / len(successful)

The published script adds validation and explicit missing-cost reporting around this arithmetic. Its results are a data audit, not a reproduction package for the original sandbox, inference service, or verifier execution.

Why restoring the environment is only part of fairness

Anthropic’s infrastructure study reports that changing resource enforcement can alter coding-benchmark results with the model and harness held fixed. It distinguishes reserved capacity from the hard limit that kills a process. That is a concrete reason to record how limits are enforced, not just a nominal memory allocation. (Segato, February 5.)

For an operational comparison, freeze the task environment, tool versions, model identifier, gateway behavior, retry policy, and scoring logic. Record network access, process limits, timeouts, and cache treatment. Separate agent failures from evaluator infrastructure failures using rules declared before running the comparison.

The FrontierHarness repository itself provides an important limitation: its current evaluation workflow defaults new runs to non-comparable because the original baselines did not record the applied network allowlist. It calls for a matched control under the new environment before claiming comparability. Respect that boundary when publishing a candidate result. (Repository evaluation guidance.)

This is a reason to preserve provenance, not to abandon benchmarking. Reproducibility improves when previously implicit environmental choices become explicit.

A production selection experiment

Start with a representative task inventory. Include straightforward successes, tasks needing clarification, difficult recoveries, and tasks the system should decline. Choose tasks before inspecting candidate rankings. Keep a final test partition protected from both implementers and the optimization loop.

Run at least two candidate harnesses with the intended model and provider path. If the decision includes a model upgrade, cross models and harnesses so a harness effect is not confused with a model effect. Pair by task and initial state; repeat enough tasks to estimate the reliability relevant to the business decision.

Set a budget for exploration separately from the budget for final evaluation. Otherwise, a candidate selected after many attempts has an unreported advantage over one tested once. Publish the search effort even if the final deployed harness is cheap.

The release decision should satisfy constraints before optimizing cost:

1. Required authority boundaries and effect checks hold.
2. Task quality meets the workload’s acceptance definition.
3. Recovery and repeated-run behavior meet the operating requirement.
4. Among eligible candidates, cost, latency, and human effort support the choice.

Do not combine a prohibited external effect and a useful answer into one average score. A safety failure may be a release blocker even when another metric improves.

What would change the conclusion

A larger, representative, repeated evaluation could establish a robust quality difference. A native-provider comparison could change the observed economics. A matched control could make new candidate results comparable. A corrected table label would resolve the presentation discrepancy without changing the underlying data.

For now, the defensible conclusion is narrower and more useful than a universal winner: the published configurations have materially different reported economics, the cost denominator and missing costs need explicit labeling, and the quality ordering carries substantial uncertainty.

Sources and reproducibility

1. Shilin Zhu and Shiqi Mei, Runta. Introducing FrontierHarness Eval. September 1, 2026. Experimental setup and the table label.
2. FrontierHarness Eval contributors. Pinned public dataset. Revision dated September 8, 2026; accessed September 10. Input to the original calculations above.
3. FrontierHarness Eval contributors. Evaluation workflow and comparability notes. Accessed September 10, 2026.
4. Gian Segato, Anthropic. Quantifying infrastructure noise in agentic coding evals. February 5, 2026.
5. ContextOS. Executable audit and full derived results. September 10, 2026. No original agent trials were rerun.

What to read next

• September harness engineering research map.
• The broader agent harness benchmark protocol.
• The experiment-driven publishing roadmap.