Agent Evaluation After the Leaderboard: The KDD 2026 Production Assurance Playbook

On August 9, 2026, the KDD Workshop on Evaluation and Trustworthiness of Agentic AI made a quiet but important break with the usual agent leaderboard.

Its accepted papers did not ask only whether an agent reached the expected answer. They asked whether the benchmark measured a coherent capability, whether the harness created the score, whether a tool trajectory was efficient and grounded, whether external state remained consistent, whether a policy violation was silently accepted, whether an LLM judge hid review work, and whether a safe system drifted after deployment.

The shared message is sharper than “use more evals”:

An agent result is not trustworthy until the protocol, path, state transition, and operating lifecycle are trustworthy.

This post turns the workshop’s most production-relevant results into a release and monitoring playbook. It is not a ranking of the papers, and the reported findings belong to each author’s experimental setup. Most are workshop papers rather than broad deployment evidence. Where this article derives a ContextOS engineering pattern, it labels that move as an implementation inference rather than a result from the paper.

Our July review, Your Agent Benchmark Is Measuring the Harness, established that model, harness, tools, environment, evaluator, and budget form the real unit under test. The August research goes one level deeper: what evidence should that compound system produce before anyone is allowed to trust or ship it?

What changed in the August 9 research program

The workshop’s production signal can be organized into eight corrections to common evaluation practice:

The unit under test has expanded with the authority we give the system:

Those corrections are related. A model can finish the task while the database is wrong. A score can rise because the harness spends forty times more tokens. A judge can look accurate while silently transferring more cases to humans. A multi-agent panel can agree because debate erased its independent evidence.

The mistake is treating each of those as an edge case around the “real” evaluation. They are the evaluation.

1. There is no scalar called agent ability

Construct Validity Failures in Agentic AI Benchmarks audits five benchmarks across 15 models using a psychometric question: do tests presented as measures of “agent ability” actually behave like measures of one coherent construct?

The reported evidence says no:

• the mean cross-benchmark Spearman correlation was 0.67, with a range from 0.10 to 0.92;
• 22.1% of comparable model pairs reversed order across benchmark pairs;
• reasoning-specialized models that ranked near the top on GPQA Diamond and MMLU-Pro fell to sixth through twelfth on tool-use benchmarks;
• the correlation between τ-bench Airline and MMLU-Pro was 0.10 on a small overlapping sample.

The paper is careful about sample size and protocol comparability. Those caveats make the production conclusion more, not less, important: published scores already mix incomplete coverage and differing provider protocols, yet procurement and architecture decisions often treat the resulting order as stable.

A customer-support team does not need “the best agent model.” It needs evidence for a defined construct:

policy-compliant refund handling
under multi-turn ambiguity
with typed tools
inside a fixed authority and latency budget
against the actual support policy distribution

That construct should decompose into a vector—task utility, policy correctness, evidence sufficiency, state integrity, recovery, latency, and cost—not collapse into one average that allows strength in scientific reasoning to compensate for weak tool behavior.

Production rule 1: Measure named capabilities, not a scalar called “agent ability.”

2. The harness changes the economics even when pass rate barely moves

The Scaffold Effect in Coding Agents holds the model constant while comparing Goose, OpenCode, and OpenHands-SDK on a 50-task Terminal-Bench Pro subset.

Harness choice induced up to a 40× difference in tokens per solved task, while paired within-model pass-rate differences were usually between zero and eight percentage points. The failure signatures also followed the harness: reasoning failures clustered in one scaffold, verification and turn-limit failures in another, and idle-loop or timeout behavior in the third.

This changes the product question from:

Which model has the highest pass rate?

to:

Which harness–model pair produces an accepted result within our token, latency, supervision, and recovery budgets?

Tokens are not the only hidden cost. A no-action turn is a wait tax on the human. A verification loop that rarely changes the patch is a latency tax. A scaffold that stops early can lower spend while increasing rescue work. The scorecard needs the entire operating profile.

Production rule 2: Benchmark the harness–model pair, including its economics.

3. Every score needs a protocol card

Beyond Leaderboards: Protocol Cards for Trustworthy Coding-Agent Evaluation separates three sources of protocol dependence:

1. Information protocol: what context, files, diffs, tools, and prior state the agent receives.
2. Procedure protocol: how the agent generates, verifies, repairs, retries, and terminates.
3. Measurement protocol: how success, semantic quality, cost, and review work are counted.

Across 7,200 code-review records and 1,350 oracle-file apply-level repair records, the paper reports protocol-sensitive conclusions. Under its conditional apply-level setup, clean-apply rate moved from 8.0% with diff-only context to 50.4% with structured context and 54.7% with full raw context. Alternative cost accounting reversed the relative order of two multi-call workflows.

The lesson is not “always send the full repository.” The lesson is that a score without its information and accounting protocol is uninterpretable.

At minimum, a protocol card should record:

protocol_card:
  task_set: support_refunds_holdout@sha256:91d...
  model_profile: support_route_v18
  harness: refund_agent@3.2.0
  context_protocol:
    compiler: support.refund@4.4.1
    evidence_snapshot: policy_2026_08_09@sha256:2af...
    memory_policy: recall_policy_v7
  action_protocol:
    tool_manifest: support_tools@sha256:b31...
    approval_mode: local_write
    retry_policy: bounded_retry_v3
  measurement_protocol:
    evaluators: refund_scorecard@2.1.0
    repeats: 5
    cost_basis: model_plus_tools_plus_review
  budgets:
    max_steps: 12
    max_tool_calls: 8
    timeout_ms: 45000

This is an illustrative artifact, not a new ContextOS schema. In a ContextOS implementation, the canonical runtime primitives remain the contracts documented in the API contracts and evaluation and observability docs.

Production rule 3: A score without a bound protocol is not reproducible evidence.

4. Grade the path without collecting hidden chain-of-thought

Beyond the Final Answer argues that final answer matching misses efficiency, hallucination, and adaptability in tool-augmented trajectories. Its TRACE framework uses an evidence bank accumulated across observable steps to perform multi-dimensional, reference-free trajectory evaluation.

Production teams should not interpret “trajectory” as a demand to store private chain-of-thought. The audit surface can remain operational and typed:

• context manifest and omitted evidence;
• tool name, arguments, result, error, and duration;
• evidence references and freshness;
• plan revision and termination reason;
• policy and approval decisions;
• external state before and after a proposed mutation;
• retry, compensation, and rollback receipts;
• evaluator verdicts and their supporting observations;
• token, time, tool-call, and monetary budgets consumed.

That observable path answers the engineering questions that matter: which evidence was used, which boundary admitted the failure, whether a retry changed state, and whether the final result can be reproduced.

Production rule 4: Evaluate observable execution, not hidden reasoning.

5. Task completion can coexist with broken state

ACID-Bench targets state-changing tool agents with 52 deterministic fault-injection scenarios and a state-based evaluator that does not use an LLM judge.

Its central warning is brutally practical: an agent can report success while leaving duplicated, stale, partially committed, or insufficiently verified state.

No promoted final record carried runtime_failure, even though the underlying logs contained 20 failed attempts affecting 14 rows that later produced completed final records after retry. The evaluator separately recorded 525 integrity-violation labels across the 780 promoted records. Importantly, only 335 occurred on fault-exposed outcomes; 98 occurred on safe-handoff outcomes, and 92 on invalid or non-exposure outcomes. Not every label therefore demonstrates integrity failure after an exposed fault. The broader result survives that accounting nuance: completion status, interaction outcome, and state integrity are different dimensions and must be measured separately.

For an action-taking agent, every release evaluation should distinguish:

An output-only benchmark cannot answer those questions. The evaluator needs a deterministic view of the application state and stable effect identities.

Production rule 5: A successful output is not proof of correct external state.

6. Put high-precision policy at the action boundary

Reason Less, Verify More studies a failure in which a policy-permissive tool executes a well-formed call even when the domain policy forbids the state transition.

On its budget-agent setup, 78% of observed failures were silent wrong-state failures with no tool error. A suite of read-only deterministic pre-execution gates increased full-benchmark success from 29.6% to 42.0% on GPT-4o mini, with a similar lift on a disjoint replication set.

The qualification matters: almost all of the gain came from one gate with 100% precision over 161 fires, while another gate had only 5% precision. “Add rules” is not the finding. Measure each rule’s precision, firing slice, and side effects is the finding.

This maps cleanly to the Tool Gateway: the model proposes an action; a deterministic control reads the current state and rejects a known-invalid transition before any mutation. The gate does not make the model correct. It prevents one defined class of wrong state from becoming real.

Production rule 6: Enforce deterministic invariants at the mutation boundary.

7. Your evaluator is part of the system under test

Selecting LLM Judges for Agent Evaluation Pipelines shows why a judge cannot be selected by one headline accuracy number.

Across three evaluated agents, three web-agent benchmarks, and five judge conditions, task-success accuracy varied by only 1.6–4.4 percentage points between judges. Side-effect accuracy varied by 45–50 points across those same judges. The apparent tie on task scoring concealed radically different monitoring behavior.

The operational tradeoff was just as large. The best tested low-review-burden strategy reached 40.6–52.9% side-effect recall while sending 17.3–25.2% of trajectories to humans. GPT-4o screening reached 90.6–94.1% recall, but required reviewing 71.1–77.9% of trajectories. High recall did not remove the operating cost; it moved much of that cost into the review queue.

The study’s side-effect-positive slices were small—17 to 32 cases per evaluated agent—and the authors treat the judge profiles as directional rather than precise production estimates. That limitation reinforces the deployment lesson: re-audit the judge against the target agent’s own trajectory distribution and the decision the judge will support.

An evaluator should therefore have an operating profile, not merely an accuracy score:

evaluator_profile:
  target_dimensions:
    - task_success
    - side_effect
    - policy_violation
    - trajectory_quality
  target_agent_distribution: support_agent_v18
  confusion_matrix: per_dimension
  calibration_error: measured
  escalation_recall: measured
  human_review_rate: measured
  cost_per_1k_traces: measured
  latency_p95: measured
  judge_model_version: pinned

This is an illustrative evaluation artifact, not a new ContextOS runtime schema. A judge acceptable for offline benchmark scoring may be unacceptable for production triage. A judge that catches almost every side effect may simply convert model uncertainty into human workload.

Production rule 7: Evaluate the evaluator against its decision, target-agent distribution, and review budget.

8. Post-release evaluation is not optional

The workshop explicitly centers real-time post-market monitoring, model evolution, API risk, changing user contexts, and the absence of stable ground truth. That agenda recognizes a fact static benchmarks avoid: the product tuple changes after the test.

A production agent can drift because:

• the provider silently changes an inference route;
• a model version or quantization mode changes capability and refusal behavior;
• the evidence distribution shifts;
• policy is revised while memory retains an obsolete interpretation;
• tools add fields or change error semantics;
• users learn how to route around approvals;
• an evaluator’s judge model changes;
• a successful workflow increases its own authority or traffic mix.

Monitor at three cadences:

Per run

Validate schema, budgets, policy, state integrity, evidence sufficiency, approvals, and effect receipts before closing the run.

Per release

Replay the frozen regression set, run adversarial and high-authority slices, compare the exact candidate tuple against the current tuple, and require no critical policy regression.

Over time

Track outcome distributions, override and rescue rates, tool-denial patterns, evidence freshness, per-slice utility, cost per trusted outcome, and the distance between live and evaluation traffic.

An aggregate success curve can remain flat while a high-consequence slice degrades. Preserve slice identity and alert on distributional change.

Monitoring is not causal evaluation

Monitoring tells you that production changed. It does not, by itself, tell you whether your release caused the change.

The workshop’s Agent Arena industry keynote described a complementary approach: rankings derived from randomized, real-world agent traces using causal-inference methods. The program reports the scale and methodology as a keynote abstract, not a workshop-paper validation result, so the appropriate production inference is narrower: where risk and traffic permit, preserve a candidate/control comparison that can separate release effect from concurrent changes in users, tasks, tools, and providers.

That extends the lifecycle beyond “offline eval, canary, monitoring”:

Randomization is not permission to experiment with irreversible harm. The exposure unit, eligible risk slice, stop conditions, spillover assumptions, and rollback path must be declared before the test. High-risk or sparse workflows may require matched replay, stepped rollout, or other quasi-experimental designs instead of live randomization.

Production rule 8: Treat evaluation as a production control loop, and distinguish drift detection from causal attribution.

A production assurance scorecard

Use this as the release-review artifact for one candidate agent system.

Do not average stop-ship conditions into a composite. Ten utility gains must not cancel one newly permitted destructive action.

Assurance depth should scale with action risk

Not every agent needs the same release threshold. But a single A0–A4 ladder would also be too simple: network reach, delegated authority, reversibility, interaction mode, and data sensitivity are different risk dimensions. ContextOS evaluates the native ActionRisk vector conjunctively rather than treating those dimensions as one universal order.

Evaluation depth should scale with action risk and authority, not perceived model intelligence.

The 90-day roadmap

Days 1–15: make every score identifiable

• inventory current benchmarks and name the construct each one actually tests;
• bind model, harness, tools, data, environment, evaluator, and budgets into a protocol card;
• retain per-case results rather than only aggregate scores;
• measure human rescue and review time as first-class costs.

Days 16–30: verify trajectory and state

• capture observable tool, evidence, approval, and termination events;
• add deterministic pre/post state checks for every mutating workflow;
• inject timeouts, duplicate responses, stale reads, partial commits, and retry races;
• assign stable effect and idempotency identities.

Days 31–60: build release gates

• create high-authority, refusal, ambiguity, and recovery slices;
• compare the complete candidate tuple with the current release at equal budgets;
• audit every deterministic gate’s precision and false-deny rate;
• define rollback triggers before canary traffic begins.

Days 61–90: operate the evaluator and rollout

• shadow real traffic and measure evaluation-to-production drift;
• canary one bounded segment with sampled trace review;
• pre-register the eligible risk slice, exposure unit, hypothesis, stop conditions, and rollback path for candidate/control evaluation;
• randomize candidate/control exposure only where authority and safety constraints permit, and check for interference between units;
• track live slice performance, override rate, state-integrity violations, and cost per trusted outcome;
• promote new evaluation cases from incidents and operator corrections through review.

The future research agenda is equally concrete: standardized protocol cards, better construct validation, deterministic stateful environments, evaluator operating profiles and drift tests, causal attribution across single- and multi-agent systems, and assurance methods that remain useful when the model and traffic distribution both evolve.

What these papers do not prove

The workshop does not establish one universal assurance stack. Several studies use limited task sets, selected models, conditional protocols, synthetic faults, sparse positive slices, or workshop-scale evidence. The Agent Arena methodology appears in a keynote abstract rather than a workshop paper. A large percentage change in one benchmark—or a production-scale claim in a program description—is not a production guarantee.

The durable contribution is the measurement discipline:

• define what the test measures;
• disclose the protocol that produced the score;
• evaluate the observable path;
• verify the external state;
• keep deterministic controls at known action boundaries;
• profile the evaluator for the decision and review budget it supports;
• monitor the deployed system as a changing product tuple;
• use controlled comparisons, where safe, to distinguish release impact from background drift.

After the leaderboard, the evaluation question becomes operational:

Can this exact version of the model, harness, context, tools, policy, evaluator, and operating budget produce a correct, authorized, recoverable state transition under the conditions in which we will actually run it—and can we prove that the release caused the improvement?

If the evidence cannot answer that, the score is interesting research—not a release decision.

Research base

• KDD Workshop on Evaluation and Trustworthiness of Agentic AI 2026, program and accepted papers, August 9, 2026.
• Construct Validity Failures in Agentic AI Benchmarks.
• The Scaffold Effect in Coding Agents.
• Beyond Leaderboards: Protocol Cards for Trustworthy Coding-Agent Evaluation.
• Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents.
• ACID-Bench: Auditing Transactional Reliability in State-Changing Tool Agents.
• Reason Less, Verify More.
• Selecting LLM Judges for Agent Evaluation Pipelines: Accuracy Masking and Review-Burden Tradeoffs.

What to read next

• Your Agent Benchmark Is Measuring the Harness
• Scorecards Over Vibes
• Replay Harness in Code
• Evaluation and observability