Your Agent Followed the Rule—or Did It? Harness-IF and A2E Change Agent Evaluation

An agent can produce the right answer and still be unsafe to ship.

It can edit the correct file while ignoring the required approval. It can pass the unit test while using a forbidden command. It can solve the task but omit the evidence block, overwrite user work, reveal a secret in a tool call, or violate a project-local convention that never appears in the final response.

Outcome-only evaluation collapses all of those trajectories into “pass.” Harness engineering needs a stricter question:

For every applicable rule, did the agent exhibit the required behavior at the point where the rule mattered?

Two research releases from August 2026 make that question operational. Harness-IF measures rule compliance across the instruction surfaces of coding agents. A2E separates tasks, harness bindings, runners, traces, and lifecycle-aligned metrics across multiple agent frameworks. One studies where behavioral requirements enter; the other studies how execution is observed and compared.

Neither is a complete production evaluation system. Combined carefully, they provide the architecture for one.

The blind spot in ordinary agent evals

Suppose a coding task says:

• preserve unrelated local changes;
• run the targeted test before the full suite;
• do not use destructive Git commands;
• use an existing project abstraction;
• summarize skipped verification explicitly.

The final repository state may look correct even when four of five rules were violated. A final-state test sees only the artifact. A transcript reviewer may see the violations but has no stable rule inventory or applicability logic. A generic LLM judge may reward a polished explanation while missing the dangerous command inside a tool trace.

A production evaluation therefore needs at least four independent lenses:

The first lens is necessary. It is not sufficient.

What Harness-IF actually measured

Harness-IF starts from a useful decomposition: coding-agent instructions do not arrive through one prompt. They occupy several surfaces.

The paper constructed a 642-rule library, selected 60 realistic multi-turn coding items, and placed 302 rules across the configurable surfaces. It scored 256 placed rules. Twelve frontier models ran each item across three rounds, yielding 2,160 agent-item runs, 40,104 rule-level verdict rows, and 37,616 eligible binary verdicts.

The aggregate accuracy range was 72.1%–85.9%. That result alone is less interesting than three diagnostic findings.

1. Against-prior rules expose hidden fragility

Some rules reinforce what the model or harness would likely do anyway. Others ask it to behave differently. Harness-IF labels the latter against-prior and reports a separate AP-Acc metric.

Every tested model scored lower on against-prior rules. The gap between overall accuracy and AP-Acc was 3.6–7.4 percentage points, averaging 5.81 points. AP-Acc ranged from 66.1% to 78.6%.

This does not prove a causal model “prior” in the mechanistic sense. It is an evaluation label for rules designed to oppose expected baseline behavior. Its value is practical: a compliance suite made only of natural, agreeable rules can reward the behavior the system would have produced without reading them.

An effective regression set includes rules such as:

• use a project-specific command even though a common global command exists;
• produce a deliberately constrained response format;
• avoid an otherwise convenient tool;
• preserve an unusual but valid project convention;
• require an extra verification step that the default loop tends to skip.

The goal is not to make tasks arbitrary. It is to prove that the deployed instruction channel has causal influence over behavior.

2. Most failures are omissions

Harness-IF classifies rule failures as shortfalls or oversteps. Shortfalls—missing required behavior—accounted for 77.1% of failures. Their reported failure rate was 23.8%, compared with 20.8% for oversteps.

That changes how teams should design evaluators. Security reviews often focus on forbidden acts: did the agent run the disallowed command? The larger observed failure class was an absent act: did it forget to run the check, include the citation, preserve the artifact, or report the skipped step?

Negative-policy logs alone cannot see omissions. The trace must represent required milestones.

3. Output and workflow rules dominate the failure mass

Output-control rules had the study’s lowest family accuracy at 70.9%. Workflow rules scored 76.0%. Together, those two families produced 53.9% of all failures.

This is where a harness earns its name. Formatting constraints, tool sequence, verification requirements, and stopping behavior are not model trivia. They determine whether the result can enter a downstream system or a governed workflow.

Instruction precedence is not a universal ladder

In a pilot experiment, Harness-IF created synthetic conflicts between surfaces. Pooled results suggested a tendency for system prompts, project files, and user instructions to outrank tool and skill descriptions. But only six of nine older harness builds reproduced the exact pooled ordering.

Do not turn that finding into this rule:

system > project > user > tool > skill

The paper explicitly presents it as a pooled tendency, not a universal hierarchy. Harness defaults, prompt construction, instruction discovery, and runtime versions can change the result. User instructions in the main experiment were also all against-prior, so pooled surface accuracies are confounded and should not be compared as a clean surface leaderboard.

The production response is to make precedence a tested contract:

type InstructionSurface =
  | "harness-default"
  | "system"
  | "tool-description"
  | "skill-description"
  | "project-file"
  | "user"
 
type HarnessRule = {
  id: string
  owner: string
  surface: InstructionSurface
  authority: "platform" | "organization" | "repository" | "task"
  requirement: string
  appliesWhen: string
  conflictGroup?: string
  criticality: "blocker" | "major" | "minor"
  evaluator: "deterministic" | "human" | "model-judge" | "hybrid"
  evidencePaths: string[]
}

Each release should test representative conflicts: organization policy versus repository convention, repository convention versus user request, tool contract versus skill advice, and harness default versus explicit task rule. The expected winner must be declared before the run.

A2E supplies the execution-side architecture

Harness-IF gives us a rule and surface model. A2E contributes a protocol boundary for executing the same tasks through different harnesses.

Its Agent Task Protocol uses four core concepts:

1. TaskInput defines the benchmark or workload input.
2. AgentBinding connects that task to a harness-specific agent configuration.
3. AgentRunner executes it.
4. TaskTrace stores the observed trajectory.

The public A2E repository includes adapters for nine harnesses: Agno, AutoGen AgentChat, CrewAI, Google ADK, LangGraph, LlamaIndex, the OpenAI Agents SDK, smolagents, and the Claude Agent SDK. It uses OpenTelemetry and OpenInference spans, then organizes metrics under reasoning, action, final answer, and runtime quality.

The study ran 23 benchmarks with 5 tasks per harness–benchmark cell under a shared DeepSeek-V4-Pro configuration, producing 1,035 scored runs. Four sandbox-dependent benchmarks lacked the same full instrumentation; the remaining 19 produced 855 full trajectories and 19,665 score records.

The most useful result is qualitative. Single-turn benchmark cells were frequently tied across harnesses. Multi-turn tasks separated them sharply. Reported harness scores on tau-bench and gdpval spanned 0 to 0.60, while traject-bench spanned 0.20 to 1.0. On eight instrumented harnesses, correctness varied from 0.568 to 0.663 while token consumption varied by about 3.5×.

The harness matters most when state, tools, recovery, and multiple decisions accumulate.

Do not turn A2E into a framework leaderboard

A2E’s matrix is broad but shallow within each cell. Five tasks mean one outcome changes a score by 0.20. Benchmark selection changes aggregate order. Four sandbox benchmarks are treated separately. CrewAI lacks comparable token measurements, and instrumentation-dependent values such as invocation accounting should not be interpreted as behavior when the necessary trace was unavailable.

The authors say there is no universally best harness. That is not a weak conclusion. It is the basis for workload-specific evaluation.

Use the protocol pattern, not the aggregate rank:

Pair the runs by task and seed where possible. Preserve raw evidence. Compare the candidate against the current production build rather than against an abstract global winner.

Build a rule-level evaluation pipeline

The minimal viable pipeline has seven stages.

Stage 1: define the release tuple

Store every behavior-shaping version:

{
  "model": "provider/model@revision",
  "harness": "runtime@version",
  "systemPrompt": "sha256:...",
  "projectInstructions": "sha256:...",
  "skillCatalog": "sha256:...",
  "toolCatalog": "sha256:...",
  "policyBundle": "sha256:...",
  "memorySchema": "v4",
  "evaluatorBundle": "2026-08-13.1"
}

Without this tuple, a regression cannot be attributed. “The model changed” is often a guess.

Stage 2: compile the instruction manifest

At run start, enumerate the rules that should influence the agent. Resolve file scope and nesting. Record source, authority, priority, and content hash. Mark conflicts instead of silently merging them.

This manifest is distinct from the final compiled prompt. The prompt proves exposure; the manifest proves intent and provenance.

Stage 3: determine applicability before scoring

Not every rule applies to every task. A “run browser tests” rule should not fail a documentation-only task without a browser surface. A tool-specific safety rule is irrelevant when the tool was neither selected nor available.

Applicability must be explicit and auditable:

type RuleVerdict = {
  ruleId: string
  applicable: boolean
  applicabilityReason: string
  verdict: "pass" | "fail" | "unknown" | "not-applicable"
  failureMode?: "shortfall" | "overstep" | "conflict" | "insufficient-evidence"
  evidence: Array<{ spanId?: string; artifact?: string; excerpt?: string }>
  evaluatorVersion: string
  confidence?: number
}

Do not convert “unknown” into “pass.” Missing trace evidence is a runtime-quality defect.

Stage 4: prefer deterministic evaluators

Use executable checks for properties that have an objective representation:

• file exists and contains a required section;
• forbidden path was not modified;
• command family was or was not invoked;
• approval preceded side effect;
• tool arguments match a schema;
• tests ran after the last code modification;
• citations resolve to allowed domains;
• output parses against the contract.

Use model judges for semantic properties that resist deterministic checks, such as whether a summary accurately discloses a limitation. When a judge is used, store its prompt, model, votes, and evidence.

Harness-IF demonstrates why this distinction matters: 86.8% of its verdict rows involved a three-vote GPT-5.2 judge or hybrid evaluation. A judge-swap comparison on a small clean subset showed only 62.1% agreement and low kappa. The benchmark remains useful, but absolute accuracy is partly a property of the measurement instrument.

Stage 5: score shortfalls separately from oversteps

A single compliance rate hides the intervention.

This classification connects an eval failure to a harness change.

Stage 6: compare against a baseline and a prior-conflict subset

Maintain three views:

• overall rule pass rate for all applicable rules;
• against-prior pass rate for rules that oppose expected default behavior;
• critical-rule pass rate where one failure blocks release.

If feasible, run a controlled baseline without the injected rule. The strongest evidence that an instruction works is a paired behavioral change, not merely a passing run in which default behavior already agreed.

Stage 7: gate releases on distributions, not one average

This example is deliberately schematic; thresholds must follow risk and baseline data:

function releaseDecision(report: EvalReport): "promote" | "hold" {
  if (report.criticalRuleFailures > 0) return "hold"
  if (report.permissionAmbiguities > 0) return "hold"
  if (report.missingRequiredEvidence > 0) return "hold"
  if (report.againstPriorPassRate < report.minimumAgainstPriorRate) return "hold"
  if (report.outcomeLowerConfidenceBound < report.currentProductionFloor) return "hold"
  if (report.costPerAcceptedOutcome > report.approvedCostCeiling) return "hold"
  return "promote"
}

Do not let a high average compensate for a failed critical permission rule.

A six-week implementation roadmap

Week 1: inventory

• Collect system prompts, project instructions, skills, tool descriptions, hooks, and runtime defaults.
• Assign owners and stable rule IDs.
• Identify rules that currently exist in more than one surface.
• Record conflicts; do not blend competing conventions.

Week 2: trace schema

• Normalize messages, tool calls, approvals, artifacts, retries, compaction, and errors.
• Add model, harness, instruction, policy, and evaluator versions.
• Preserve correlation from a rule to the spans that can prove it.

Week 3: deterministic coverage

• Implement checks for tool ordering, file scope, required verification, approvals, structured output, and forbidden actions.
• Add explicit applicability conditions.
• Treat missing evidence as unknown or failure according to criticality.

Week 4: conflict and against-prior set

• Create paired rules that test real project conventions against common defaults.
• Move selected rules across surfaces while holding content constant.
• Add authority conflicts and define the expected winner.
• Run current production and candidate builds side by side.

Week 5: semantic judging and calibration

• Add human-reviewed examples for semantic rules.
• Measure judge disagreement and false-pass risk.
• Route low-confidence or critical semantic verdicts to a person.
• Version judge prompts as production code.

Week 6: release gate

• Establish outcome, compliance, recovery, and cost baselines.
• Define blocker rules and statistical floors.
• Canary the candidate on representative tasks.
• Store the decision packet and rehearsed rollback.

Common mistakes

“The system prompt contains the rule, so the agent received it”

Prompt presence is not behavioral compliance. Confirm exposure in the compiled context and evaluate the resulting trajectory.

“The final tests passed, so the workflow was safe”

Tests rarely prove permission order, secret handling, approval timing, or preservation of unrelated work.

“An LLM judge can score everything”

Judges are valuable instruments with calibration error. Objective events should remain objective checks.

“The highest framework average is the best framework”

Small cells, benchmark mix, tools, sandboxes, and instrumentation can reverse an aggregate order. Evaluate on the production task distribution.

“More instructions produce better compliance”

Duplicated or conflicting rules create precedence ambiguity. Minimize, scope, and test the instruction set.

“A rule failure means the model is bad”

The rule may have been undiscovered, truncated, placed in a weak surface, overridden, or impossible under the available tools. Diagnosis begins with the harness trace.

What these papers do not yet settle

Harness-IF is English- and coding-centered, uses 60 selected items, and evaluates 256 rule placements. Item selection favored difficulty and discriminativeness, which can create selection optimism. Most verdicts are judge-mediated, and raw provider traces are not part of the release described in the paper. The planned benchmark should be treated as an evaluation instrument under development, not a certification.

A2E samples only five tasks per harness–benchmark cell. Some benchmarks require sandbox infrastructure that changes comparability, and not every harness exposes identical telemetry. Its value is the open protocol and broad matrix, not a permanent rank order.

The next research step should combine both designs: run a versioned, surface-aware rule manifest through multiple harness candidates, capture normalized traces, and score outcome, compliance, recovery, and economics with calibrated uncertainty.

The new unit of quality

The model is no longer the smallest meaningful release unit. The harness is not either.

The useful unit is the evaluated runtime bundle: a model inside a specific harness build, receiving a versioned set of instructions, operating through a declared tool and policy surface, producing a durable trace, and judged by a versioned evaluator bundle.

That definition sounds heavier than “run the benchmark.” It is also much closer to the system that users trust with real work.

What to read next

• Harness Engineering, August 3–13, 2026: The Runtime Became the Product
• The Production Harness Roadmap: Lessons from Claude Code, Copilot Studio, and Thea
• Scorecards Over Vibes: How to Evaluate an Agent
• The Trace Is the Agent Debugger
• Agent Hijacking Security Evals