Harness Candidates Are Model Checkpoints: How to Improve Agents Without Silent Mutation

When an ML team trains a model, nobody says “we changed production weights because one example got better.”

They create a candidate. They evaluate it. They compare it to a baseline. They check regressions. They review tradeoffs. They stage rollout. They keep rollback.

Agent harnesses deserve the same discipline.

A harness candidate is a proposed version of the system around the model: Context Pack, retrieval settings, tool schema, planner template, Critic rubric, policy threshold, evaluator suite, rollout gate, or memory promotion rule. The model may be unchanged. The agent behavior can still change dramatically.

That is why great AI engineers treat harness candidates like model checkpoints. This is a release-management analogy, not the durable ExecutionCheckpoint contract used to resume an owned run. Reviewed on October 16, 2026: the candidate JSON, score strings, counts and local artifact names below are illustrative application profiles and synthetic examples, not additional exported schemas or measured deployment results.

What counts as a harness candidate

Almost any agent behavior change is a harness candidate.

Some of these are software diffs. Some are configuration. Some are data. The release system should not care. If it changes agent behavior, it is a candidate.

Candidate shape

A useful harness candidate has the same basic fields every time:

{
  "candidate_id": "hc_2026_05_12_refund_004",
  "baseline_tuple": {
    "pack": "ctxpack.support@5.2.0",
    "policy": "policy.returns@4.1.0",
    "tools": "tools.support@3.7.0",
    "evals": "evals.refund@2.4.0",
    "model_profile": "model.fast@2026-05-01"
  },
  "target": {
    "intent": "support.refund",
    "primary_metric": "operator_corrected_rate",
    "direction": "decrease"
  },
  "patches": [
    {
      "surface": "context.retrieval",
      "field": "supplier_policy.max_hops",
      "from": 2,
      "to": 3
    },
    {
      "surface": "decision.planner",
      "field": "verify_supplier_exception_before_denial",
      "from": false,
      "to": true
    }
  ],
  "guardrails": {
    "policy": "== 1.0",
    "safety": "== 1.0",
    "approval_bypass_rate": "== 0"
  }
}

A production registry must additionally bind immutable full candidate and baseline digests, release/configuration/model/data/profile definitions, exact evaluation inputs and trusted reviewer verdicts. A display version or an owner-name string is not enough.

The candidate names the baseline, the target, the patch, and the guardrails. Without those, the team cannot tell whether the change helped or merely changed behavior.

Candidate generation

Candidates come from four sources.

The source should be recorded. A human-authored candidate and an autotune candidate go through the same gate, but they deserve different review questions.

Candidate scoring

Freeze the candidate after search and before sealed release evaluation. Bound repeat submissions and feedback, separate related cases across splits, and refresh compromised held-out sets under an independent owner. A candidate should be scored before review, not after debate, using the same scorecard and replay machinery as other harness releases.

baseline:  release.support.refund@2026-05-09
candidate: hc_2026_05_12_refund_004
dataset:   support.refund/release_test@2026-05-12
 
policy:    1.000 -> 1.000
safety:    1.000 -> 1.000
utility:   0.901 -> 0.929
latency:   2180ms -> 2260ms
cost:      0.91c -> 0.97c
changed:   54 / 900
regressed: 0 policy, 0 safety, 3 utility review
verdict:   needs_human

This is a good outcome. It does not auto-ship. It gives reviewers a precise tradeoff: utility rose, latency and cost rose slightly, three utility regressions need inspection. These illustrative scores do not satisfy mandatory controls. Select exact deployment profiles and supply independently verified workflow/release/environment evidence of useful successful work, required denials and fault behavior. Missing, stale, synthetic or unverified runtime evidence remains insufficient under conformance.

Trace diff

The most useful part of candidate review is the trace diff.

Trace diff keeps review grounded. Instead of arguing over whether the prompt “seems better,” the team inspects the mechanism.

Release stages

Harness candidates should roll out like product changes through the feature-flagged harness rollout.

Shadow mode can expose live language and retrieval conditions while withholding destructive effects. Reads, inference, telemetry and exports still need current scope, an active boundary, credential limits, actual recipient/processor checks and retention controls. Candidate, reviewer, child, retry and resumed work all consume the original shared budget; the shadow label grants neither free capacity nor data access.

Rollback

Recovery readiness needs both eligible replay evidence and actual provider/effect semantics. Replaying recorded outputs does not reverse an external effect or guarantee identical future model bytes.

Every release must pin:

The rollback target is the prior tuple for future work. Test it using eligible replay inputs; respect deletion, retention and changed authority rather than resurrecting old data to recreate a run. Resolve uncertain writes through the original stable operation and verified provider recovery. Compensation is a separate governed effect. Neither re-pinning a release nor requesting cancellation proves a payment reversed, all copies deleted or every child stopped.

Candidate registry

Keep candidates in an append-only registry.

harness/candidates/
  hc_2026_05_12_refund_004/
    candidate.json
    patch.diff
    search-scorecard.json
    heldout-scorecard.json
    trace-diff-summary.json
    reviewer-verdicts.json
    rollout-record.json

Retain eligible rejection metadata under an explicit retention policy; append-only history is not permission to keep raw personal traces forever. Track derived copies and correction/deletion receipts, recording pending or externally unknown propagation honestly. Rejected candidates can prevent repeat mistakes. If a retrieval budget reduction failed because evidence coverage dropped, the next autotune run should know not to rediscover that path.

What should never be tunable

Not every field should be part of search.

The optimizer may propose a governance change for humans to review. It may not silently relax these fields because a metric improved.

The ContextOS version

ContextOS already has the right shape:

candidate harness tuple
  -> replay against search set
  -> freeze candidate and bound submissions
  -> independent held-out evaluation
  -> reviewer verdicts
  -> staged rollout
  -> DecisionRecords and scorecards
  -> Improvement Loop proposals

The Trust plane owns promotion. The other planes expose tunable surfaces and invariants. The candidate is the unit that moves between them.

Current authority and durable execution

A candidate may propose a change; it cannot authorize its own release. A personal or proactive worker needs a current owned mandate, tenant and audience. Child scopes intersect every ancestor and honor revocation. Bind entitled independent approval to the exact proposal, full action, evidence and immutable pins; a changed candidate requires fresh evaluation and approval. Policy-expression errors stop compilation with a structured error.

Protected atomic ownership, approval redemption and shared reservations, actual sandbox/credential/data enforcement, and generation fencing remain runtime responsibilities. Every wake or resume rechecks current authority, approval, boundary, preconditions and remaining original capacity. Preserve stable operation identity and uncertain reservations across crashes; a key alone does not make retry safe. The execution contract supplies those reference boundaries, not a deployed registry or rollout service.

The bar

A harness candidate is release-ready when:

That is the standard that keeps agent improvement from becoming silent mutation.

The harness can improve like a model. It should also be governed like one.

What to read next

• Autotune the Harness
• Replayable Environments
• Pack Rollout in Five Stages