Autotune the Harness: Baking the Improvement Loop into ContextOS

Autotune is a dangerous word because it sounds automatic.

Most teams hear it and picture a background job that rewrites prompts after a few bad answers. That is not autotune. That is an unreviewed deployment pipeline with a better name. If the system can change its own instructions, retrieval budgets, tool preferences, or approval thresholds without replay, scorecard guardrails, and a rollback path, it is not learning. It is mutating.

The useful version is stricter: autotune proposes bounded harness changes against a declared metric, proves them on replay, and sends the winner through the same release path as any other ContextOS artifact.

This post fills the gap between the earlier pieces:

The short version: ContextOS already has the substrate. DecisionRecords give evidence. Scorecards give metrics. Replay gives cheap historical testing. The Improvement Loop gives typed proposals. Rollout gives blast-radius control. Autotune is the loop that connects those pieces without bypassing governance.

The research signal

The useful research direction is not “make the prompt longer.” It is “treat the harness as the optimization target.”

The Meta-Harness work from Stanford / MIT / KRAFTON, “Meta-Harness: End-to-End Optimization of Model Harnesses”, makes the point directly: changing the surrounding harness while holding the model fixed can produce large performance swings. Their loop stores prior code, scores, and traces, then searches over harness variants. That maps cleanly to ContextOS: the thing being tuned is not just text. It can be a context pack, retrieval rule, planner skill, evaluator rubric, approval threshold, or tool-selection strategy.

OpenAI’s evaluation stack is moving in the same practical direction. Trace grading treats agent traces as the object being evaluated, not just final strings. Evals and graders turn datasets and rubric-bound checks into release signals. Prompt optimization is useful, but it is still one surface of a broader system.

The ContextOS distinction is this: prompt optimization is a special case of harness optimization. If the best candidate is a prompt tweak, fine. If the best candidate is top_k: 10 -> 6, a stricter evidence requirement, a different tool preference, or a higher approval threshold, the loop should be able to discover and propose that too.

The autotune contract

An autotune run must answer seven questions before it is allowed to start.

Without those answers, the optimizer has too much freedom. Too much freedom is how a cost optimizer learns to skip evidence, a latency optimizer learns to avoid tools, and a utility optimizer learns to route around approval gates.

The target should be boring and local:

{
  "target": {
    "intent": "support.refund",
    "metric": "economics_cents_per_decision",
    "direction": "decrease",
    "baseline_tuple": {
      "pack": "ctxpack.support@5.2.0",
      "policy": "policy.returns@4.1.0",
      "tools": "tools.support@3.7.0",
      "evals": "evals.refund@2.4.0"
    },
    "guardrails": {
      "policy": { "min": 1.0 },
      "safety": { "min": 1.0 },
      "utility_delta": { "min": -0.02 },
      "p95_latency_delta": { "max_ms": 250 },
      "approval_bypass_rate": { "max": 0 }
    }
  }
}

Do not tune “the agent.” Tune one intent. Do not tune “quality.” Tune a named metric. Do not let a metric stand alone. Bind it to guardrails.

The experience store

Autotune needs an experience store, not a dashboard screenshot. The store is the governed filesystem or database surface for eligible replay records. Collection and candidate storage already need current purpose, tenant/owner readers, actual processors and retention; not every prior transcript is eligible for reuse. Pin data and transformations, minimize sensitive payloads and track derived/cache/export/delegated copies through correction/deletion receipts. Pending or externally unknown copies remain unresolved.

At minimum, each run needs:

A repo-local version can be plain:

harness/
  experience/
    runs/
      2026-05-12/
        run_01HY9.../
          trace.json
          decision-record.json
          scorecard.json
          tool-transcripts.jsonl
          evidence-manifest.json
    goldens/
      support.refund/
        search.jsonl
        test.jsonl
    autotune/
      runs/
        at_2026_05_12_refund_cost/
          target.json
          candidates.jsonl
          replay-results.jsonl
          selected-proposal.json

That layout is intentionally not fancy. The important property is that autotune can read prior experience without asking a human to explain what happened.

The search space

The optimizer can propose changes only within an owned declared search space. It cannot weaken mandatory control definitions, required tests, policy prohibitions or an already admitted grant/approval. A threshold proposal for a future policy requires the governing owner and a separately authorized release; merely listing a field as tunable cannot permit it.

export type TunableSurface =
  | {
      layer: "retrieval"
      field: "top_k" | "max_hops" | "source_priority"
      bounds: { min?: number; max?: number; allowed?: string[] }
    }
  | {
      layer: "context_budget"
      field: "evidence_tokens" | "memory_tokens" | "tool_tokens"
      bounds: { min: number; max: number; step: number }
    }
  | {
      layer: "planner"
      field: "tool_preference" | "decompose_before_tool"
      bounds: { allowed: string[] }
    }
  | {
      layer: "prompt"
      field: "instruction_block"
      bounds: { max_added_tokens: number; forbidden_terms: string[] }
    }
  | {
      layer: "approval_gate"
      field: "threshold"
      bounds: { min: number; max: number; step: number }
    }

The surface belongs in the Context Pack or harness manifest. If a field is not declared tunable, autotune does not touch it. This keeps the optimizer from discovering “creative” shortcuts such as disabling an evaluator, lowering a safety threshold, or removing expensive evidence.

Start with three surfaces:

Prompt tuning is fourth, not first. Prompts are easy to change and hard to reason about. Retrieval and budget candidates are often easier to replay, diff, and rollback.

The loop in code

The autotune runner is a pipeline. It should be deterministic enough that another engineer can reproduce why a proposal won.

Non-executable control flow; implementation and reporting validation required:
1. Admit the search owner, exact profiles, data scope and shared capacity.
2. Validate the allowed patch space and pinned search/evaluator inputs.
3. Run candidates only on the search set; reject missing/invalid required results.
4. Choose a candidate using declared deterministic tie-breaks and hard guardrails.
5. Freeze that candidate, artifacts and proposal digest before release testing.
6. A separate release owner evaluates the frozen candidate on the held-out set.
7. Return pass/fail/insufficient evidence; do not expose per-case feedback to search.
8. Require current entitled review/approval and deployment evidence before rollout.

A few details matter more than the code volume.

The held-out set is not used for candidate ranking or iteration. The previous loop tested every candidate on it, which leaked the release set into selection; the flow above separates those owners. Repeated aggregate pass/fail feedback can also leak information: bound submissions and refresh the test set under an independent release policy. The proposer can search against the search set. The release gate checks the held-out set. If you let autotune repeatedly see the held-out set, you have converted it into training data.

The selection function should be Pareto-aware. Do not collapse Policy, Safety, Utility, Latency, and Economics into one weighted score unless you want arguments about weights to become your release process. Hard guardrails block. Among surviving candidates, choose the one that improves the target with the smallest blast radius.

The result is a proposal, not a patch applied to production.

Search, reviewers, retries and child experiments consume original shared reserved capacity. A scheduled wake or resume requires current owner/ancestor authority, exact approval when required, active execution/credential/data controls, external preconditions and current fenced ownership. Keep experimental provider effects disabled or explicitly governed. Rolling back a harness version does not undo previous external effects; unknown results need stable operation identities and provider-specific reconciliation, and retain uncertain reservations.

An illustrative application proposal

Autotune output should be typed, reviewable, replay-backed and bound to evidence. The local JSON and TunableSurface type in this article illustrate an application design; they are not extra canonical Context Pack fields or a schema consumed by the reference compiler. Map them explicitly to the published contracts. Example scores are synthetic, not measured results.

{
  "tuning_proposal_id": "at_2026_05_12_refund_cost_01",
  "status": "proposed",
  "target": {
    "intent": "support.refund",
    "metric": "economics_cents_per_decision",
    "direction": "decrease"
  },
  "candidate_change": {
    "layer": "retrieval",
    "target_pack": "ctxpack.support@5.2.0",
    "patch": {
      "evidence.order_history.top_k": { "from": 10, "to": 6 },
      "evidence.supplier_policy.max_hops": { "from": 3, "to": 2 }
    }
  },
  "replay": {
    "search_set": "goldens/support.refund/search@2026-05-12",
    "heldout_test_set": "goldens/support.refund/test@2026-05-12",
    "evaluation_run_id": "eval_2026_05_12_91f",
    "baseline_tuple": "release.support.refund@2026-05-09"
  },
  "observed_delta": {
    "economics_cents_per_decision": -0.21,
    "utility": -0.006,
    "policy": 0,
    "safety": 0,
    "p95_latency_ms": -180
  },
  "guardrails": {
    "passed": true,
    "regression_runs": []
  },
  "release_path": {
    "requires_reviewers": ["reliability", "policy", "domain_owner"],
    "rollout_stage": "0%_shadow",
    "rollback_target": "ctxpack.support@5.2.0"
  }
}

Notice what is not present: “autotune changed production.” The proposal still needs review, approval, and staged rollout.

What candidates should be allowed to win

Autotune should prefer candidates that are small, explainable, and reversible.

The optimizer is not rewarded for cleverness. It is rewarded for producing candidates a reviewer can understand in five minutes.

The operating cadence

Autotune works best as a routine, not a special project.

Daily:

1. Build intent-level slices from the last 24 hours of traces.
2. Detect targets where one metric moved outside its band.
3. Queue at most one autotune run per intent and risk class.
4. Emit proposals only when held-out replay passes guardrails.

Weekly:

1. Review proposal adoption rate.
2. Add corrected production failures to the golden search set.
3. Refresh the held-out set without letting the proposer inspect it.
4. Retire tunable surfaces that keep producing rejected proposals.
5. Promote approved candidates through the five-stage rollout path.

Quarterly:

1. Rebaseline evaluator rubrics.
2. Re-split search and test sets.
3. Audit whether guardrails still match business risk.
4. Remove stale candidate generators.

This cadence keeps autotune from becoming a novelty button. It becomes part of the harness maintenance cycle.

Where autotune lives in ContextOS

Autotune is not a separate service with special power. It is one producer inside the Improvement Loop.

traces + scorecards + corrections
        ->
InsightSynthesizer / StrategyCompiler / Autotune
        ->
typed proposals
        ->
replay gate
        ->
human review
        ->
pack / policy / tool / evaluator release
        ->
staged rollout
        ->
new traces + scorecards

That placement matters. The loop can improve retrieval, prompts, planner choices, and budgets, but every candidate still passes through Trust-plane controls. The optimizer does not get an escape hatch from the same harness it is trying to improve.

Failure modes

The common failures are predictable.

The key is to treat each failure as a harness property, not a model behavior. The model did not overfit the eval by itself. The harness gave the optimizer an overfit path.

The first implementation

The smallest useful implementation is one metric, one intent, and one tunable surface.

Pick an intent with volume and low external risk, such as a read-mostly support triage flow. Pick a metric that matters but is not safety-critical, such as economics_cents_per_decision or p95_latency_ms. Declare exactly one tunable surface, such as retrieval top_k for one evidence bucket. Split the existing golden traces into search and held-out test sets. Run three candidates. Emit one proposal. Do not ship it automatically.

A possible first-slice sequence, with duration dependent on existing controls:

If that loop works, add surfaces slowly. The goal is not to make the optimizer powerful. The goal is to make improvement routine.

Definition of done

These are initial design checks, not a production certificate. Add current independently verified evidence for the actual workflow, implementation, release and environment, useful positive work and required denials under the exact control profiles and conformance requirements:

That is the practical meaning of “improvement loop baked into the system.” Typed proposals, review and replay make changes inspectable; they do not prove every downstream effect can be reversed.

Autotune should make the harness better without making the organization less certain about what changed. In ContextOS, that is the line: search is welcome, silent mutation is not.

What to read next

• Harness Candidates Are Model Checkpoints
• Replayable Environments
• From Correction to StrategyRule