Harness Engineering

Harness Engineering is the cross-plane discipline that ContextOS encodes. It is the practice of building the controlled execution environment around an AI agent — the code that decides what context the model sees, which tools it can call, what it must justify, what gets logged, what gets rolled back, and how all of that improves over time.

Traditional software engineering asks how do we write good code? Harness Engineering asks how do we create a governed system where agents can understand context, use tools, validate outputs, observe failures, recover safely, and continuously improve?

The model reasons. The harness governs execution.

Without a harness, agents behave like clever but inconsistent assistants. With a harness, agents become repeatable, observable, policy-governed execution units. The five planes of ContextOS — Intelligence, Context, Decision, Action, Trust — are the harness, decomposed into primitives that can be specified, versioned, evaluated, and replayed.

Definition

A harness is a stateful program that wraps a language model and determines, at every step, what context it sees, what tools it can call, and what is preserved from the result. Harness Engineering is the discipline of designing, evaluating, and continuously improving that program.

Concretely, a harness is the union of:

A ContextOS deployment is not “a model plus a few prompts” — it is a harness with named, typed, governed components. Every guarantee the platform makes is a property of the harness, not of the model.

Why it exists

Single-prompt and “model-plus-tools” architectures decay quickly under enterprise constraints. The bottleneck shifts from model intelligence to execution control:

• Can the system ensure that the answer or action is correct, safe, validated, observable, reversible, and compliant?
• Can a regression be detected before it ships, and rolled back when it does?
• Can the harness improve from its own failures without a human rewriting prompts?

Harness Engineering exists because three things are true at once: model behavior changes (drift, upgrades, new vendors); the world the model acts on changes (data, policies, suppliers); and the cost of an unbounded mistake is high. The only durable response is to externalize control from the model into a versioned, executable harness — and then to engineer that harness as deliberately as the rest of the platform.

The Stanford / MIT / KRAFTON Meta-Harness work (Lee et al., 2026) makes this concrete: changing the harness around a fixed model produced material benchmark gains across text classification, retrieval-augmented reasoning, and agentic coding. The harness, not just the weights, is an optimization target.

Implementation evidence. SecondBrain is an open-source local-first agent OS that exercises many of these patterns. Use its source map and runnable paths as evidence for specific claims; do not infer that every obligation or deployment profile on this page is implemented end-to-end.

How it works

Every ContextOS request passes through a harness in the same shape:

intent / task contract
        ↓
[Intelligence plane]   evidence + identity + memory recall
        ↓
[Context plane]        Context Pack → CompiledContext (eight stages)
        ↓
[Decision plane]       Planner → Critic.verify → Executor → Critic.score → Consolidate
        ↓
[Action plane]         Tool Gateway: risk-, authority-, and evidence-bound calls
        ↓
[Trust plane]          policy + evidence + audit + telemetry on every step
        ↓
DecisionRecord (evidence_refs, approvals, controls_active, trace_id)

The harness is the whole pipeline plus the control surfaces around it: the registry of tools, the policy bundles, the evaluator suite, the trace store, the replay harness, the review queue, and the improvement-loop primitives that consume their output.

Eight properties every harness must guarantee

Every harness ContextOS specifies must make every agent action:

These eight properties are not aspirations. Each maps to a typed primitive that must be present, versioned, and replayable for the harness to be considered production-grade.

The reference architecture

                  Human / Product / Architect
                            |
                            v
                    Intent / Task Contract
                            |
                            v
+-------------------------------------------------------+
|                 Harness Control Plane                 |
|-------------------------------------------------------|
| Context Builder | Policy Engine | Tool Registry       |
| Eval Engine     | Review Agents | Approval Gates      |
| Memory Layer    | Observability | Rollback Manager    |
| Improvement Loop                                      |
+-------------------------------------------------------+
                            |
                            v
                    Agent Execution Runtime
                            |
         +------------------+------------------+
         |                  |                  |
         v                  v                  v
   Read tools        Reversible writes    Sensitive actions
   GraphRAG          Memory updates       Payment / booking
   Lookup            Draft saves          / cancellation
                            |
                            v
                Validation + Telemetry + Feedback
                            |
                            v
                    DecisionRecord + Replay

The control plane is the long-lived, governed surface; the runtime is what executes per request; tools are the only path to external effect; and every step is observable, with recovery required where applicable and non-recoverability explicit.

The harness as a search target

A subtle but load-bearing claim: the harness itself is a target of optimization, not just an output of design. ContextOS treats every harness component — pack version, policy bundle, evaluator suite, planner skill, retrieval rule — as a versioned artifact that lives in change control alongside the code.

The Improvement Loop makes this concrete. Operator corrections, failed runs, and approval-gate overrides become candidate proposals under a locally versioned profile. Proposals run against deterministic fixtures and model-appropriate scorecards. Proposals that pass guardrails are promoted by an independent authority. Over time, the harness becomes the cumulative product of validated change—not a static prompt that ages.

This framing aligns with the Meta-Harness result: richer access to prior code, scores, and execution traces enabled an outer-loop search to discover stronger harnesses than its baselines. ContextOS does not require an autonomous outer-loop optimizer or ship a production experience store. It specifies the substrate requirements: version pins plus policy-scoped observable evidence, multi-objective evaluation, and a release-gated promotion path.

The five planes, viewed as a harness

Every plane is a slice of the harness with its own ownership, contract, and failure modes:

Cross-cutting primitives (RunContext, RunBudget, native ActionRisk, the v1 ApprovalMode projection, ContextPack, CompiledContext, DecisionRecord, and ToolEnvelope) are the typed seams between planes.

The release unit is the complete harness tuple

A production result is attributable only when the run identifies the complete release tuple that produced it:

runtime code + model route + prompt/skill versions + Context Pack
+ policy bundle + tool/adapter manifests + evaluator/rubric versions
+ environment/sandbox profile

ContextOS calls the immutable aggregate of those pins a release manifest. It is a documented protocol, not a new canonical core type: implementations assemble it from the version and hash fields already carried by the RunContext, CompiledContext, tool envelopes, trace bundle, evaluator verdicts, and DecisionRecord. A run with only a model name or Git SHA is not release-attributable.

The tuple is also the unit of rollback and comparison. A candidate is evaluated against a pinned baseline while varying only declared tuple members. Promotion records the candidate tuple, baseline tuple, datasets, graders, scorecard deltas, approver, and rollback target.

Proof-carrying execution packet

A terminal response or external effect must be reconstructable as a proof-carrying execution packet. The packet answers:

• who requested the work, on whose behalf, and for which declared intent;
• which release tuple, evidence, policy, authority, and budget were active;
• what decision was made, which tool request was authorized, and what external state changed;
• whether the postcondition was observed and which durable state was written;
• what failed or remained uncertain; and
• which rollback, retraction, compensation, or escalation path remains available.

This is a composition of canonical records, not a license to duplicate them. Identity claims come from RunContext; reasoning and citations from the DecisionRecord; effects from ToolCallEnvelope / ToolResultEnvelope; and evaluation, recovery, and lineage from trace and replay artifacts. No artifact, no claim is the release rule.

Repo-local harness layout

A ContextOS-aligned engineering repo should expose its harness on disk so that humans and agents can both navigate it:

repo/
  AGENTS.md                       # primary navigation for agents
  ARCHITECTURE.md
  SECURITY.md
  RELIABILITY.md
 
  docs/
    decision-records/             # ADRs, dated, append-only
    execution-plans/
      active/
      completed/
    runbooks/
    known-limitations/
 
  harness/
    packs/                        # Context Pack bundles, versioned
    policies/                     # JsonLogic / DSL bundles
    tools/                        # Tool manifests with schema + ownership
    evals/                        # golden sets, scenario rubrics
    fixtures/                     # synthetic scenarios for simulation
    validators/                   # interface tests run before full eval
    observability/                # trace schemas, span attribute conventions
    reviewers/                    # review-agent skills + rubrics
    skills/                       # planner / executor / critic skills
    feedback/                     # captured corrections + lineage

Two principles govern this layout:

1. Anything the agent must follow is visible, versioned, and machine-checkable — close to the code, not in a wiki.
2. The proposer can find every retained prior experience through a filesystem-shaped index — each retained run exposes permitted code/version refs, scores, and observable traces under classification, redaction, and retention policy. Hidden chain-of-thought and prohibited data are not required. This is one deployment profile for the substrate the Improvement Loop consumes.

A minimal AGENTS.md is a navigation file, not a prompt dump:

# AGENTS.md
 
You are operating inside this engineering repository.
 
Start here:
1. Read ARCHITECTURE.md for system boundaries.
2. Read SECURITY.md before touching PII, auth, payment, or sensitive data.
3. Read RELIABILITY.md before changing runtime flows.
4. Read docs/execution-plans/active/ for active work.
5. Run `make verify` before opening a PR.
6. Do not bypass policy gates.
7. Every change must include tests, telemetry, and rollback notes.
 
Harness layout:
- harness/packs/        — Context Pack bundles (versioned)
- harness/policies/     — policy bundles; JsonLogic; do not edit without ADR
- harness/tools/        — tool manifests; schema-checked in CI
- harness/evals/        — golden sets; replay before any pack change
- harness/reviewers/    — reviewer-agent skills; one per concern

The Meta-Harness paper finds that a short, well-written skill that constrains outputs and forbidden actions — while leaving the proposer free to inspect anything — is the strongest lever on harness search quality. AGENTS.md plays the same role for any coding agent operating in the repo.

Put learned behavior at the smallest durable layer

Mature coding-agent runtimes make a useful harness distinction concrete: durable repository guidance, reusable skills, deterministic hooks, and scheduled work are different control surfaces. ContextOS adopts the same separation without treating any one surface as a universal prompt.

The ordering matters. Do not schedule a prompt that still needs regular steering. First make the method explicit as a skill, prove it on replay, and move every mechanical rule into an enforcement surface. Cadence is configuration around a stable method, not a substitute for one.

Skills use progressive disclosure: the compiler should surface compact discovery metadata first, then load the full workflow, references, scripts, and assets only when the intent selects that skill. This keeps reusable expertise available without paying its full context cost on every run.

Corrections move down this table only after evidence shows that the narrower surface is insufficient. A one-off user preference should not become a global repository rule; a deterministic safety invariant should not remain prose in AGENTS.md; and a scheduled run must never become an unreviewed path for mutating the harness.

Feature-flagged harness rollout

A new harness component (a pack version, a policy change, a tool addition, a planner skill) is never released to all traffic at once. The canonical rollout stages:

Every stage carries a kill switch: a control-plane operation that stops affected execution or restores the validated prior version and routing. Managed profiles must declare an achievable fallback when provider code cannot be restored; see managed runtime conformance. Replay against the pinned snapshot must reproduce deterministic artifacts exactly and re-check model-dependent stages against declared properties or scorecards. That mixed comparison contract—not universal byte equality—is what makes rollback evidence meaningful.

Lifecycle security and skill release gates

Commit-time tests are only one slice of harness safety. A release-grade security gate covers six phases: configuration, extension loading, runtime execution, persistence, external action, and recovery. Results are sliced by phase, affected asset or effect, release tuple, entry surface, tool or extension, effective authority, delegated or resumed execution, and stable versus candidate release. Critical classes require zero successful attacks or unauthorized persistence events; outcome assertions use objective effect oracles rather than the agent’s own report.

Skills are executable release artifacts, not prompt snippets. Each skill is tested in paired conditions—same task and harness tuple with the skill available versus withheld—and must demonstrate positive Skill Lift on its target tasks. The gate also checks activation when relevant, silence when irrelevant, capability and identity use, trajectory evidence, security, latency, cost, neighboring-skill interference, and rollback. A skill that improves the final answer while producing a less safe or unauditable trajectory does not ship.

The invariant across both gates is simple: untrusted information may inform reasoning, but it cannot grant itself authority. Instructions, retrieved text, tool metadata, memory, or another agent’s message never widen identity, capability, policy, data, or effect boundaries.

Reviewer agents

Specialized review agents inspect every proposed change before it reaches a human approver. They are not a replacement for human judgment; they are a way to make humans focus on judgment instead of repetitive checks. The full taxonomy is documented in Reviewer Agents; the seven canonical roles are:

Reviewer output is a typed envelope (status, findings, severity, policy_id, recommendation) that lands in the same change-control queue as Improvement Loop proposals.

Managed runtime conformance

Hosting the execution loop does not transfer the application’s responsibility for authorized, accepted effects. Implementations using a managed harness must identify owners for session continuity, execution isolation, credential resolution, tool authorization, external-state reconciliation, outcome acceptance, evidence retention, and incident response.

Record the service configuration, exposed harness/model revisions, runtime dates, and environment profile in the release evidence. When a provider does not expose an immutable version, mark that limitation and maintain representative sentinel evaluations; do not invent an exact replay guarantee for an unpinned remote implementation. Durable session recovery and business-operation acceptance are separate states.

A production profile must demonstrate delegation scope preservation, revocation during pause, lost-response reconciliation, cancellation races, and retained evidence under interruption. The API acceptance cases define the required observations. Provider-managed credentials still require application checks that bind the requested effect to the caller, tenant, task, target, and approval.

The existing rollout stages and policy floors remain in force. A kill switch must name an achievable fallback. If the prior hosted harness cannot be restored, stop or constrain affected execution and use a validated alternative or accountable manual recovery; an application configuration rollback must not be represented as restoration of unavailable provider code.

This profile draws on managed session/harness/sandbox separation and caller-specific credential resolution. It adds deployment acceptance guidance without changing v1 serialization or claiming that this repository supplies a managed runtime.

Interfaces

Inputs

• Intent / task contract (from product surface or upstream agent)
• RunContext (user, agent, tenant, claims, budget, safety mode)
• Context Pack version pin
• Policy bundle version pin
• Tool registry version pin

Outputs

• CompiledContext (per request, by the Context Pack compiler)
• DecisionRecord (per turn, with evidence_refs, approvals, controls_active, trace_id)
• OTEL trace bundle (per run)
• Scorecard (per run, on the five evaluators)
• Improvement-loop proposals (per pattern, typed)

Versioned artifacts

• Context Pack bundle (packs/)
• Policy bundle (policies/)
• Tool manifest (tools/)
• Evaluator suite + golden sets (evals/)
• Planner / Executor / Critic skills (skills/)
• Reviewer skills (reviewers/)

Failure modes

• One giant prompt file. Becomes stale and ignored; the harness loses any property that was supposed to live in it.
• No tool schema. Agents misuse tools; the Tool Gateway cannot enforce side-effect classification or approval binding.
• No policy engine. Unsafe actions reach production; “the model knows the rules” is never a safety property.
• Only manual review. Does not scale; every additional intent multiplies the review queue linearly.
• Only happy-path evals. Real-world failures escape because the golden set captures success, not the long tail of edge cases.
• No telemetry. Failures cannot be debugged; there is nothing for the Improvement Loop to consume.
• No feature flags. Rollout becomes a binary cutover; a bad change hits 100% before it can be detected.
• No rollback. A small issue becomes an incident because the prior version is not pinned for replay.
• No shared semantic contract. Entity and relationship meanings drift; evidence joins become ambiguous and replay comparisons become unreliable.
• No ownership. The harness decays — no team owns the policy bundle, the eval set, or the tool registry, so they age silently.
• Compressed feedback to the optimizer. The Improvement Loop or any outer-loop search reads only summaries and scalar scores, losing the diagnostic signal needed to identify confounds (Meta-Harness Section 4.1 ablation).
• Search-set / test-set leakage. Goldens and scenarios used for tuning are also used for release gating; scorecards stop being honest.

Operational concerns

• Pack, policy, tool, and eval bundles are versioned independently; a release is a tuple of pinned versions, not a single SHA.
• Every long-running session has a kill switch wired to the rollback manager; the kill switch operates on the pinned-version tuple, not on individual primitives.
• Trace retention bands track data classification: traces touching regulated data follow the data’s retention rule, not the platform default.
• Run-budget headroom is measured per intent; intents trending toward budget exhaustion trigger an Autotune proposal before they fail under load.
• The reviewer-agent suite is itself a harness component; reviewer skills are versioned and golden-set-evaluated like any other.
• Search-set and test-set are kept disjoint by construction; the proposer (human or automated) never sees test-set scores during iteration.
• Lightweight interface validation runs before any full evaluation: import the candidate, instantiate, call its public methods on a tiny fixture, and require it to pass. Most malformed candidates die in seconds.

Evaluation metrics

Engineering metrics for the harness itself:

Product metrics carried by the harness’s DecisionRecord and replay log are documented in the Metrics Glossary.

Example

A condensed end-to-end harness flow for a sensitive action:

1. Intent: "support.refund.execute" arrives with RunContext.
2. Context plane: Context Pack v5.2.0 compiles a CompiledContext (8 stages).
   - intelligence_refs:  customer + order + prior corrections
   - policy_layer:       external/user-delegated/compensatable risk; approval required
   - tooling_layer:      adp_payments.issue_refund (destructive), adp_policy.eval (read_only)
   - evaluation_layer:   scorecard target { policy>=1.0, safety>=1.0 }
3. Decision plane: Planner proposes [eval, refund]; Critic.verify checks
   evidence coverage, ActionRisk, authority, and v1 projection before execution.
4. Trust plane: Policy Engine evaluates JsonLogic bundle; rule
   R_HIGH_VALUE_REQUIRES_APPROVAL requires approval gate GATE_FINANCE_APPROVAL.
5. Action plane: Tool Gateway blocks adp_payments.issue_refund pending approval token.
6. Reviewer agents (security, compliance, cost) emit findings; all clear.
7. Approver signs the gate; reversal token issued alongside the refund call.
8. DecisionRecord written: status=COMPLETED, evidence_refs=[...],
   approvals=[GATE_FINANCE_APPROVAL/op_22], reversal_token=rev_x7y, trace_id=...
9. Scorecard: policy=1.00, utility=0.94, safety=1.00, latency=1830ms, cost=$0.0091.
10. Replay reproduces deterministic artifacts exactly and re-checks the declared refund outcome properties without calling the live payment tool.

The same intent on pack v5.1.0 would produce a different DecisionRecord only where v5.2.0’s policy bundle changed; replay isolates the cause.

Common misconceptions

• The harness is not a prompt. The prompt is one input to the Context Pack compiler. Most harness behavior lives outside any prompt: policy, gates, evidence binding, evaluation, replay.
• Harness Engineering is not “prompt engineering done well.” Prompt engineering is one tactic inside one stage of the Context Pack compiler. Harness Engineering covers what the model sees, what it can do, what counts as done, and how the system improves.
• The model is not the optimization target. The harness is. Changing the harness around a fixed model can produce 6× swings on the same benchmark; weight changes are the smaller lever in most enterprise settings.
• “Reviewed by a human” is not a sufficient guarantee. Reviewer agents and replay suites catch patterns humans drift on; an independent authority approves, but the harness is what enforces.
• Improvement is not auto-pilot. Every Improvement Loop primitive emits proposals. Proposals run through the same release gate as packs and policies. Nothing auto-applies without an approver, except where policy explicitly permits it for a specific class.
• The harness is not a one-time build. It is a long-lived versioned artifact whose components age, decay, and must be re-baselined on a cadence. Treat it like any other production system.