Harness Engineering

Harness engineering designs and evaluates the system that lets a model do useful work: its context, tools, execution environment, durable state, feedback, and controls. ContextOS contributes the portable decision-control and evidence contract across those boundaries. The adoption guide applies it to one workflow in an existing runtime.

The engineering question is concrete: what must this system observe, what may it change, how do we establish completion, and what happens when it fails? Reliable behavior belongs to the model, harness, and environment working together; changing any of them can change the result.

The model reasons. The harness governs execution.

The five planes of ContextOS — Intelligence, Context, Decision, Action, Trust — assign logical ownership. They do not prescribe five services or guarantee correctness. A small implementation can combine components, while a managed provider may own the loop or sandbox. Conformance depends on the controls and evidence that the implementing workflow actually demonstrates.

Definition

A harness is a stateful program that wraps a language model and determines, at every step, what context it sees, what tools it can call, and what is preserved from the result. Harness Engineering is the discipline of designing, evaluating, and continuously improving that program.

Concretely, a harness is the union of:

A ContextOS-conforming workflow makes these responsibilities explicit and testable. Typed components establish inspectable interfaces; enforcement and outcome evidence establish the deployment claim. A prompt, schema, or favorable evaluator score alone cannot establish that an effect was authorized or correct.

The compatible Execution Contract binds exact Control Profiles, actual enabled features and immutable releases to the authenticated run. The reference checks authority/approval, active-boundary observations, data/memory policy, durable operations, fenced resume and aggregate budgets using normalized facts. A real deployment still needs unavoidable enforcement adapters, transactional stores and current scoped conformance evidence.

Why it exists

Model capability, harness design, and environment quality can each limit a workflow. An implementation needs evidence to locate its bottleneck:

• Does the system complete the intended task within declared authority, with observable outcome evidence and an applicable recovery path?
• Can a regression be detected before it ships, and rolled back when it does?
• Can the harness improve from its own failures without a human rewriting prompts?

Harness Engineering exists because three things are true at once: model behavior changes (drift, upgrades, new vendors); the world the model acts on changes (data, policies, suppliers); and the cost of an unbounded mistake is high. The only durable response is to externalize control from the model into a versioned, executable harness — and then to engineer that harness as deliberately as the rest of the platform.

The Meta-Harness work (Lee et al., 2026) motivates treating harness code as an optimization target. Its task-specific results support experiments with context and feedback; they do not establish that harness changes always dominate model improvements or transfer to a new domain.

Implementation evidence. SecondBrain is an open-source local-first agent OS that exercises many of these patterns. Use its source map and runnable paths as evidence for specific claims; do not infer that every obligation or deployment profile on this page is implemented end-to-end.

How it works

Start with the smallest enforceable workflow

Define accepted outcomes, required evidence, forbidden effects, and budgets before selecting an execution pattern. Start with the simplest workflow that can satisfy them. Use typed context, a policy-bound tool path, a DecisionRecord, and failure cases before adding graph retrieval, durable memory, specialist reviewers, or delegation. The canonical loop below names responsibilities; it does not require a separate model invocation or deployed agent for every role.

The environment is part of the harness design. Give the agent discoverable, current instructions; clear tool schemas; reproducible setup; inspectable application state; and feedback that can establish whether work passed. OpenAI’s harness engineering report emphasizes repository legibility and mechanical feedback. Apply that lesson to business workflows through explicit state and outcome checks, without assuming coding results predict their performance.

Separate controls from compensating scaffolding

Authorization, credential isolation, tenant boundaries, evidence requirements, budget ceilings, and effect reconciliation are control obligations. Planning prompts, forced decompositions, context resets, extra reviewers, and skills are candidate methods for meeting the task. Re-evaluate the methods when the model or workload changes; removing one must not remove a control obligation.

Anthropic’s managed-agent design describes a context-reset workaround becoming unnecessary with a newer model. LangChain’s Deep Agents v0.7 report describes trimming its base harness and evaluating the changes across models. These are evidence for measured simplification, not a rule to remove planning from every agent.

Compare a candidate with the existing model/harness/environment configuration on held-out tasks, including failures and prohibited effects. Keep latency, full cost, intervention effort, and unresolved outcomes visible. Use the evaluation protocol to distinguish fixed-model experiments from whole-system changes.

Every ContextOS request passes through a harness in the same shape:

intent / task contract
        ↓
[Intelligence plane]   evidence + identity + memory recall
        ↓
[Context plane]        Context Pack → CompiledContext (eight stages)
        ↓
[Decision plane]       Planner → Critic.verify → Executor → Critic.score → Consolidate
        ↓
[Action plane]         Tool Gateway: risk-, authority-, and evidence-bound calls
        ↓
[Trust plane]          policy + evidence + audit + telemetry on every step
        ↓
DecisionRecord (evidence_refs, approvals, controls_active, trace_id)

The harness is the whole pipeline plus the control surfaces around it: the registry of tools, the policy bundles, the evaluator suite, the trace store, the replay harness, the review queue, and the improvement-loop primitives that consume their output.

Eight properties a conforming harness must demonstrate

Every harness ContextOS specifies must make every agent action:

These properties map to implementation obligations and versioned evidence. Their presence in an architecture description does not establish that a deployed effect path enforces them.

The reference architecture

                  Human / Product / Architect
                            |
                            v
                    Intent / Task Contract
                            |
                            v
+-------------------------------------------------------+
|                 Harness Control Plane                 |
|-------------------------------------------------------|
| Context Builder | Policy Engine | Tool Registry       |
| Eval Engine     | Review Agents | Approval Gates      |
| Memory Layer    | Observability | Rollback Manager    |
| Improvement Loop                                      |
+-------------------------------------------------------+
                            |
                            v
                    Agent Execution Runtime
                            |
         +------------------+------------------+
         |                  |                  |
         v                  v                  v
   Read tools        Reversible writes    Sensitive actions
   GraphRAG          Memory updates       Payment / booking
   Lookup            Draft saves          / cancellation
                            |
                            v
                Validation + Telemetry + Feedback
                            |
                            v
                    DecisionRecord + Replay

The control plane is the long-lived, governed surface; the runtime is what executes per request; tools are the only path to external effect; and every step is observable, with recovery required where applicable and non-recoverability explicit.

The harness as a search target

A subtle but load-bearing claim: the harness itself is a target of optimization, not just an output of design. ContextOS treats every harness component — pack version, policy bundle, evaluator suite, planner skill, retrieval rule — as a versioned artifact that lives in change control alongside the code.

The Improvement Loop makes this concrete. Operator corrections, failed runs, and approval-gate overrides become candidate proposals under a locally versioned profile. Proposals run against deterministic fixtures and model-appropriate scorecards. Proposals that pass guardrails are promoted by an independent authority. Over time, the harness becomes the cumulative product of validated change—not a static prompt that ages.

This framing aligns with the Meta-Harness result: richer access to prior code, scores, and execution traces enabled an outer-loop search to discover stronger harnesses than its baselines. ContextOS does not require an autonomous outer-loop optimizer or ship a production experience store. It specifies the substrate requirements: version pins plus policy-scoped observable evidence, multi-objective evaluation, and a release-gated promotion path.

The five planes, viewed as a harness

Every plane is a slice of the harness with its own ownership, contract, and failure modes:

Cross-cutting primitives (RunContext, RunBudget, native ActionRisk, the v1 ApprovalMode projection, ContextPack, CompiledContext, DecisionRecord, and ToolEnvelope) are the typed seams between planes.

The release unit is the complete harness tuple

A production result is attributable only when the run identifies the complete release tuple that produced it:

runtime code + model route + prompt/skill versions + Context Pack
+ policy bundle + tool/adapter manifests + evaluator/rubric versions
+ environment/sandbox profile

ContextOS calls the immutable aggregate of those pins a release manifest. It is a documented protocol, not a new canonical core type: implementations assemble it from the version and hash fields already carried by the RunContext, CompiledContext, tool envelopes, trace bundle, evaluator verdicts, and DecisionRecord. A run with only a model name or Git SHA is not release-attributable.

ExecutionContractEnvelope.release_refs now supplies the additive run-bound reference list; ActionBinding, approval and checkpoint pins carry it through protected effects and recovery. The full release manifest payload remains an explicitly versioned deployment artifact. Providers that expose no immutable model/harness revision must retain that limitation rather than inventing a pin.

The tuple is also the unit of rollback and comparison. A candidate is evaluated against a pinned baseline while varying only declared tuple members. Promotion records the candidate tuple, baseline tuple, datasets, graders, scorecard deltas, approver, and rollback target.

Proof-carrying execution packet

A terminal response or external effect must be reconstructable as a proof-carrying execution packet. The packet answers:

• who requested the work, on whose behalf, and for which declared intent;
• which release tuple, evidence, policy, authority, and budget were active;
• what decision was made, which tool request was authorized, and what external state changed;
• whether the postcondition was observed and which durable state was written;
• what failed or remained uncertain; and
• which rollback, retraction, compensation, or escalation path remains available.

This is a composition of canonical records, not a license to duplicate them. Identity claims come from RunContext; reasoning and citations from the DecisionRecord; effects from ToolCallEnvelope / ToolResultEnvelope; and evaluation, recovery, and lineage from trace and replay artifacts. No artifact, no claim is the release rule.

Profile-aware packets also retain complete action digests, current grant/approval verification, boundary/data/memory observations, operation/checkpoint/budget ledger refs and scoped conformance results. They preserve unknown/partial effects, pending deletion and unverified runtime properties. A hash proves byte binding only; a checkpoint or stop acknowledgment cannot prove an in-flight remote effect never happened.

Repo-local harness layout

A ContextOS-aligned engineering repo should expose its harness on disk so that humans and agents can both navigate it:

repo/
  AGENTS.md                       # primary navigation for agents
  ARCHITECTURE.md
  SECURITY.md
  RELIABILITY.md
 
  docs/
    decision-records/             # ADRs, dated, append-only
    execution-plans/
      active/
      completed/
    runbooks/
    known-limitations/
 
  harness/
    packs/                        # Context Pack bundles, versioned
    policies/                     # JsonLogic / DSL bundles
    tools/                        # Tool manifests with schema + ownership
    evals/                        # golden sets, scenario rubrics
    fixtures/                     # synthetic scenarios for simulation
    validators/                   # interface tests run before full eval
    observability/                # trace schemas, span attribute conventions
    reviewers/                    # review-agent skills + rubrics
    skills/                       # planner / executor / critic skills
    feedback/                     # captured corrections + lineage

Two principles govern this layout:

1. Anything the agent must follow is visible, versioned, and machine-checkable — close to the code, not in a wiki.
2. The proposer can find every retained prior experience through a filesystem-shaped index — each retained run exposes permitted code/version refs, scores, and observable traces under classification, redaction, and retention policy. Hidden chain-of-thought and prohibited data are not required. This is one deployment profile for the substrate the Improvement Loop consumes.

A minimal AGENTS.md is a navigation file, not a prompt dump:

# AGENTS.md
 
You are operating inside this engineering repository.
 
Start here:
1. Read ARCHITECTURE.md for system boundaries.
2. Read SECURITY.md before touching PII, auth, payment, or sensitive data.
3. Read RELIABILITY.md before changing runtime flows.
4. Read docs/execution-plans/active/ for active work.
5. Run `make verify` before opening a PR.
6. Do not bypass policy gates.
7. Every change must include tests, telemetry, and rollback notes.
 
Harness layout:
- harness/packs/        — Context Pack bundles (versioned)
- harness/policies/     — policy bundles; JsonLogic; do not edit without ADR
- harness/tools/        — tool manifests; schema-checked in CI
- harness/evals/        — golden sets; replay before any pack change
- harness/reviewers/    — reviewer-agent skills; one per concern

The Meta-Harness paper reports benefits from concise output/action constraints and inspectable prior artifacts in its tested search setting. In a deployment, inspection remains inside authenticated task, tenant, classification and retention permissions. AGENTS.md guides repository work; it cannot grant access to unrelated secrets or override an execution boundary.

Put learned behavior at the smallest durable layer

Mature coding-agent runtimes make a useful harness distinction concrete: durable repository guidance, reusable skills, deterministic hooks, and scheduled work are different control surfaces. ContextOS adopts the same separation without treating any one surface as a universal prompt.

The ordering matters. Do not schedule a prompt that still needs regular steering. First make the method explicit as a skill, prove it on replay, and move every mechanical rule into an enforcement surface. Cadence is configuration around a stable method, not a substitute for one.

Skills use progressive disclosure: the compiler should surface compact discovery metadata first, then load the full workflow, references, scripts, and assets only when the intent selects that skill. This keeps reusable expertise available without paying its full context cost on every run.

Corrections move down this table only after evidence shows that the narrower surface is insufficient. A one-off user preference should not become a global repository rule; a deterministic safety invariant should not remain prose in AGENTS.md; and a scheduled run must never become an unreviewed path for mutating the harness.

Feature-flagged harness rollout

A new harness component (a pack version, a policy change, a tool addition, a planner skill) is never released to all traffic at once. The canonical rollout stages:

Every stage carries a kill switch: a control-plane operation that stops affected execution or restores the validated prior version and routing. Managed profiles must declare an achievable fallback when provider code cannot be restored; see managed runtime conformance. Replay against the pinned snapshot must reproduce deterministic artifacts exactly and re-check model-dependent stages against declared properties or scorecards. That mixed comparison contract—not universal byte equality—is what makes rollback evidence meaningful.

Lifecycle security and skill release gates

Commit-time tests are only one slice of harness safety. A release-grade security gate covers six phases: configuration, extension loading, runtime execution, persistence, external action, and recovery. Results are sliced by phase, affected asset or effect, release tuple, entry surface, tool or extension, effective authority, delegated or resumed execution, and stable versus candidate release. Critical classes require zero successful attacks or unauthorized persistence events; outcome assertions use objective effect oracles rather than the agent’s own report.

Skills are executable release artifacts, not prompt snippets. Each skill is tested in paired conditions—same task and harness tuple with the skill available versus withheld—and must demonstrate positive Skill Lift on its target tasks. The gate also checks activation when relevant, silence when irrelevant, capability and identity use, trajectory evidence, security, latency, cost, neighboring-skill interference, and rollback. A skill that improves the final answer while producing a less safe or unauditable trajectory does not ship.

The invariant across both gates is simple: untrusted information may inform reasoning, but it cannot grant itself authority. Instructions, retrieved text, tool metadata, memory, or another agent’s message never widen identity, capability, policy, data, or effect boundaries.

Reviewer agents

Specialized review agents inspect every proposed change before it reaches a human approver. They are not a replacement for human judgment; they are a way to make humans focus on judgment instead of repetitive checks. The full taxonomy is documented in Reviewer Agents; the seven canonical roles are:

Reviewer output is a typed envelope (status, findings, severity, policy_id, recommendation) that lands in the same change-control queue as Improvement Loop proposals.

Managed runtime conformance

Hosting the execution loop does not transfer the application’s responsibility for authorized, accepted effects. Implementations using a managed harness must identify owners for session continuity, execution isolation, credential resolution, tool authorization, external-state reconciliation, outcome acceptance, evidence retention, and incident response.

Record the service configuration, exposed harness/model revisions, runtime dates, and environment profile in the release evidence. When a provider does not expose an immutable version, mark that limitation and maintain representative sentinel evaluations; do not invent an exact replay guarantee for an unpinned remote implementation. Durable session recovery and business-operation acceptance are separate states.

A production profile must demonstrate delegation scope preservation, revocation during pause, lost-response reconciliation, cancellation races, and retained evidence under interruption. The API acceptance cases define the required observations. Provider-managed credentials still require application checks that bind the requested effect to the caller, tenant, task, target, and approval.

Personal/proactive deployments additionally bind the authenticated owner and trigger/channel/device mandate, support inspection/pause/revoke/stop, and separate personal/work namespaces. Enterprise deployments own tenant/principal lifecycle, required independent approval and protected policy/release administration. High-assurance claims need authorized fault/recovery evidence and a workflow-specific domain assurance case. None follows from a provider’s sandbox or session feature name alone.

The existing rollout stages and policy floors remain in force. A kill switch must name an achievable fallback. If the prior hosted harness cannot be restored, stop or constrain affected execution and use a validated alternative or accountable manual recovery; an application configuration rollback must not be represented as restoration of unavailable provider code.

This profile draws on managed session/harness/sandbox separation and caller-specific credential resolution. It adds deployment acceptance guidance without changing v1 serialization or claiming that this repository supplies a managed runtime.

Interfaces

Inputs

• Intent / task contract (from product surface or upstream agent)
• RunContext (user, agent, tenant, claims, budget, safety mode)
• Context Pack version pin
• Policy bundle version pin
• Tool registry version pin

Outputs

• CompiledContext (per request, by the Context Pack compiler)
• DecisionRecord (per turn, with evidence_refs, approvals, controls_active, trace_id)
• OTEL trace bundle (per run)
• Scorecard (per run, on the five evaluators)
• Improvement-loop proposals (per pattern, typed)

Versioned artifacts

• Context Pack bundle (packs/)
• Policy bundle (policies/)
• Tool manifest (tools/)
• Evaluator suite + golden sets (evals/)
• Planner / Executor / Critic skills (skills/)
• Reviewer skills (reviewers/)

Failure modes

• One giant prompt file. Becomes stale and ignored; the harness loses any property that was supposed to live in it.
• No tool schema. Agents misuse tools; the Tool Gateway cannot enforce side-effect classification or approval binding.
• No policy engine. Unsafe actions reach production; “the model knows the rules” is never a safety property.
• Only manual review. Does not scale; every additional intent multiplies the review queue linearly.
• Only happy-path evals. Real-world failures escape because the golden set captures success, not the long tail of edge cases.
• No telemetry. Failures cannot be debugged; there is nothing for the Improvement Loop to consume.
• No feature flags. Rollout becomes a binary cutover; a bad change hits 100% before it can be detected.
• No rollback. A small issue becomes an incident because the prior version is not pinned for replay.
• No shared semantic contract. Entity and relationship meanings drift; evidence joins become ambiguous and replay comparisons become unreliable.
• No ownership. The harness decays — no team owns the policy bundle, the eval set, or the tool registry, so they age silently.
• Compressed feedback to the optimizer. The Improvement Loop or any outer-loop search reads only summaries and scalar scores, losing the diagnostic signal needed to identify confounds (Meta-Harness Section 4.1 ablation).
• Search-set / test-set leakage. Goldens and scenarios used for tuning are also used for release gating; scorecards stop being honest.

Operational concerns

• Pack, policy, tool, and eval bundles are versioned independently; a release is a tuple of pinned versions, not a single SHA.
• Every long-running session has a kill switch wired to the rollback manager; the kill switch operates on the pinned-version tuple, not on individual primitives.
• Trace retention bands track data classification: traces touching regulated data follow the data’s retention rule, not the platform default.
• Run-budget headroom is measured per intent; intents trending toward budget exhaustion trigger an Autotune proposal before they fail under load.
• The reviewer-agent suite is itself a harness component; reviewer skills are versioned and golden-set-evaluated like any other.
• Search-set and test-set are kept disjoint by construction; the proposer (human or automated) never sees test-set scores during iteration.
• Lightweight interface validation runs before any full evaluation: import the candidate, instantiate, call its public methods on a tiny fixture, and require it to pass. Most malformed candidates die in seconds.

Evaluation metrics

Engineering metrics for the harness itself:

Product metrics carried by the harness’s DecisionRecord and replay log are documented in the Metrics Glossary.

Example

A condensed end-to-end harness flow for a sensitive action:

1. Intent: "support.refund.execute" arrives with RunContext.
2. Context plane: Context Pack v5.2.0 compiles a CompiledContext (8 stages).
   - intelligence_refs:  customer + order + prior corrections
   - policy_layer:       external/user-delegated/compensatable risk; approval required
   - tooling_layer:      adp_payments.issue_refund (destructive), adp_policy.eval (read_only)
   - evaluation_layer:   scorecard target { policy>=1.0, safety>=1.0 }
3. Decision plane: Planner proposes [eval, refund]; Critic.verify checks
   evidence coverage, ActionRisk, authority, and v1 projection before execution.
4. Trust plane: Policy Engine evaluates JsonLogic bundle; rule
   R_HIGH_VALUE_REQUIRES_APPROVAL requires approval gate GATE_FINANCE_APPROVAL.
5. Action plane: Tool Gateway blocks adp_payments.issue_refund pending approval token.
6. Reviewer agents (security, compliance, cost) emit findings; all clear.
7. Validate the exact stored approval, current authority, boundary/data controls,
   reservation and external preconditions; atomically redeem and dispatch the bound operation.
   Verify the provider postcondition; retain any supported compensation reference outside model-visible secrets.
8. DecisionRecord written: status=DECIDED, evidence_refs=[...],
   approvals=[...canonical approval receipt...], lineage={effect_observation_ref: ...}, trace_id=...
9. Scorecard: policy=1.00, utility=0.94, safety=1.00, latency=1830ms, cost=$0.0091.
10. Replay reproduces deterministic artifacts exactly and re-checks the declared refund outcome properties without calling the live payment tool.

Compare pack v5.1.0 and v5.2.0 against the same pinned task and recorded observations to inspect deterministic policy/artifact differences. Model-dependent outcomes require repeated property/scorecard evaluation; version pins alone do not establish causal attribution or byte-identical decisions.

Common misconceptions

• The harness is not a prompt. The prompt is one input to the Context Pack compiler. Most harness behavior lives outside any prompt: policy, gates, evidence binding, evaluation, replay.
• Harness Engineering is not “prompt engineering done well.” Prompt engineering is one tactic inside one stage of the Context Pack compiler. Harness Engineering covers what the model sees, what it can do, what counts as done, and how the system improves.
• The harness is one optimization target. Model capability, context, tools, environment, and evaluation interact. Compare declared configurations on your tasks; a benchmark result does not establish the largest lever for your deployment.
• “Reviewed by a human” is not a sufficient guarantee. Reviewer agents and replay suites catch patterns humans drift on; an independent authority approves, but the harness is what enforces.
• Improvement is not auto-pilot. Every Improvement Loop primitive emits proposals. Proposals run through the same release gate as packs and policies. Nothing auto-applies without an approver, except where policy explicitly permits it for a specific class.
• The harness is not a one-time build. It is a long-lived versioned artifact whose components age, decay, and must be re-baselined on a cadence. Treat it like any other production system.