Evaluation and Observability

Evaluation and Observability is the Trust-plane capability that turns runtime behavior into measurable, replayable evidence. Without it, the other planes are unverifiable.

Definition

A coordinated stack of: OTEL-first tracing that records every plane and primitive that touched a request; evaluators that score each run on Policy / Utility / Latency / Safety / Economics; golden sets and replay harnesses that catch regressions before release; and continuous improvement primitives that turn observed failures into proposed fixes.

The Execution Contract and exact Control Profiles make evidence requirements explicit for the admitted features. ReferenceValidationReport records actual deterministic fixture results with runtime controls unevaluated. ConformanceReport binds real implementation/workflow/release/environment evidence; its assessor requires current trusted verification for mandatory controls, registered useful/denied cases and required utility cases. Empty, stale, unsupported, synthetic or self-asserted evidence cannot pass. See Conformance and Evidence.

Why it exists

LLM-driven systems drift. Models change, evidence changes, policies change, tools change. A trustworthy runtime cannot be “tested at release and trusted forever” — it must be measured on every run, replayable for any past run, and improved on a closed loop. This plane is what makes that loop concrete.

How it works

1. Instrument: every primitive (compiler, planner, executor, critic, gateway, memory) emits OTEL spans with W3C trace context propagated end-to-end.
2. Score: each completed run is scored against evaluators with rubric-bound metrics.
3. Sample online: a configurable fraction of production runs route to LLM-as-judge plus rule-based scorers.
4. Replay offline: golden sets and recorded production runs are replayed on every change to the planner, prompt, pack, or model.
5. Gate releases: a release is blocked if scorecard deltas exceed configured thresholds.
6. Improve: failed and corrected runs feed the Insight Synthesizer, Strategy Compiler, and Feedback Store that propose changes back into the catalogs and packs.

Observable trajectory and protocol evidence

Outcome-only grading cannot distinguish a robust run from a lucky answer. A release-grade evaluation keeps a portable observable trajectory containing the release tuple, available skills and tools, activation and file-read events, tool calls and results, observations and structured errors, workspace or external-state diff, final response, subagent links, token/cache/latency/cost data, security and policy events, and grader version plus raw verdict.

This is not hidden chain-of-thought. ContextOS records governed runtime decisions and effects, not private model reasoning. A protocol card for the evaluation names the task contract, environment and reset procedure, evidence visible to the agent and verifier, tool/effect rules, grader operating profile, assurance depth, pass conditions, and known blind spots. The trajectory and protocol card make results comparable across harness releases and portable across evaluation systems.

Evaluators

The Critic scores every completed run on five dimensions:

Scorecard deltas are tracked per intent, per tenant, and per Context Pack version, so regressions are localizable to the cause.

Skills require a paired scorecard. Run the same representative tasks against the same release tuple with a candidate skill available and withheld. Skill Lift is the verified-success delta attributable to the skill after also checking activation precision, irrelevant-task silence, trajectory compliance, security, latency, cost, and interference with neighboring skills. A final-answer improvement does not pass when the path violates authority, leaks data, or makes the result less replayable.

Eval-type taxonomy

The five evaluator dimensions above are the axes every run is scored on. Beneath them, a release-grade harness runs multiple eval types, each answering a different question about whether the change is safe to ship:

Unit, contract, and interface-validation tests run cheaply on every commit. Policy / tool-use / reasoning / regression / simulation / UX tests run on the relevant change classes. Business and safety evals run continuously against production traffic with stratified sampling. Search-set and test-set are kept disjoint: the proposer (human or automated, including the Improvement Loop) uses development and selection evidence; the release gate uses protected final evaluation as defined by the comparison protocol.

Harness comparison protocol

Evaluate a released model, harness, and environment configuration, not a model name in isolation. An agent’s statement of completion, a valid DecisionRecord, and a favorable judge score are distinct from the observed result. Accepted completion needs the workflow’s required outcome evidence; policy or safety failures cannot be averaged away by higher utility.

Include a simpler candidate when evaluating added planners, skills, resets, or reviewer calls. Report whether the additional mechanism earns its cost on the intended models and workload, and retain failure/interruption cases. This makes removal of unnecessary machinery a testable release decision rather than a stylistic preference.

A harness comparison must declare its experimental unit and the decision it is intended to support. Distinguish fixed-model harness comparisons from whole-system comparisons; do not pool their results as one causal estimate. A version tuple enables attribution but does not by itself establish causality.

The locally versioned protocol card must retain:

For intended use across models, cross baseline and candidate harnesses with each intended executor and report the interaction. A confidence interval spanning zero does not establish equivalence. A group discovered after inspection is exploratory until tested on fresh cases. Infrastructure failures and missing measurements stay visible in denominators and exclusions.

Use three partitions when tuning candidates: development evidence, validation for candidate selection, and protected final evaluation after the candidate is frozen. Once final-test feedback informs another edit, that partition has become development evidence for the next claim; replace or replenish the protected evaluation and record access. Audit replay checks pinned decisions without side effects; an interactive comparison must let each candidate choose its own actions in reset, bounded environments.

This protocol is supported by same-model comparison and telemetry corrections, infrastructure experiments, and HarnessDev’s transfer evaluation. The studies motivate controls; their scores are not ContextOS release thresholds.

Usage accounting and incomplete evidence

Preserve raw provider usage and its semantic version before normalization. Cache-inclusive and cache-exclusive input fields must not be added under one assumed convention. Record exposed cache reads/writes, uncached input, output, tools, environments, failed attempts, and all subagent work; prevent parent/child double counting. Reconcile estimates against billed totals when available and disclose unreconciled coverage.

Missing usage is unknown, never zero. Report measured coverage and sensitivity bounds where possible; withhold a cost-ranking claim if plausible missing spend changes its order. Include packet construction and reviewer work when comparing context policies. Keep optimization/search cost separate from execution cost, then report both in adoption economics.

For live profile-aware work, reserve root/child/retry/replan capacity in the same BudgetLedger before spending. Cumulative verified usage settles against an explicit currency/quantum; convertLegacyCostToUnits bridges existing fractional cents exactly or rejects unrepresentable precision. Unknown/cancelled usage retains reservations, and replay/resume cannot reset consumed capacity. Measurement completeness and atomic enforcement remain separate properties to demonstrate.

Safety and useful completion remain separate endpoints. Measure benign utility alongside prohibited effects across configuration, extension, operation, persistence, action, and recovery. Recognizing an attack is not preventing its effect; fresh reviewer context is not proof of independence. Zero observed violations on a finite test set is a release observation, not a universal safety guarantee.

Release gates and feature-flagged rollout

A passing scorecard is necessary but not sufficient. New harness components — pack versions, policy bundles, tool additions, planner skills — flow through staged rollout, with each stage carrying an explicit advance gate:

Every stage carries a kill switch: a control-plane operation that stops affected execution or restores the validated prior version and routing. Managed profiles must declare an achievable fallback when provider code cannot be restored; see managed runtime conformance. Replay must reproduce deterministic artifacts exactly and re-check declared outcome properties for probabilistic stages; byte equality is required only where canonical serialization and transcript substitution make it meaningful.

OTEL-first tracing

The runtime treats OTEL as a primitive, not a sidecar:

• W3C Trace Context (traceparent, tracestate) propagates across compiler → planner → critic → gateway → adapter. Optional W3C Baggage is a separate governed metadata mechanism: allowlist eligible keys and never propagate credentials, unrestricted personal data or authority assertions.
• Every plane writes spans. Use stable OpenTelemetry attributes where defined; version ContextOS-specific attributes such as pack version, policy decision id, native-risk dimensions, v1 mode projection, evidence count, and budget usage.
• The Run Context’s trace_id and run_id appear on every span and on the resulting Decision Record.
• Tail-based sampling preserves all runs that crossed an approval gate, hit a loop guard, or failed scorecard thresholds.

Telemetry is a governed data flow. Preserve correlated run/trace/operation/decision/control refs while minimizing payloads, excluding credentials and applying tenant/reader/processor/residency/retention policy. A trace attribute saying a control was active is not a verified runtime attestation. Collect authorized boundary, broker, store and provider observations separately; missing evidence stays visible even when an overall scorecard is favorable.

Replay

Replay is the contract that makes evaluation honest. Given:

• the Context Pack version,
• the Knowledge Graph snapshot,
• the recorded invokeAgent envelope,
• the recorded tool transcripts,

deterministic checks and artifacts must reproduce without re-executing tools. Probabilistic planner/Critic outputs are compared against recorded verdicts, required properties, and scorecards. This lets evaluation re-score historical runs without treating model sampling as deterministic.

ContextOS distinguishes four replay modes:

The portable input is a ReplayPacket. It carries the source reference, pinned pack, compiled-context hash, transcript-chain hash, side-effect policy, and expected outcome.

Continuous improvement loop

Failures and corrections are first-class signals, not log lines. The full set of improvement-loop primitives — Insight Synthesizer, Strategy Compiler, Feedback Store, Chief-of-Staff, Research Queue, Autotune — has its own foundation doc: see Improvement Loop.

Outputs of these primitives are proposals, not auto-applied changes. They land in the same change-control system as Context Packs, Decision Specs, and policies.

Interfaces

Inputs

• OTEL spans from every plane
• Tool transcripts and evidence manifests
• Operator corrections and approval-gate decisions
• Golden sets per intent
• Release-gate thresholds

Outputs

• Per-run scorecards
• Trace bundles consumable by audit and post-incident review
• Replay datasets
• Improvement proposals (insights, strategy rules, tuning suggestions)
• Release-gate verdicts

Failure modes

• Sampled-in only the easy cases — production sampling not stratified by intent or risk, blinding regressions on rare-but-high-risk paths.
• LLM-as-judge drift between evaluator model versions; mitigated by pinning judge model and rubric, and re-baselining golden sets on judge upgrades.
• Replay drift caused by an unpinned dependency, missing transcript, changed evaluator, or an invalid exact-equality expectation for probabilistic model output.
• Trace gaps when a custom adapter forgets to propagate W3C headers — caught by a budget-coverage assertion.
• Improvement loop loops on its own corrections without human review.

Operational concerns

• Sampling stratified by intent, risk tier, and tenant.
• Human audit allocation combines deterministic gates for high-risk runs with stratified random coverage of nominal runs; agent self-confidence never decides whether a run escapes review.
• Reviewer quality is evaluated for missed defects, unsupported approvals, disagreement resolution, and non-vacuity—whether review evidence could actually change a release or action verdict.
• Multi-agent and evaluator independence is measured across evidence, method, authority, and state; correlated judges are not counted as independent assurance.
• Tail-based sampling forced for runs crossing destructive gates or failing scorecard thresholds.
• Trace retention bands by data classification.
• Cost budget per evaluation run separate from the production Run Budget.
• Quarterly rebaselining of golden sets and judge rubrics.

Evaluation metrics

• Coverage: fraction of production runs with full trace, full scorecard, and resolvable evidence_refs.
• Detection lead time: time from regression introduction to release-gate block.
• Replay conformance: exact equality for deterministic artifacts plus required-property/scorecard agreement for probabilistic stages.
• Improvement adoption rate: fraction of proposals that ship after review.
• Mean time to safe completion per intent.
• Skill Lift: paired verified-success delta with safety, cost, latency, activation, and trajectory guardrails.
• Oversight non-vacuity: fraction of sampled reviews with sufficient evidence and authority to uphold, revise, or block the proposed verdict.
• Cost per trusted outcome: total inference, retrieval, tool, retry, evaluation, observability, infrastructure, and human-review cost divided by accepted, grounded, policy-compliant outcomes; reported through contextos.budget.cost_per_verified_success.

Example

A condensed scorecard for one run:

This illustrative scorecard is not a ConformanceReport; release_gate: "pass" describes its configured evaluation gate only and certifies no production profile.

{
  "run_id": "run_a1b2c3d4e5f60718",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "intent": "support.refund",
  "pack_version": "ctxpack.support@5.2.0",
  "decision_id": "support.refund.execute",
  "scores": {
    "policy":   { "score": 1.00, "violations": [] },
    "utility":  { "score": 0.92, "judge_rationale": "..." },
    "latency":  { "wall_clock_ms": 1840, "p99_target_ms": 3000 },
    "safety":   { "redaction_success": true, "evidence_coverage": 1.00, "hallucination": false },
    "economics":{ "tokens_used": 4720, "tool_calls": 3, "cost_usd_cents": 0.91 }
  },
  "release_gate": "pass",
  "improvement_proposals": []
}

Executable paired report evaluation

The Astra research article proposes paired evaluation of accepted work, review burden, authorized scope, and total cost. The deterministic reference scenario in src/lib/contextos/evaluation-scenario.ts implements its fixed-harness comparison with synthetic recorded observations, not model calls or measured Astra performance. It leaves the portable runtime contracts and compiler unchanged.

Run locally with Node 24 or newer and installed repository dependencies:

npm ci
npx vitest run tests/evaluation-scenario.test.ts --silent=false --disableConsoleIntercept

The test prints candidate summaries and paired task deltas. Applications and tests can also call await compileEvaluationScenario() to obtain every attempt’s full CompiledContext, including manifests, evidence gates, admission controls, budget report, and context hash. evaluationFixtures() returns fresh editable fixtures for negative cases.

The pinned protocol records fixture model identifiers, date, reasoning effort, prompt, tools, context policy, attempt limit, time and execution-cost budgets, dataset, and grader version. Both candidates start each task from the same snapshot reference; no external state is modified. The six cases cover normal completion, duplicate delivery, a stale result after steering, an uncertain write, an unavailable evidence source, and a constraint surviving compaction with an injected upload instruction. Recovery and honesty observations remain separate from completion: an honest unresolved handoff earns no acceptance point.

The independent fixture grader requires a completed artifact for the requested quarter, correct recovery, an honest handoff, no unauthorized actions, sufficient compiler-admitted evidence, and cumulative execution within budget. The final attempt is the predeclared selector; the grader never selects the best attempt. Any unauthorized attempt anywhere in the task disqualifies acceptance, even when blocked or followed by a clean retry. The grader and its observations are kept out of the compiled task prompt. Evidence sufficiency comes from the existing report.accept gate requiring current, verified period, dataset, and calculation evidence; it is not inferred from a completion claim.

Failed and abandoned attempts remain in every applicable numerator. When acceptance is zero, per-accepted cost and review values are null. Execution budgets cover model, tools, and infrastructure; human review and rework are accounted separately in total cost. The fixture results are 2/6 accepted baseline tasks and 4/6 accepted Astra-style tasks. These numbers are chosen examples and provide no empirical evidence of model superiority.

This is a transcript-substitution reference-validation example: recovery and honesty are recorded fixture judgments, not a live implementation of steering, cancellation, compaction, or tool execution. The fixed baseline-first review order is reproducible; it does not claim randomized or blinded human review. A real study must independently inspect artifacts and tool receipts, conceal identities where practical, randomize review order, repeat held-out tasks, and report uncertainty grouped by task. The article’s separately configured whole-system comparison requires a separately labeled study and comparable tuning budgets; it must not be pooled with this fixed-harness example. This small synthetic suite estimates neither deployment reliability nor rare-event safety.

Common misconceptions

• Evaluation complements CI and QA. Release-scoped tests and ongoing measurements both need declared outcomes, failure cases and evidence.
• Observability is not logging. Logs are unstructured; this plane emits typed spans tied to a Decision Record.
• Replay is not optional. Without replay, post-incident analysis becomes guesswork and improvement proposals cannot be validated.
• Improvement proposals are not auto-applied. They land in the same change-control system as packs and policies.