Observability

Reference Architecture

The Observability reference component owns OTEL spans, audit records, and the trace-store substrate that makes replay possible. DecisionRecord, DecisionAudit, and ReplayPacket are canonical contracts; aggregate TraceBundle, standalone PolicyDecision, and SecurityEvent shapes remain deployment profiles.

This is an instrumentation/storage design, not a shipped collector. Correlated traces, schema-valid receipts and signed records do not prove current authority, sandbox enforcement or external completion. Link exact contract/control/release refs, action/approval digests, stable operations, ownership generations and confirmed/partial/unknown effect evidence under the execution contract. Conformance assessment separately checks current independent runtime evidence and all mandatory useful/denied cases.

Definition

A trace + audit infrastructure that propagates W3C Trace Context end-to-end (compiler → planner → critic → gateway → adapter), tags spans with namespaced ContextOS attributes, and persists trace bundles + tool transcripts + decision records as the substrate for replay and audit.

Why it exists

Logs alone are insufficiently structured for audit. Audit needs typed, replay-addressable artifacts tied to a trace_id and a Decision Record. Signing and hash chaining strengthen deployments that adopt that profile; they are not universal base-schema requirements.

Inputs

• OTEL spans from every plane component
• Tool transcripts from the Tool Manager
• Approval-gate decisions from the Policy Engine
• Decision Records from the Decision Record store
• scorecards from the Evaluation Engine

Outputs

• Trace bundles with full W3C context
• Audit records (one per gate, one per policy decision, one per memory write proposal)
• Indexed trace store consumable by replay harness
• Security events (cross-tenant denials, credential rotations, sandbox violations)

OTEL contract

• Propagate W3C Trace Context through traceparent and tracestate. Optional W3C Baggage carries separately governed metadata; do not blindly forward inbound baggage or put credentials, personal data or authority assertions in it.
• Span attributes namespaced under contextos.*:
  • contextos.run_id
  • contextos.context_pack_ref
  • contextos.action.effect
  • contextos.action.authority
  • contextos.action.reversibility
  • contextos.action.interaction
  • contextos.action.data_scope
  • contextos.action.approval_mode_effective
  • contextos.trust.policy_decision_id
  • contextos.intelligence.snapshot_version
  • contextos.budget.tokens_used
• Tail-based sampling forced on: any run crossing an approval gate, failing scorecard thresholds, hitting the loop guard, producing a security event, or committing material external state.

Use the global run/pack correlation names above consistently with the architecture. Existing local names such as contextos.run.run_id need an explicit collector migration/alias mapping; do not silently blend them. Trace IDs and sampled flags remain untrusted correlation metadata, never authentication or a reason to bypass mandatory audit retention.

How it works

1. Every plane component emits spans against the inbound trace_id.
2. The Tool Manager re-emits traceparent on every outbound call to adapters.
3. Audit records are written append-only and indexed by (trace_id, decision_key, time_window).
4. Replay queries fetch the trace bundle + recorded transcripts + pack version + snapshot version → re-derive deterministic artifacts exactly and evaluate model-dependent stages against declared properties without re-executing tools.

Failure modes

• Custom adapter omits W3C headers — trace gap; caught by coverage assertion in CI.
• Tail-based sampler back-pressure drops a high-risk run — alert; sampler must be sized for peak.
• Audit store availability — critical path of every governed run; replicate.
• Cardinality explosion in tag values — mitigated by namespaced attribute schema and lint-on-merge.

Operational concerns

• Trace retention bands by data_classification (PUBLIC / INTERNAL / CONFIDENTIAL / RESTRICTED).
• Per-tenant cost budgets for trace storage.
• PII redaction applied at trace emission, not at query time.
• Quarterly incident-response drill exercising replay end-to-end.
• Bound storage and sampling by explicit retention, incident holds and privacy policy. Govern actual telemetry recipients/processors/residency, caches and exports with verified source labels and minimized fields; refs and hashes can also disclose sensitive relationships.
• Declare fail-closed or bounded durable-buffer behavior for audit outages under the workflow’s required controls. Missing audit evidence remains a gap; replication alone is not an availability or durability proof. Preserve uncertain effects and protected accounting facts if an export fails after provider commit.
• Test owned stop/recovery and evidence-store outage paths, recording observed recovery time/data loss and limitations for the specific release/environment rather than asserting universal targets.

Evaluation metrics

• Trace coverage (fraction of runs with full plane span chain).
• Audit completeness (fraction of governed actions with a matching audit record).
• Exact-match rate for deterministic replay artifacts and property-pass rate for model-dependent stages.
• Mean time to fetch a replay bundle.