An incident review can fail even when every service kept logs: the retrieval index was rebuilt, the policy edited and the model replaced. Reconstructing the run then requires assumptions about evidence that may no longer be available. This is an illustrative failure scenario, not an independently verified regulatory incident.
Replay provides a disciplined way to check retained artifacts and investigate differences. It is one part of an audit story alongside authenticated collection, access controls, provider observations and declared evidence limits.
October 16 revision: byte equality is appropriate for deterministic artifacts under identical retained inputs and versions. Model-dependent reruns, changed environments and missing/expired evidence need separate outcomes; a mismatch alone does not prove tampering. No general replay runtime or production audit was executed by this article.
2026 update: traces are the carrier, records are the contract
Open standards have already solved part of the problem. W3C Trace Context defines a common way to propagate trace identity between services, and OpenTelemetry gives teams a mature vocabulary for traces, metrics, and logs. Use them.
But do not mistake propagation for audit. A trace lets you correlate the run. A Decision Record tells you what was decided, which evidence and policy it relied on, which approvals were collected, and whether replay can reproduce it. ContextOS uses the trace as the spine and the Decision Record as the audit contract.
The shape of the problem
There is a moment in any post-incident review where someone looks at the on-call channel and asks: “what did the agent actually do, and why?” The default tools answer that question with prose — model output, paraphrased reasoning, a stack trace, a screenshot of the user’s chat. It feels like an answer. It is not.
Three failure modes show up reliably.
The first is that the logs are unstructured. Some lines are JSON, some are stack traces, some are the model’s recollection of its own behavior. Reconstructing the run requires interpretation, which means the answer depends on who reads the file. Two engineers given the same logs will reach different conclusions in non-trivial cases. That is bad enough internally; it is a disaster in front of a regulator.
The second is that the logs are not tamper-evident. Anyone with shell access to the log store could edit the JSON. The auditor knows this. So do you. The polite thing to do at this point is to stop calling them an audit trail.
The third is that the run is not reproducible. The model has been swapped. The retrieval index has been rebuilt. The policy has been edited. Even if the logs were perfect, the world they describe no longer exists. The team has to assume the logs match what happened; they cannot prove it.
These are different problems with the same root cause: logs were never a contract. They were a hope.
The contract
The contract must declare its replay mode and retained inputs:
Given an authorized trace reference, verify available artifact integrity and compare deterministic outputs exactly; evaluate model-dependent outputs against declared properties, reporting unavailable evidence and unexplained differences explicitly.
This can support audit; it cannot prove source truth, authority or external effects by itself.
Replay rests on four pinned inputs:
| Input | What it pins | Refused if |
|---|---|---|
| Context Pack version | ctxpack.support@5.2.0 with content hash | unverified provenance or unpinned |
| Knowledge Graph snapshot | kg_2026_04_07_T0930 | snapshot has been garbage-collected |
Recorded invokeAgent envelope | RunContext, claims, scopes, intent, budgets | envelope hash mismatches |
| Recorded tool transcripts | every toolCall / toolResult pair | chain hash breaks |
Those are a starting inventory, not a complete universal set: retain all applicable policy/profile/release pins, tool/evaluator schemas, approved action basis, trusted control observations and accounting/state transitions. The offline comparison must never dispatch provider effects. A deterministic mismatch is a typed difference to investigate; model variability, implementation drift, missing evidence and integrity failure are distinct causes. Reserve tampering claims for supported integrity/authenticity evidence.
What a replayable Decision Record looks like
{
"record_id": "dr_2026_04_08_a17",
"decision_key": "support.refund.execute",
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
"evidence_refs": [
"kg:order:ord_881#snapshot_kg_2026_04_07_T0930",
"tool:orders.lookup:tc_117",
"tool:policy.eval:tc_119",
"tool:payments.refund:tc_121"
],
"policy_decisions": [
{ "policy_decision_id": "pol_9900", "rule_ids": ["R_REFUND_REQUIRES_IDV"] },
{ "policy_decision_id": "pol_9901", "rule_ids": ["R_REFUND_LIMIT_BY_ROLE"] }
],
"approvals": [
{
"gate_id": "GATE_HIGH_VALUE",
"approver": "user_finance_lead_77",
"approval_mode_effective": "destructive",
"evidence_snapshot_hash": "sha256:b2a1...",
"decided_at": "2026-04-08T09:31:30Z"
}
],
"lineage": {
"pack_version": "ctxpack.support@5.2.0",
"snapshot_version": "kg_2026_04_07_T0930",
"model": { "id": "model_xyz", "sha256": "..." }
},
"audit": {
"chain_prev_hash": "sha256:7c4a...",
"signed_by": "kid_runtime_2026Q2"
}
}This JSON is an abbreviated application projection with placeholder digests, not the complete canonical DecisionRecord schema or verified observations. Structured fields can also be fabricated. Current full-action stored approval and entitled quorum, trusted collection and provider postconditions must be established independently; an approver-name string or signature does not authenticate itself.
Hash chains, in plain language
Decision Records, tool transcripts, and policy decisions for a given run form an append-only chain keyed by trace_id. Each entry’s hash includes the previous entry’s hash. Tampering with any entry breaks the chain at that point, which the harness detects on replay.
A chain detects some changes relative to a trusted retained head; an attacker who can rewrite both the entries and an unprotected head can recompute it. Protect authenticated collection, sequence/completeness, trusted anchors, key history and external evidence. Integrity does not prove that an observation was true or that omitted events never happened.
What changes when replay is the IR primitive
Incident response stops being narrative reconstruction and starts being a query.
The investigator resolves an authorized trace reference and selects a replay mode. Check retained input hashes and trusted anchors, then compare deterministic artifacts or declared model-dependent properties. Report unavailable inputs, divergent stages and unknown provider effects separately. A matching artifact does not prove that the original execution was authorized or that its supplier effect succeeded; inspect current incident evidence and the dispatch-time control observations.
What changes for compliance
Compliance asks change shape entirely. A request like “pull every refund over 10,000 rupees in Q1, who approved them, and what evidence was on file” used to be a multi-week join across systems. With Decision Records, it is a projection:
Illustrative query plan, not executable SQL:
- authenticate the investigator and authorize tenant/purpose/field access
- select refund decisions within [2026-01-01, 2026-04-01)
- interpret business money with explicit minor units and currency
- join trusted exact approval, provider effect and evidence references
- disclose unavailable or retention-limited artifactsThe runtime may have retained part of the audit evidence. The compliance team is reading what the runtime emitted, not reconstructing what the team thinks happened.
Replay also catches drift before the regulator does
This is the part most teams under-use. Replay is not only an incident tool; it is a continuous test against the live runtime. Sample historical trace_ids, replay them on today’s runtime, and any non-match is signal:
A model upgrade silently changed verdicts on inputs the team thought were canonical. A policy bundle change made a previously-allowed call a denial. A pack edit shifted which tools the model surfaces. A retrieval change altered which evidence the run grounded against.
These are candidate drift cases for a controlled comparison. Freeze candidates before protected final evaluation, bound repeated feedback and treat newly observed differences as evidence to investigate rather than automatic permission to release.
For more on this loop, see evaluation and observability.
What it costs you operationally
A few implementation commitments need explicit owners and evidence.
Tail-based sampling has to retain eligible minimal control evidence for consequential, denied, escalated and failed-scorecard work, subject to purpose/access/retention requirements. These are the IR-relevant runs by construction. If your sampling pipeline drops them, your audit story drops with them.
Time-to-replay for a given trace_id becomes an SLO. If the team cannot stand up a replay within hours, the contract is theoretical.
Signing keys rotate, and replay has to remain valid against historical keys. The signing-key registry needs effective windows and a “revoked-keys remain queryable for replay” rule. This is unglamorous and the kind of thing that gets missed until the first key rotation.
A quarterly drill end-to-end on a real production trace_id keeps the contract honest. If the team has not done it cold in six months, they probably cannot.
What people get wrong
The two arguments against this approach are both wrong, and they are wrong in the same way.
“OTEL spans are enough.” Spans are part of the audit. They are not the audit. Spans without a Decision Record give you observability without provenance — you can see what happened, but you cannot prove the verdict.
“Replay is too expensive.” Replay does not re-execute tools. It re-runs the canonical loop against recorded transcripts. Cost depends on retained artifacts, replay mode and model/evaluator calls; measure it and charge replay/review work to its permitted budget.
Both arguments treat replay as an optional addition to a logging-based audit. It is not an addition. It is a different kind of artifact that does the work logs cannot.
Audit-readiness checklist
| Check | Required evidence |
|---|---|
| Trace propagation | One trace_id connects compile, plan, policy, tool, approval, evaluator, and final record spans. |
| Pinned inputs | Pack version, policy bundle, KG snapshot, model profile, and request envelope hashes are recorded. |
| Tool transcripts | Every tool call/result pair is stored with schema version, idempotency key, and hash chain position. |
| Human approvals | Approver, gate id, effective mode, frozen evidence hash, and timestamp are first-class fields. |
| Replay result | Deterministic artifacts compare exactly; model-dependent properties, unavailable inputs and integrity failures are distinct outcomes. |
| Retention | Eligible minimal evidence has declared access, sampling, retention and copy-propagation rules; raw sensitive content is not retained indefinitely. |
file:line evidence.Current controls on investigation and continuation
Trace Context propagates traceparent/tracestate; separately governed Baggage must not carry credentials or unrestricted sensitive metadata. A trace ID is a correlation reference, never a bearer authorization token. Current owned tenant/audience/purpose grants and ancestor restrictions/revocation govern investigators, replay workers and every reader/processor. All nine data stages and derived trace/cache/export copies need retention and correction/deletion receipts; lawful evidence removal can make replay unavailable without implying tampering.
Replaying recorded calls offline cannot authorize live re-execution. A requested new effect needs current full-action/evidence/pin approval and entitled independent quorum plus native risk, guardrails, active boundaries/credentials/data, provider preconditions and capacity. Applicable policy errors stop compilation. Protected atomic stores and generation fencing govern resume; original shared spent/reserved capacity and unknown provider reservations survive. Cancellation or future release rollback cannot prove stop or reversal.
Exact profiles and current independent conformance evidence require useful, denial and fault observations for the actual workflow/release/environment. Source replay tests cannot certify a deployed audit pipeline.
Closing
When an authorized investigator asks what happened, supply the retained typed record, provenance and integrity results, applicable replay comparison and external-effect evidence. State every missing input and unresolved outcome. That is a reviewable evidence package; byte equality alone is not the definition of audit.
What to read next
- Run the Harness Audit — score replay and 39 other controls against your real code
- Replay Harness in Code
- Context Graphs
- Approval Gates in Code