Build an AI Agent Replay Test with the Context Pack Compiler

An AI agent replay harness checks whether recorded inputs reproduce an expected result without repeating the original external actions. Start with the part you can make deterministic: context compilation. Then test deliberate changes to permissions, approval thresholds, and evidence availability so the harness proves it can detect a difference.

This guide runs four tests against the actual ContextOS Context Pack compiler. The inputs are synthetic support-refund fixtures. There is no model call, payment execution, production trace, or latency benchmark. The code below is the complete test file, not pseudocode with missing runtime imports.

Run the example

Use Node.js 24 or newer. The runtime repository holds an unmodified, checksummed snapshot of the reference compiler and its fixtures, with a test setup that resolves the imports below. Save the test file as conformance/generator/replay-tutorial.gen.ts in a clone, then run it:

git clone https://github.com/contextosai/ContextOS.git
cd ContextOS
npm ci --prefix conformance --ignore-scripts
# save the test file below as conformance/generator/replay-tutorial.gen.ts
npm test --prefix conformance -- replay-tutorial

The expected result is one test file passing, four tests passing. Record the checkout’s commit with git rev-parse HEAD when sharing a result. Test duration varies with the machine and is not an agent-runtime performance measurement.

The scenario fixture supplies a versioned support pack, request, run context, evidence, and promoted memory to the reference compiler. compileScenario returns a projection of the compiled context, including its context hash, surfaced tools, runtime controls, and budget report.

The executable tests

import { describe, expect, it } from "vitest"
import { compileScenario, DEFAULT_KNOBS } from "@/lib/contextos/scenario"
 
// Executable companion to /blog/replay-harness-in-code.
// Uses synthetic fixtures and the real compiler; no model or external tools.
describe("replay tutorial: Context Pack compiler", () => {
  it("reproduces the same result from the same fixture", async () => {
    const original = await compileScenario({ ...DEFAULT_KNOBS })
    const replayed = await compileScenario({ ...DEFAULT_KNOBS })
 
    expect(replayed).toEqual(original)
    expect(replayed.contextHash).toBe(original.contextHash)
    expect(original.contextHash).toMatch(/^sha256:[a-f0-9]{64}$/)
  })
 
  it("detects a changed permission ceiling and withholds the payment tool", async () => {
    const original = await compileScenario({ ...DEFAULT_KNOBS })
    const restricted = await compileScenario({ ...DEFAULT_KNOBS, safetyMode: "read_only" })
 
    expect(original.surfacedTools.some((tool) => tool.adapter_id === "adp_payments")).toBe(true)
    expect(restricted.surfacedTools.some((tool) => tool.adapter_id === "adp_payments")).toBe(false)
    expect(restricted.contextHash).not.toBe(original.contextHash)
  })
 
  it("keeps the finance approval boundary observable", async () => {
    const atThreshold = await compileScenario({ ...DEFAULT_KNOBS, refundAmount: 3000 })
    const aboveThreshold = await compileScenario({ ...DEFAULT_KNOBS, refundAmount: 3001 })
 
    expect(atThreshold.runtimeControls.approval_gates_active).not.toContain("GATE_FINANCE_APPROVAL")
    expect(aboveThreshold.runtimeControls.approval_gates_active).toContain("GATE_FINANCE_APPROVAL")
    expect(aboveThreshold.contextHash).not.toBe(atThreshold.contextHash)
  })
 
  it("blocks commitment when the evidence budget excludes required evidence", async () => {
    const result = await compileScenario({ ...DEFAULT_KNOBS, evidenceTokens: 0 })
    const gate = result.runtimeControls.evidence_gates.find(
      (candidate) => candidate.required.includes("order_lookup"),
    )
 
    expect(result.budget.droppedBlockIds.length).toBeGreaterThan(0)
    expect(gate?.commit_allowed).toBe(false)
  })
})

What the results tell you

The threshold is part of this synthetic fixture, not a recommended financial policy. A surfaced tool also is not permission to execute it: the downstream runtime must enforce the emitted approval and evidence gates.

The first test checks the result projection and the compiler’s context hash. It does not compare every byte of a historical DecisionRecord. It also does not prove every possible compiler input is deterministic. The changed-input tests make the example useful beyond “run the same function twice”: each change has a specific, asserted control consequence.

For a cross-release regression suite, retain a reviewed expected result separately from the compiler under test. Generating both the expected and actual result with the new compiler would accept the same regression twice. Pin the pack contents, fixture, compiler revision, dependencies, and expected result; review intentional baseline changes alongside the policy or code change that caused them.

Extend this into production replay

The compiler test is one layer. A full agent replay system needs additional implementation and retained artifacts:

ContextOS publishes a ReplayPacket schema and runtime contracts. Those contracts describe what implementations exchange; the compiler test does not implement a trace store, signature service, or full replay runner.

The current ReplayPacket contract includes context_pack_ref, compiled_context_hash, side_effect_policy, and an expected result. Its result vocabulary includes replay_equal, replay_mismatch, and invalid. Use those canonical fields and statuses when implementing the contract. This tutorial’s Vitest assertions are test outcomes, not a new replay API.

Pack signing is required only when the deployment’s supply-chain profile requires it; it is not a required field in the base Context Pack contract. A content mismatch can come from a changed input, implementation drift, corruption, or tampering. The mismatch alone does not identify which happened.

Limits that matter

• No external effects: the example compiles a tool manifest; it does not call the listed tools or prove that a production gateway blocks calls.
• No fresh model generation: recording and replaying a model response checks downstream behavior. Calling a model again tests new behavior and does not guarantee identical text, even with the same prompt.
• No authenticity claim: equality and context hashes establish a repeatability check within this fixture. They do not establish an independent chain of custody.
• No legal or compliance conclusion: reproducing a record is one useful audit input. It does not certify that the original decision was correct or authorized.
• No performance claim: these tests measure neither production latency nor replay cost. A benchmark needs its workload, environment, retained artifacts, and measurement method published with the result.

Try one change yourself

Run the suite, then temporarily change the above-threshold test’s amount from 3001 to 3000. The assertion that finance approval is active should fail. Restore the amount and rerun. That small failure demonstrates the value of a replay check: a concrete boundary changes, and the test tells you which expectation stopped holding.

For a browser view of the same compiler, use the Context Pack playground. For your own runtime, start with one retained decision and one explicit invariant, then add transcript and integrity checks before claiming end-to-end replay.

Sources and revision

This article is an implementation exercise based on the repository’s compiler, synthetic fixture, canonical types, and the executable tests above. The October 4 revision replaces illustrative full-runtime modules and an unsupported timing anecdote with the narrower tested example above. The November 4 revision runs the tests against the reference snapshot in the runtime repository, because the specification’s source repository is not public; the tests and expected results are unchanged. See the editorial and corrections policy.

What to read next

• Context Pack: schema, compiler, and example
• DecisionRecord: the retained decision artifact
• Runtime API contracts and replay schema