Context Pack Compiler

Reference Architecture

The Context Pack Compiler is the only Context-plane component. It turns the pinned Context Pack plus the materialized Run Context into the CompiledContext envelope that drives the Decision plane.

Definition

A pure pipeline (no I/O, no clock, no global state) with eight ordered stages. Reads pack + Run Context + caller-retrieved structured evidence + promoted memory; emits compiled_prompt, manifests, runtime_controls, budget_report, and context_ledger.

Why it exists

The compiler centralizes policy/tool/evidence/memory composition so the same complete pinned inputs produce the same CompiledContext: pack, request, Run Context/budget, supplied evidence/conflicts and promoted memory. A pack/snapshot alone is not the full replay input.

The eight stages

1. Intent classification — use request.input.intent when supplied; otherwise apply the reference’s deterministic refund/lookup keyword fallback. Production validates supported intents against its catalog protocol before relying on this result; the compiler does not query a catalog service.
2. Policy resolution — JsonLogic over the merged eval context (Run Context + request); traverse priority-sorted bundles and preserve applicable rule obligations. An applicable predicate error throws ContextCompileError; false/else behavior remains distinct. Compiler tool prohibitions are cumulative, not replaced by a higher-priority allow.
3. Tool surfacing — compute Registry ∩ Permissions − Prohibitions, evaluate declared ActionRisk against action_risk_policy, and apply the required v1 projection against RunContext.safety_mode.
4. Evidence normalization — caller supplies retrieved candidates and conflict markers; compiler verifies supplied hashes and labels source, integrity, freshness, and covered requirements. It does not call out.
5. Memory recall — caller supplies promoted memory only (never raw capture) with source refs and freshness.
6. Token budget allocation — distribute RunBudget.bucket_tokens across context buckets (default proportional split if not declared).
7. Bucket admission and assembly — assign instruction treatment, admit eligible evidence, pack each bucket by priority, and emit structured omissions — never silently.
8. Manifests + runtime controls — evaluate sufficiency and emit the admitted evidence_manifest, context_admission, conflict-aware per-decision evidence_gates, policy/tool manifests, and the audit ledger.

Inputs

• Pinned ContextPack
• RunContext with RunBudget
• InvokeRequest (intent, message, channel, locale, structured context)
• Caller-retrieved RetrievedEvidence objects with source, content, integrity, freshness, and satisfies
• Caller-retrieved EvidenceConflictMarker objects with evidence refs, affected requirements, status, and explicit resolution when closed
• Caller-supplied PromotedMemory objects with source lineage and freshness

Outputs

• CompiledContext envelope:
  • compiled_prompt (system, developer, task, context_blocks)
  • provenance-bearing compiled_prompt.context_blocks
  • manifests (policy_manifest, tool_manifest, admitted evidence_manifest)
  • runtime_controls (context_admission, evidence_gates, policy controls)
  • budget_report (allocations, usage, bucket_truncations, dropped_block_ids, structured omissions, warnings)
  • context_ledger (pack, request, policy, tool, admitted evidence, memory, budget, omissions, hash)

Implementation learning

The reference compiler exposes diagnostics that operators need when a compile looks wrong:

• context_ledger: a deterministic audit summary for the compile, including pack ref, request id, policy bundles, surfaced tool identifiers, evidence refs, memory refs, budget summary, and a content hash for replay comparison.
• budget_report.dropped_block_ids: the exact blocks that did not fit, grouped by bucket. A boolean bucket_truncations flag is useful for dashboards, but it is not enough for debugging.
• tool_manifest[].capability_metadata: per-capability metadata such as native action_risk, its deterministic evaluation, the v1 approval_mode projection, and derivation/source. This lets the Critic and UI explain why a tool was surfaced without re-reading the registry.
• ContextBlock.provenance: the source_refs, origin, integrity, freshness, instruction_treatment, inclusion_reason, and derived content_hash for every material block.
• runtime_controls.evidence_gates: the deterministic explanation of which required evidence is satisfied, unresolved, conflicted, or rejected; its sufficiency and commit_allowed fields are derived rather than model-authored.
• budget_report.omissions: structured block, bucket, reason, token, priority, and source-ref data shared with the context ledger.

These are diagnostics, not execution knobs. They should be produced by the compiler from pinned inputs and included in replay fixtures; callers should not mutate them after compile.

Compiler eligibility is one boundary. Exact execution-profile admission, current authority/revocation, complete stored action approval, actual boundary/data routes, stable operations and aggregate reservations remain separate checks at dispatch. The compiler does not authenticate evidence labels, resolve credentials, redact payloads, create sandboxes or atomically enforce runtime budget stores. Production must supply trusted normalized inputs and verify referenced artifacts.

How it works (sequence)

RunContext + InvokeRequest + ContextPack
   ↓ classify intent
   ↓ resolve policies (JsonLogic, priority-sorted bundles)
   ↓ surface tools (registry ∩ permissions − prohibitions, native-risk policy, v1 ceiling)
   ↓ normalize evidence + memory (hash, integrity, freshness, conflicts, lineage)
   ↓ admit blocks (control vs data_only; reject ineligible evidence)
   ↓ pack to bucket budgets (priority-sorted; structured omissions)
   ↓ evaluate evidence sufficiency (coverage + admission + conflict state)
   ↓ emit manifests + runtime_controls + budget_report + context_ledger
CompiledContext

Failure modes

• Pack pinned by name without version — refuse at the boundary, before the compiler runs.
• Applicable JsonLogic predicate throws — stop compilation with ContextCompileError (code: policy_eval_error, stage: policy_resolution, bundle/rule IDs and expression_path: if). Safely serialize its fixed diagnostic; do not expose the raw expression/request/stack or fabricate successful output. There is no base non_enforcing bypass field. Inapplicable rules are not evaluated.
• Bucket assembly silently truncates — bug; the compiler must always set bucket_truncations[bucket] = true and list the dropped block IDs.
• Budget omission is recorded only as a block ID — diagnosability gap; emit a structured omission in both the budget report and ledger.
• Tool surfacing admits a capability with a denied native risk dimension or whose v1 projection exceeds safety_mode — bug; both checks must reject before manifest emission.
• Capability is governed by action_risk_policy but lacks native risk metadata — fail closed with unsupported_action_risk.
• Tool manifest omits capability metadata — diagnosability and enforcement gap; new classified capabilities must carry native risk, its evaluation, and the v1 projection.
• Caller supplies a content hash that does not match the evidence or memory content — reject the compile visibly.
• Evidence is untrusted, stale, unknown-freshness, or omitted by budget — retain it in rejection diagnostics; it must not satisfy an evidence gate.
• Required evidence has an unresolved conflict — retain every branch and the marker, report sufficiency: insufficient, and block commit until a recorded resolution selects evidence.
• Legacy string evidence has no integrity metadata — normalize it as untrusted/unknown and emit a migration warning.
• Retrieval or memory block is emitted as control — bug; external content must remain data_only.

Operational concerns

• Cold-start latency dominated by pack registry hydration; cache aggressively.
• Token estimation discrepancy between compiler and runtime LLM tokenizer; reconcile at the edge.
• Compile-time idempotency: identical inputs must return identical outputs (tested via replay).
• Treat all caller-retrieved content as data. Verification raises factual integrity; it does not grant instruction authority.
• Alert when a required-evidence gate changes from closed to open after a pack/compiler change.
• Per-pack-version compile-time metrics for regression detection.
• Catch compilation failures at invocation boundaries; do not silently reuse an earlier compiled context under changed policy or resume pins. Keep legacy compiler/replay hash behavior distinct from domain-separated canonical hashing for new attachments.

Evaluation metrics

• Compile-success rate per pack version.
• Compile-time p50 / p99.
• Bucket-truncation rate per bucket per intent (target: low; alert on spikes).
• Evidence-gate closure rate and unresolved requirements per decision key.
• Untrusted-evidence gate-satisfaction rate (target: zero).
• Stale/unknown/legacy evidence rejection rate and content-hash mismatch failures.
• Tool-surface size and native-risk declaration coverage at compile time per intent (target: minimal sufficient set and complete coverage).
• Replay determinism (identical CompiledContext on identical inputs).

Reference implementation

src/lib/contextos/compiler.ts is a typed reference implementation of the pipeline. It is not the production runtime but mirrors the contract.