Failure Modes and Recovery Playbooks

Reference Architecture · Cognitive Core

Governed failures in ContextOS are normalized into typed verdicts at contract boundaries; transport and implementation faults may begin as exceptions but must not cross the boundary silently. This page is a reference recovery map, not a claim that this repository ships the runtime.

Assign an owner, stop authority, response objectives and escalation/reconciliation queue to each applicable playbook under the exact admitted control profiles. Record observed recovery time/data loss and denied/useful-case evidence for the specific release/environment. Illustrative error labels below travel in the boundary’s declared diagnostic protocol (for example tool metadata); they do not add required fields to the base schemas.

Intelligence plane

• Edge missing evidence_ref — write rejected at the Knowledge Substrate boundary; emit ingest error; do not allow into the snapshot.
• Stale Knowledge Graph snapshot — block current-decision use if outside the freshness policy, then explicitly admit a new pin and reevaluate dependent context/approval. Historical replay retains its original pin; never silently flip it.
• Entity-resolution false positive — quarantine into the candidate review queue; never auto-merge below the confidence threshold.
• Embedding drift — re-embed with the pinned model version against the current snapshot; backfill index in place; alert on coverage delta.
• Cross-tenant traversal — denied at the storage layer; emit security event; runbook escalates to security on-call.
• Capture log unbounded — tier-by-tier compaction; alert if compaction lag exceeds budget.

Context plane

• Pack pinned by name without version — runtime refuses at boundary; return structured error to caller.
• Pack references missing ontology / decision spec / adapter — fail closed at compile; the Compiler does not start.
• Bucket truncation under budget — emit budget_report.bucket_truncations[bucket] = true explicitly; the Planner must handle.
• Applicable JsonLogic predicate throws — stop compilation with structured ContextCompileError (policy_eval_error, policy_resolution, safe bundle/rule/if-path details). Safely serialize the fixed diagnostic without raw expression/request/stack; do not treat the predicate as false or return a successful partial context. The base rule has no non_enforcing skip flag; independent local scoring failures cannot override required policy controls.
• Token estimator drift between Compiler and runtime tokenizer — reconcile at the edge; alert if drift exceeds tolerance.

Decision plane

• Plan contains disallowed tools or unsatisfied policy obligations — Critic emits replan or reject at verify; capped by max_replan_attempts.
• Loop guard trip — repeated identical tool calls or no-progress reflection short-circuits with structured loop_detected reason; routed back to Planner with reason populated.
• Run Budget exhausted — stop new work, emit the appropriate decision/transition diagnostic and preserve operations/reservations. A checkpoint cannot replenish capacity. Only separately admitted new work or explicit governed migration can change limits; continue authorized accounting/reconciliation of existing effects.
• Subagent lane mutates parent effects — invariant violation; lane terminated; emit security event.
• Background session resumes against a different pack version — refuse with pack_version_mismatch; require explicit migration.

Action plane

• Tool schema drift between registered and live endpoint — Tool Gateway returns failed with error_code = VALIDATION_ERROR; Executor returns control to Planner under re-plan budget.
• Approval gate timeout — a v1 paused receipt can accompany waiting/checkpoint state. A session ID locates work, not resume authority; expired approval needs fresh valid approval within remaining run limits.
• Approval gate denied — record REJECTED; do not dispatch the protected effect. Retain prior effects honestly and minimize audit under authorized retention.
• Lost acknowledgement or idempotency cache miss — keep stable operation identity and unknown effect/reservations. Cache absence or a key alone cannot permit retry; require current verified provider-specific idempotency or definitive absence evidence under the configured strategy, otherwise reconcile/escalate.
• Compensation needed — use the protected compensation reference for a new independently governed operation with fresh authority, approval and budget checks; it may fail or leave a partial outcome.
• Egress rate limit reached — Tool Gateway emits failed with error_code = RATE_LIMITED; Critic decides between retry-with-backoff and escalate.
• Cached read-only alias serves stale capability after a policy change — alias cache invalidated on bundle promotion; if missed, the alias’s first call re-validates against the live registry.

Trust plane

• Policy decision cache stale after bundle promotion — content-addressed bundle hashing forces fresh evaluation.
• Required revocation/identity/boundary evidence stale or unavailable — block protected dispatch/wake/resume; do not fall back to model judgment or a weaker profile.
• Trace gap because a custom adapter omitted W3C headers — caught by trace-coverage assertion; release-gate failure.
• judge model drift after upgrade — re-baseline golden sets; pin judge model and rubric.
• Required evidence store unavailable — fail closed for actions whose policy requires durable audit/effect evidence; a degraded path must be explicitly scoped and tested.
• Cross-tenant denial at read — emit security event; do not return any tenant data, even redacted.

Operator surface

• Operator action lacks the assurance required by its policy profile — refuse; WebAuthn is one deployment option, not a base-schema requirement.
• Promotion bypass attempt — refused by RBAC; emit security event.
• Approval binding mismatch, changed arguments or expired entitlement — reject; freeze the complete new action/evidence/policy/profile/release binding and obtain current eligible approval/quorum. Atomically redeem the stored approval only for its stable operation; a stale screen cannot authorize substitution or a second effect.

Cross-plane recovery

• Replay supports diagnosis and regression: pin complete artifacts and recorded transcripts, compare deterministic artifacts exactly and model-dependent stages by declared properties. Offline replay does not reverse a payment or authorize live recovery.
• Checkpoint resume requires verified bytes/pins, current atomic store ownership/revision/generation and required target fencing, plus current support, authority, approval, boundary, external preconditions and original aggregate budget. Frozen evidence alone is insufficient. Changed interpretation requires explicit migration/reauthorization.
• Cancellation and stale workers require stopping new dispatch, propagating child cancellation and enforcing fences. Verified stop acknowledgement does not erase confirmed/partial/unknown effects; retain their ledger and uncertain capacity until reconciled. Offline/edge reconnect follows the same rule rather than blindly replaying queued writes.
• Data/processor or retention failure requires blocking the prohibited actual route, minimizing protected payloads and reconciling memory/cache/index/export copies with bound receipts. An allowed deletion is not completed until the scoped complete inventory and every required propagation receipt are verified; external unknown copies remain explicit.
• The improvement loop consumes failed runs and operator corrections as typed proposals; review and ship under the same change-control process as packs and policies.