Reviewer Agents

Reviewer Agents are specialized harness components that inspect proposed changes—pack diffs, policy edits, tool additions, planner skill updates, agent-generated PRs—before they reach the independent promotion authority selected by policy. In the high-assurance reference profile on this page, that authority is human. Reviewers reduce the linear burden of mechanical checks so accountable people can focus on judgment, architecture, and risk.

A reviewer agent is itself a versioned harness component: it has a skill, a rubric, a locally versioned output schema, a golden set, and a release gate. It is treated like any other component of the Improvement Loop—its findings produce candidate proposals that land in the same change-control queue as everything else. ContextOS does not currently publish a canonical reviewer-output schema or exported reviewer type.

Why specialized reviewers, not one giant reviewer

A single “review the change” agent fails in three predictable ways:

1. Concerns get blurred. Security findings get diluted by stylistic noise; cost regressions hide behind feature wins.
2. Severity calibration drifts. A reviewer that sees everything calibrates to the median, missing tail-risk findings.
3. Ownership becomes diffuse. Nobody owns the rubric, so it ages silently.

Specialized reviewers solve all three: each has a single concern, a tightly-scoped rubric, and a named owner. Severity is calibrated against a concern-specific golden set, not a generic one.

The seven reference reviewers

Architecture reviewer

Owns: plane boundaries, dependency direction, primitive layering.

Checks include:

• No business-agent path that bypasses the Tool Gateway to call a database or external system directly; an admitted DB adapter may access its declared backend inside the gateway boundary.
• No cross-domain dependency without an adapter manifest.
• No business action without a decision_id declared in the Decision Catalog.
• No tool added without a schema and capability class.
• No agent response written without a trace_id propagated end-to-end.

Severity floor: any finding that breaks the reference architecture is blocker.

Security reviewer

Owns: sensitive data, secrets, auth, sandbox profile, injection, isolation.

Checks include:

• No plaintext sensitive field in logs or traces (policy_id: no_plaintext_sensitive_data_logging).
• All secrets resolved through the secrets adapter; no inline credentials.
• Tool sandbox profile matches the tool’s side-effect classification.
• Principal chain, tenant, workload identity, and stable subject/evidence refs are preserved across every Tool Gateway call. SIDs are model features and never substitute for audit identity or authority.
• No prompt fragment includes user-controlled content without explicit injection-safe rendering.

Severity floor: any sensitive-data leak is blocker and emits a security event regardless of approval state.

Reliability reviewer

Owns: timeouts, retries, fallbacks, idempotency, rollback paths.

Checks include:

• Every tool call has an explicit timeout; no inherited defaults from a generic HTTP layer.
• Retries are scoped (max attempts, backoff, idempotency key); no unbounded retry loops.
• Every state-changing tool declares idempotency and recovery behavior; reversible and compensatable actions expose the corresponding mechanism.
• Irreversible actions declare that fact explicitly and require the stricter evidence and authority obligations selected by policy.
• Failure-playbook coverage exists for the typed verdicts the change can produce.
• New planner / executor / critic skills declare their loop guard and budget posture.

Severity floor: a state-changing tool that falsely claims reversibility or omits required idempotency/recovery semantics is blocker.

Product reviewer

Owns: user experience, intent fidelity, edge cases.

Checks include:

• Response is clear, minimal, and useful at the user surface.
• Edge cases on the intent (empty result, expired state, partial failure) are explicitly handled.
• User-facing messages do not expose internal system details, internal IDs, or stack-shaped strings.
• Clarification questions are asked only when the harness genuinely lacks the information; otherwise the agent acts on its best evidence and says so.
• Tone, naming, and surface area match the product principles registered in the repo.

Severity floor: leaking internal details to the end user is blocker.

Data reviewer

Owns: event schema, evidence coverage, analytics impact.

Checks include:

• Every DecisionRecord carries the evidence_refs declared by the Decision Spec.
• New event types are registered in the schema registry before they ship.
• Analytics events include the canonical correlation IDs (trace_id, run_id, decision_id) — no event is “for product” only.
• Memory promotions (working → episodic → semantic → durable) carry the consent and contradiction evidence required by the Memory doc.

Severity floor: a DecisionRecord without resolvable evidence_refs is blocker.

Cost reviewer

Owns: tokens, infra cost, retrieval cost, run-budget headroom.

Checks include:

• Token usage per decision stays within the intent’s RunBudget band; new pack versions cannot regress economics by more than the configured guard.
• Tool-call count per decision is bounded; loop guards activate before the budget is exhausted.
• Retrieval breadth (top_k, max_hops) is justified by an evaluator metric, not by intuition.
• Caching: any new prompt fragment that can be reused is part of the cache surface.
• Model selection: the AI Gateway & LLM Router profile is appropriate for the native risk dimensions and evaluation requirements; v1 risk class remains a historical slice.

Severity floor: a regression on economics_cents_per_decision greater than the guardrail is blocker.

Compliance reviewer

Owns: audit, consent, regulated actions, native ActionRisk, policy obligations, and the v1 ApprovalMode projection.

Checks include:

• Every regulated action has the typed obligations and named accountable role required by applicable policy; where approval is required, it has a named gate and approver role.
• Native effect, authority, reversibility, interaction, data scope, and decision TTL match the capability and concrete call. The v1 ApprovalMode fields remain conservative compatibility projections.
• Consent records exist for any action touching regulated data.
• Audit trail covers the full lifecycle: proposal, approval, execution, reversal.
• Regulatory mapping (NIST AI RMF, ISO/IEC 42001, sector-specific frames) is current for any new control.

Severity floor: an action missing a policy-required gate, consent record, authority binding, or native risk declaration is blocker.

Output schema

Every reviewer adopting this reference profile emits the same locally versioned envelope, regardless of concern:

{
  "review_id": "rv_2026_05_08_a17",
  "reviewer_agent": "security-reviewer",
  "reviewer_version": "1.4.0",
  "subject": {
    "kind": "pack_change",
    "pack_id": "ctxpack.support",
    "from_version": "5.1.0",
    "to_version": "5.2.0",
    "diff_ref": "git:repo@a13b9c2..7777eaa"
  },
  "status": "fail",
  "findings": [
    {
      "finding_id": "f_01",
      "severity": "blocker",
      "file": "harness/tools/payment.refund.json",
      "issue": "The compensatable effect declares no compensation evidence in the tool result contract",
      "policy_id": "no_state_change_without_recovery_evidence",
      "recommendation": "Add a protected compensation reference and observable postcondition to the result contract."
    }
  ],
  "evidence_refs": ["dr_2026_05_07_q88"],
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "completed_at": "2026-05-08T09:14:00Z"
}

Within this profile, severity vocabulary is fixed: blocker, must-fix, should-fix, nit. A blocker from any reviewer prevents promotion past the relevant rollout stage; a must-fix requires either the fix or a change-control decision explaining the deferral. That change-control artifact is not the governed business DecisionRecord unless it actually satisfies that schema and decision binding.

Integration with approval gates

Reviewer findings are inputs to the governance approval gate, not a replacement for it.

proposal arrives
    ↓
reviewers fan out in parallel
    ↓
findings aggregate by severity
    ↓
if any blocker: gate denied, proposal re-enters change-control as 'changes_requested'
if any must-fix: gate held; either fix lands or ADR is filed
if only should-fix / nit: gate goes to human approver with summary
    ↓
human approver makes the final call (judgment, risk, scope)
    ↓
if approved: rollout begins at 0%_shadow per the harness rollout stages

This high-assurance profile deliberately reserves the final promotion decision for a human: reviewers may auto-deny a blocker but never auto-approve. Other profiles may use a bounded automated authority only when policy explicitly names it and independent evaluation proves non-vacuity; the proposer still cannot approve itself.

Reviewer skills are first-class harness components

Each reviewer is a versioned skill in harness/reviewers/:

harness/reviewers/
  architecture/
    skill.md               # role, scope, rubric
    policies.json          # JsonLogic checks the reviewer applies
    fixtures/              # known-good and known-bad changes
    golden_set.json        # labeled findings for evaluator scoring
  security/
    skill.md
    policies.json
    fixtures/
    golden_set.json
  ...

The skill follows the same minimal-skill pattern used elsewhere in the harness (Meta-Harness Appendix D calls skill quality a strong lever on search quality): constrain what is forbidden and what artifacts to produce, while leaving the reviewer free to inspect any material inside its declared authorization and data scope.

A reviewer’s golden set is exercised on every reviewer-skill change. False-positive rate, false-negative rate, and severity-calibration drift are tracked the same way evaluator metrics are tracked in Evaluation and Observability.

Failure modes

• Reviewer becomes a style cop. Findings devolve into nits and humans stop reading. Mitigation: severity floor by concern; cap nit-rate per review.
• Reviewer over-blocks. Every change becomes a blocker; throughput collapses. Mitigation: track per-reviewer block rate; rebaseline rubric quarterly.
• Reviewer drifts. New patterns appear in the codebase that the rubric doesn’t cover. Mitigation: feed Improvement-Loop insights of kind gap_detected back into reviewer rubrics.
• Reviewer is bypassed. A team adds an “emergency lane” that skips reviewers. Mitigation: emergency handling is an explicit policy exception with scoped authority, expiry, a security event, and mandatory post-hoc review—not a new approval-mode tier.
• Reviewer hallucinates findings. Mitigation: every finding must cite a policy_id from the policy bundle or a rule from the reviewer’s own rubric; ungrounded findings are dropped at the gate.

Operational concerns

• Reviewers run in parallel; the slowest reviewer sets gate latency.
• Reviewer cost budget is separate from production RunBudget; a reviewer cannot block an emergency rollback by exceeding its budget.
• Reviewer skills are versioned independently; pinning a reviewer to an older skill is allowed for replay but never for live gating.
• Reviewer outputs are append-only and replayable. Deterministic validators must reproduce exactly from pinned inputs; model-based review is compared through required findings, forbidden misses, calibration, and scorecard properties rather than assumed to emit identical prose.

Evaluation metrics

• Block rate per reviewer — fraction of reviews ending in any blocker. Each deployment calibrates an evidence-backed band by concern; movement outside it triggers miscalibration review rather than automatic threshold relaxation.
• False-positive rate — blocker findings overturned at human approval.
• False-negative rate — incidents whose root cause was a finding the reviewer missed.
• Severity calibration drift — distribution of severities over time; sharp shifts trigger rubric review.
• Time-to-finding — median wall-clock from proposal arrival to reviewer verdict.
• Coverage — fraction of diffs that exercise at least one rule in the reviewer’s rubric.

Example: a security reviewer catching a regression

A pack diff adds a new tool adp_support.fetch_thread. The Architecture reviewer passes; the Security reviewer fails:

{
  "review_id": "rv_2026_05_08_b04",
  "reviewer_agent": "security-reviewer",
  "reviewer_version": "1.4.0",
  "subject": {
    "kind": "tool_addition",
    "tool_name": "adp_support.fetch_thread",
    "diff_ref": "git:repo@7777eaa..b21c5d4"
  },
  "status": "fail",
  "findings": [
    {
      "finding_id": "f_01",
      "severity": "blocker",
      "file": "harness/tools/support.fetch_thread.json",
      "issue": "Output schema includes raw user_email and phone_e164 with no redaction profile bound",
      "policy_id": "no_plaintext_sensitive_data_logging",
      "recommendation": "Bind redaction_profile=customer_pii_v3 on output_schema; or move fields behind a delegated read."
    },
    {
      "finding_id": "f_02",
      "severity": "must-fix",
      "file": "harness/tools/support.fetch_thread.json",
      "issue": "capability_class=observe but tool returns identity-mapped fields; should be capability_class=recall with consent evidence",
      "policy_id": "capability_class_matches_side_effect",
      "recommendation": "Reclassify as recall; require consent_record_ref in the input_schema."
    }
  ]
}

The gate denies promotion; the Improvement Loop captures the pattern as gap_detected so the Security reviewer’s rubric can be extended; an ADR is filed if the team wants to defer the must-fix.

See also

• Harness Engineering — the discipline that this primitive composes into
• Governance — native action risk, v1 approval-mode compatibility, and policy bundles
• Improvement Loop — how reviewer findings feed change control
• Evaluation and Observability — how reviewers are themselves evaluated