ContextOS Metrics Glossary
Unified metric glossary across the five planes — Intelligence, Context, Decision, Action, Trust.
This page is the stable metrics contract for ContextOS. It defines the names, dimensions, source artifacts, owners, and minimum scorecard that every implementation should keep consistent across the five planes.
This is a documented instrumentation protocol, not a claim that this repository emits production metrics. Teams can add local signals, but release gates, incidents, and executive scorecards should declare formulas, source artifacts and evidence scope consistently.
Naming Conventions
Metric names use one namespace and one shape:
contextos.<plane>.<component>.<signal>Cross-cutting run and aggregate accounting signals use the existing contextos.run.* and contextos.budget.* namespaces. A metric owner is responsible for trustworthy measurement; naming does not authenticate an observation.
Examples:
| Metric | Meaning |
|---|---|
contextos.decision.gateway.request_duration_ms | AI Gateway request latency. |
contextos.context.pack.evidence_coverage_rate | Share of required evidence represented in the compiled Context Pack. |
contextos.decision.plan.validity_rate | Share of generated plans passing structural validation. |
contextos.action.tool.success_rate | Share of attempted calls with validated required outputs and verified required effects/postconditions; a transport/tool status alone is insufficient. |
contextos.trust.trace.completeness_rate | Share of runs with the required trace and audit artifacts. |
Rules:
- Use lowercase snake case for components and signals.
- Use
_duration_msfor latency and wall-clock duration. - Use
_ratefor fractions,_countfor event counts,_totalfor monotonic counters, and_cost_usdor_cost_inrfor cost. - Record latency as distributions with at least p50, p95, and p99 rollups.
- Keep model names, provider names, tool IDs, and policy IDs in dimensions, not in metric names.
- Do not encode raw user text, prompts, document titles, or error messages in metric names or labels.
Required Dimensions
Every metric must be joinable to the run trace. High-cardinality IDs such as run_id and trace_id belong on spans, events, exemplars, and rollup records; only attach them as time-series labels when the backend is designed for that cardinality.
| Dimension | Required on | Purpose |
|---|---|---|
tenant_id | span, event, rollup | Tenant-level slicing and data isolation. |
environment | span, event, metric | dev, staging, prod, or equivalent. |
release_version | span, event, metric | Runtime or service release attribution. |
plane | span, event, metric | One of intelligence, context, decision, action, trust. |
component | span, event, metric | Gateway, compiler, planner, tool manager, evaluator, or observability component. |
component_version | span, event, rollup | Localizes regressions to a deployed component version. |
workflow_id | span, event, rollup | Workflow or journey-level rollup. |
intent_id | span, event, rollup | Intent-level quality, safety, and cost comparison. |
task_type | span, event, rollup | Stable task taxonomy used by scorecards. |
risk_class | span, event, rollup | V1 compatibility and historical safety tier; do not use as the only native risk dimension. |
run_id | span, event, exemplar | Joins artifacts for a single run. |
trace_id | span, event, exemplar | Joins telemetry with the trace bundle. |
Use the following dimensions when they apply:
| Dimension | Applies to |
|---|---|
model_profile_id, provider, routing_policy_id | AI Gateway and LLM Router calls. |
pack_version, context_pack_id, evidence_source_type | Context Pack compilation and evidence metrics. |
memory_tier, knowledge_snapshot_id | Memory and knowledge-substrate metrics. |
tool_id, adapter_id, native action_effect, action_authority, action_reversibility, action_interaction, action_data_scope, and v1 approval_mode_declared / approval_mode_effective | Tool Manager and adapter calls. |
policy_profile, policy_decision_id | Policy and approval metrics. |
evaluator_id, golden_set_id, replay_dataset_id | Evaluation and Observability metrics. |
channel, locale, user_cohort | User-facing adoption and UX metrics. |
exact profile/control refs, contract_id, operation ID and ownership generation | Admission, authority, effects and recovery observations; keep high-cardinality identifiers on events/artifacts rather than unbounded time-series labels. |
Minimum Scorecard
Every production scorecard should expose these metrics by tenant_id, intent_id, release_version, and pack_version, plus applicable native risk dimensions. Keep risk_class for v1 and historical comparisons.
| Metric | Definition | Owner | Direction |
|---|---|---|---|
contextos.decision.task.verified_success_rate | Tasks that pass the configured verifier divided by tasks started. | Decision plane | Up |
contextos.decision.task.safe_completion_rate | Tasks completed without policy or approval violations divided by tasks started. | Trust plane | Up |
contextos.run.end_to_end_duration_ms | Time from accepted user or system request to terminal run state. | Platform runtime | Down |
contextos.run.first_useful_response_duration_ms | Time from request acceptance to first useful response or action proposal. | Product + runtime | Down |
contextos.budget.cost_per_verified_success | Total inference, retrieval, tool, retry, evaluation, observability, infrastructure, and human-review cost divided by accepted grounded, policy-compliant outcomes in the rollup window. This is the canonical “cost per trusted outcome” metric. | Platform + product | Down |
contextos.context.answer.evidence_backed_rate | Responses requiring evidence that include valid evidence_refs. | Context plane | Up |
contextos.action.tool.success_rate | Successful tool results divided by attempted tool calls. | Action plane | Up |
contextos.trust.policy.violation_rate | Admitted runs with verified policy violations divided by all admitted runs in the declared start cohort, including failed/cancelled runs; report pending/unknown coverage separately. | Trust plane | Down |
contextos.trust.trace.completeness_rate | Runs with required spans, scorecard, evidence, and audit links. | Observability | Up |
contextos.trust.replay.conformance_rate | Replay runs whose deterministic artifacts match exactly and whose probabilistic stages satisfy declared properties or scorecards. | Evaluation | Up |
Thresholds are environment- and intent-specific. The contract defines formulas and owners; each deployment defines alert thresholds, sampling policy, and release gates.
Report unknown effects, incomplete metering and missing verification separately; never count them as successful or drop their cost from a successful-outcome numerator. Preserve exact declared currencies/accounting quantum and the legacy fractional-cent adapter semantics. A passing scorecard does not establish profile conformance or permit averaging away a mandatory denied case.
Per-Plane Metrics
Model-side metrics — Decision plane
The AI Gateway and LLM Router sit at the Decision plane’s model boundary. Earlier metric names placed these under contextos.intelligence.*, conflicting with the five-plane ownership. Migrate the gateway, provider, router, output and tokens families to contextos.decision.* with an explicit collector/dashboard alias window; do not combine both names as separate work in cost totals.
| Area | Contract metrics |
|---|---|
| Availability | contextos.decision.gateway.availability_rate, contextos.decision.provider.error_rate, contextos.decision.provider.timeout_rate |
| Latency | contextos.decision.gateway.request_duration_ms, contextos.decision.router.decision_duration_ms, contextos.decision.provider.request_duration_ms |
| Routing | contextos.decision.router.fallback_rate, contextos.decision.router.model_switch_rate, contextos.decision.router.policy_rejection_count |
| Quality controls | contextos.decision.output.invalid_schema_rate, contextos.decision.output.refusal_rate, contextos.decision.output.repair_rate |
| Budget | contextos.decision.tokens.input_total, contextos.decision.tokens.output_total, contextos.decision.gateway.cost_usd, contextos.decision.gateway.cache_hit_rate |
Context Plane
The Context plane owns per-request compilation, admitted evidence and conflict/sufficiency diagnostics. The Intelligence plane owns retrieval, memory and knowledge snapshots. Keep these ownership boundaries when instrumenting Context Pack, Memory and Knowledge Graph services. The retrieval/memory prefixes below replace historical contextos.context.retrieval.* / contextos.context.memory.* with an explicit alias migration.
| Area | Contract metrics |
|---|---|
| Pack build | contextos.context.pack.build_duration_ms, contextos.context.pack.token_count, contextos.context.pack.context_window_utilization_rate |
| Evidence | contextos.context.pack.evidence_coverage_rate, contextos.context.answer.evidence_backed_rate, contextos.context.claim.attribution_rate |
| Intelligence retrieval | contextos.intelligence.retrieval.precision_at_k, contextos.intelligence.retrieval.recall_at_k, contextos.intelligence.retrieval.stale_source_rate |
| Intelligence memory | contextos.intelligence.memory.promotion_accept_rate, contextos.intelligence.memory.correction_rate, contextos.intelligence.memory.stale_read_rate |
| Context hazards | contextos.context.noise.irrelevant_token_rate, contextos.context.conflict.detected_rate, contextos.context.conflict.resolved_rate, contextos.context.poisoning.suspected_rate |
Decision Plane
The Decision plane owns planning, execution choice, critique, loop controls, and the Decision Record. See also the Decision Catalog.
| Area | Contract metrics |
|---|---|
| Planning | contextos.decision.plan.validity_rate, contextos.decision.plan.feasibility_rate, contextos.decision.plan.revision_count |
| Execution control | contextos.decision.executor.step_success_rate, contextos.decision.executor.loop_guard_trigger_rate, contextos.decision.executor.escalation_rate |
| Critique | contextos.decision.critic.veto_rate, contextos.decision.critic.repair_success_rate, contextos.decision.critic.false_pass_rate |
| Outcomes | contextos.decision.task.success_rate, contextos.decision.task.verified_success_rate, contextos.decision.task.abandonment_rate |
| Records | contextos.decision.record.completeness_rate, contextos.decision.record.evidence_ref_count, contextos.decision.record.policy_ref_count |
Action Plane
The Action plane owns side effects through the Tool Manager and the Adapter Mesh.
| Area | Contract metrics |
|---|---|
| Tool calls | contextos.action.tool.success_rate, contextos.action.tool.error_rate, contextos.action.tool.request_duration_ms, contextos.action.tool.retry_rate |
| Approval binding | contextos.action.approval.required_rate, contextos.action.approval.honored_rate, contextos.action.approval.denied_rate |
| Native risk | contextos.action.risk.declaration_coverage_rate, contextos.action.risk.policy_denial_rate, contextos.action.risk.v1_projection_mismatch_rate |
| Idempotency | contextos.action.idempotency.replay_hit_rate, contextos.action.idempotency.duplicate_effect_rate |
| Adapter health | contextos.action.adapter.availability_rate, contextos.action.adapter.schema_validation_error_rate, contextos.action.adapter.version_drift_count |
| Evidence return | contextos.action.tool.evidence_return_rate, contextos.action.tool.audit_link_rate |
Trust Plane
The Trust plane owns policy, approval gates, evaluation, observability, audit, replay, and security posture. See Evaluation and Observability, Observability, and the Policy Engine.
| Area | Contract metrics |
|---|---|
| Policy | contextos.trust.policy.violation_rate, contextos.trust.policy.must_refuse_coverage_rate, contextos.trust.policy.decision_duration_ms |
| Evaluation | contextos.trust.scorecard.coverage_rate, contextos.trust.eval.pass_rate, contextos.trust.eval.regression_rate, contextos.trust.eval.judge_agreement_rate |
| Observability | contextos.trust.trace.completeness_rate, contextos.trust.audit.completeness_rate, contextos.trust.trace.fetch_duration_ms |
| Replay | contextos.trust.replay.conformance_rate, contextos.trust.replay.deterministic_match_rate, contextos.trust.replay.property_pass_rate, contextos.trust.replay.dataset_coverage_rate, contextos.trust.replay.duration_ms |
| Security | contextos.trust.security.event_count, contextos.trust.security.cross_tenant_denial_count, contextos.trust.security.redaction_failure_rate |
| Adoption | contextos.trust.user_correction_rate, contextos.trust.operator_override_rate, contextos.trust.human_escalation_rate |
Emitted Artifacts
Metrics are only useful when their source artifacts are stable. Each run should emit the artifacts needed for its path; for example, a read-only answer may not emit an approval decision, but any governed action must. Names matching published schemas or exports below are canonical. RoutingDecision, PolicyDecision, and TraceBundle remain locally versioned candidate/profile artifacts until ContextOS publishes corresponding schemas.
| Contract or profile artifact | Required contents | Primary owner |
|---|---|---|
CompiledContext | prompt/provenance, policy/tool/evidence manifests, runtime admission controls, budget/omission report and context ledger/hash in their declared fields | Context plane |
Candidate RoutingDecision profile | model profile, provider adapter, routing policy, rejected candidates summary, fallback index, usage, estimated cost; locally versioned profile | Decision plane model boundary |
Plan | plan ID, intent, steps/dependencies and decision checkpoints; separate declared runtime observations hold revisions and loop-guard state | Decision plane |
DecisionRecord | typed decision, evidence/policy/approval/control/budget/lineage/trace fields; link separate verifier/effect evidence where applicable | Decision plane |
ToolCallEnvelope / ToolResultEnvelope | tool ID, adapter ID, schema refs, native risk when available, v1 approval projection, idempotency key, result status, evidence refs | Action plane |
Candidate PolicyDecision profile | policy profile, decision ID, rule refs, gate status, approver ref when applicable | Trust plane |
ContextLedger / EvidenceConflictMarker | source refs, omissions, compiled hash, competing evidence, affected requirements, and resolution ref | Context + Trust planes |
MemoryProposal | proposed content, target tier, evidence refs, consent and contradiction results; flags do not self-authenticate consent | Intelligence plane |
RunScore / DecisionScorecard | evaluator dimension scores, thresholds or acceptance state, and release-gate evidence | Evaluation |
Candidate TraceBundle profile | W3C trace context, plane span chain, artifact refs, audit refs, sampling reason | Observability |
ReplayPacket | pinned request/run context, pack and compiled-context refs, tool transcripts, side-effect policy, expected outcome, and replay results | Evaluation |
Execution-control signals
Measure these obligations from current verified runtime/store observations under the exact execution contract; the reference does not emit these deployment signals for you.
| Signal family | Required interpretation | Owner |
|---|---|---|
| Admission and policy errors | Unsupported exact required control/feature and applicable expression failures; no demotion or silent false result. | Admission / policy |
| Authority and approval | Stale/revoked/unknown grants, binding/quorum/expiry failures and attempted duplicate redemption; a denial is correct enforcement, not an outage to optimize away. | Brokers / approval store |
| Boundary and data | Required active-control failures and denied actual stage/recipient/processor routes; source labels and configurations need verification. | Sandbox/egress/data owners |
| Effects and recovery | Confirmed/partial/unknown operations, uncertainty age, reconciliation backlog, cancellation propagation and stale-generation dispatch denials. | Durable controller / provider adapter |
| Aggregate accounting | Original limits, usage and outstanding reservations across root/child/retry/evaluator work; zero accepted conservation violations and visible uncertain usage. | Atomic ledger / metering |
| Memory propagation | Pending/failed/external-unknown correction/deletion targets and verified completion against the scoped complete inventory. | Memory / data owner |
| Conformance evidence | All mandatory useful/denied cases, trusted current evidence scope and insufficiencies; reference pass counts remain separate from runtime claims. | Independent evaluator / release owner |
Apply minimization, authorized readers, actual telemetry processor/residency and retention policy to dimensions, rollups and exemplars. A hash or reference can still reveal a sensitive relationship.
Owners
Every contract metric needs a named owner before it is used in release gates or incident review.
| Owner | Responsibilities |
|---|---|
| Platform runtime | Gateway latency, routing, cost, token accounting, run-level duration, service availability. |
| Context engineering | Context Pack quality, retrieval, memory promotion, evidence attribution, conflict and poisoning signals. |
| Decision engineering | Plan validity, execution control, verifier outcomes, Decision Record completeness. |
| Action platform | Tool Manager reliability, native-risk enforcement, v1 projection correctness, adapter health, idempotency, and tool evidence return. |
| Trust, security, and SRE | Policy decisions, audit, trace completeness, replay, security events, redaction, incident scorecards. |
| Product or domain owner | Intent taxonomy, done criteria, golden sets, acceptable thresholds, user correction interpretation. |
Owner duties:
- Maintain the formula, numerator, denominator, unit, and rollup windows.
- Declare the source artifact and required dimensions.
- Define alert thresholds and release-gate thresholds per environment.
- Review metric behavior after any schema, policy, model, tool, or Context Pack version change.
Link Map
- AI Gateway and LLM Router: model routing, provider calls, budgets, and token metrics.
- Tool Manager: tool-call success, approval binding, adapter health, and idempotency metrics.
- Evaluation and Observability: scorecards, replay, release gates, evaluator dimensions.
- Observability: trace bundles, audit records, sampling, and replay substrate.
- Policy Engine: policy decisions and approval gates.