AI Gateway and LLM Router

The AI Gateway is the model-side boundary for ContextOS. It sits between the Cognitive Core and model providers. It normalizes model invocation, applies policy, enforces budget, records telemetry, and delegates model selection to the LLM Router.

It is not the Tool Gateway. The Tool Gateway controls side effects. The AI Gateway controls model calls.

Design stance

The gateway exists because production systems should not scatter provider-specific model calls throughout planners, evaluators, adapters, and batch jobs. Provider APIs evolve. Model catalogs change. Pricing, context windows, tool support, data residency, and safety features vary. The rest of the runtime should depend on one ContextOS contract, not on every provider contract directly.

The router is not magic. It is a governed selection function over explicit inputs: task shape, risk, budget, latency, region, required capabilities, provider health, and evaluator feedback.

This is an integration reference design, not an implemented production gateway. Model profiles here are routing configurations, distinct from the exact versioned control profiles. The Execution Contract adds admission, current authority, data-flow and aggregate-budget obligations around every model call, including evaluator, repair and fallback calls.

Responsibilities

Placement

The important boundary is authority: a model provider can produce text, structured output, or a model-native tool-call proposal. It cannot execute ContextOS tools directly. Any external effect still returns to the Decision plane and goes through the Tool Gateway.

If a provider offers hosted tools, computer use, code execution or persistent state, those are actual enabled workflow features and separately governed effects/storage boundaries. Either disable them for this model-only route or admit and enforce their required authority, containment, data and operation controls. A provider’s tool schema or advertised sandbox does not prove those controls.

Northbound contract

The Cognitive Core calls the gateway with a provider-neutral envelope.

{
  "request_id": "req_01j9",
  "run_id": "run_88",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "tenant_id": "tenant_acme_prod",
  "intent_id": "support.refund",
  "action_risk": {
    "effect": "external_state",
    "authority": "user_delegated",
    "reversibility": "compensatable",
    "interaction": "api",
    "data_scope": "INTERNAL",
    "decision_ttl_seconds": 900
  },
  "risk_class": "destructive",
  "operation": "judge.plan",
  "compiled_context_ref": "compiled_ctx_41",
  "input": {
    "instructions": "Produce a plan that can be verified by the Critic.",
    "messages": [
      {
        "role": "user",
        "content": "Refund order ord_881"
      }
    ]
  },
  "requirements": {
    "structured_output": true,
    "tool_calling": false,
    "vision": false,
    "max_input_tokens": 24000,
    "max_output_tokens": 2000,
    "latency_slo_ms": 2500,
    "max_cost_usd": 0.08,
    "data_residency": "us"
  },
  "routing_hints": {
    "quality_tier": "standard",
    "fallback_allowed": true,
    "canary_allowed": false
  }
}

action_risk is the native routing and policy input. risk_class is the required v1 compatibility projection and remains useful for old clients and historical scorecards; it must not replace the independent native dimensions. The envelope on this page is a reference-design contract, not a published schema.

The response carries both model output and the route decision that produced it.

{
  "request_id": "req_01j9",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "status": "ok",
  "output": {
    "type": "json",
    "value": {
      "plan_id": "plan_refund_01",
      "steps": []
    }
  },
  "usage": {
    "input_tokens": 18340,
    "output_tokens": 612,
    "estimated_cost_usd": 0.041
  },
  "route": {
    "routing_decision_id": "route_01j9",
    "model_profile_id": "profile_reasoning_standard_v7",
    "provider_adapter": "provider_a",
    "routing_rule_ids": ["ROUTE_STANDARD_REASONING_US"],
    "fallback_index": 0,
    "explanation": "Selected profile supports structured output within residency, latency, and budget constraints."
  }
}

Model profile registry

The router selects from model profiles, not hard-coded model names. A profile can map to one or more provider deployments behind the adapter layer.

{
  "model_profile_id": "profile_reasoning_standard_v7",
  "provider_adapter": "provider_a",
  "deployment_ref": "secret://model-deployments/provider-a/reasoning-standard",
  "capabilities": {
    "structured_output": true,
    "tool_calling": true,
    "vision": false,
    "long_context": true,
    "streaming": true
  },
  "limits": {
    "max_input_tokens": 128000,
    "max_output_tokens": 8192,
    "rpm": 300,
    "tpm": 300000
  },
  "policy": {
    "regions": ["us", "eu"],
    "data_classes_allowed": ["PUBLIC", "INTERNAL"],
    "stores_provider_state": false,
    "action_risk": {
      "allowed_effects": ["none", "local_state", "external_state"],
      "allowed_authorities": ["agent", "service", "user_delegated", "human_approved"],
      "allowed_reversibility": ["read_only", "reversible", "compensatable"],
      "allowed_interactions": ["api"],
      "data_scope_ceiling": "INTERNAL"
    },
    "eligible_risk_classes": ["read_only", "local_write", "network", "delegated", "destructive"]
  },
  "score_hints": {
    "quality": 0.86,
    "latency_p95_ms": 1900,
    "cost_per_1k_input_usd": 0.002,
    "cost_per_1k_output_usd": 0.008
  },
  "status": "healthy"
}

Routing algorithm

The router runs in five deterministic stages.

1. Filter candidates by native action-risk dimensions, data residency, data classification, required capabilities, active policy, health, quota, and context size. V1-only clients supply the conservative risk_class projection.
2. Score remaining candidates against quality, latency, cost, reliability, and evaluator feedback for the active intent.
3. Select the highest-ranked profile with deterministic tie-breaks: policy priority, quality band, lower p95 latency, lower estimated cost, stable provider priority.
4. Invoke through the selected provider adapter with timeout, retry, and streaming settings from the Run Context.
5. Record route decision, usage, provider response model, fallback index, applied rules, and trace span.

This makes routing explainable. If the router chooses a cheaper model, the decision record must show that the cheaper profile remained inside the quality band for the current task.

Evaluate against explicit pinned observations for health, quotas, policy and effective time. No ambient-clock or random tie-break belongs in deterministic reference evaluation; declare any production canary cohort as a stable input. Validate policy expressions and report errors visibly rather than selecting a permissive default route. Scores and estimated prices are illustrative policy inputs, not universal quality or current-price guarantees.

Routing policy

Routing policy is a Trust-plane config artifact. It should be versioned, evaluated in CI, and pinned per environment. Signatures and publisher verification are required only when the deployment adopts the supply-chain profile.

policy_id: route.support.standard.v4
owner_role: platform_runtime
default_profile: profile_general_standard_v5
rules:
  - rule_id: ROUTE_HIGH_RISK_STRUCTURED
    priority: 100
    when:
      action_risk:
        effect: external_state
        authority: user_delegated
        reversibility: compensatable
        interaction: api
        data_scope_ceiling: INTERNAL
      structured_output: true
    require:
      data_residency: request
      capabilities:
        structured_output: true
      max_fallbacks: 1
    score:
      quality: 0.55
      reliability: 0.20
      latency: 0.15
      cost: 0.10
    candidates:
      - profile_reasoning_standard_v7
      - profile_reasoning_premium_v3
  - rule_id: ROUTE_LOW_RISK_FAST
    priority: 40
    when:
      risk_class: read_only
    score:
      latency: 0.45
      cost: 0.35
      quality: 0.20
    candidates:
      - profile_general_fast_v9
      - profile_general_standard_v5

Fallback rules

Fallback is allowed only when it preserves the contract requested by the caller.

Fallback must never broaden data exposure, remove required structured output, ignore native action risk (or the conservative v1 projection), or bypass policy.

Before each attempt, reserve remaining aggregate token/call/cost capacity and check the inherited deadline. Retries, repair prompts, evaluators and provider fallback consume the same root budget; they cannot receive fresh limits. Distinguish a model-only response retry from a provider operation that may have stored state or executed hosted tools. Unknown effects require the stable operation ledger and verified provider-specific recovery evidence; a transport timeout is not proof of absence. Settle verified cumulative usage, retain uncertain charges/reservations and convert decimal estimates to the declared accounting unit exactly. A cheaper route cannot demote required controls or approved data processors.

Streaming contract

The gateway may expose streaming, but the stream is still a typed envelope.

Streaming consumers should not treat raw provider chunks as final business state. The Decision plane consumes the normalized final output.

Observability

Every gateway invocation emits OpenTelemetry spans and metrics tied to the parent trace_id.

Recommended span structure:

contextos.ai_gateway.invoke
  contextos.llm_router.select
  contextos.provider_adapter.invoke

Required attributes:

Use OpenTelemetry GenAI semantic conventions where they fit, but pin the emitted convention version because those conventions are still evolving.

Telemetry, prompt/completion capture and caches are governed data-flow destinations. Minimize fields, verify actual readers/processors and residency, and apply retention/correction/deletion obligations to derived copies. A profile’s regions or stores_provider_state: false declaration must be checked against the actual provider deployment, endpoint, configuration and current evidence; it does not establish residency or zero retention on its own.

Routing lineage candidate profile

The gateway must persist routing lineage, and the final DecisionRecord can reference it through lineage.routing_decision_id. The illustrative RoutingDecision shape below is a candidate integration profile: ContextOS does not yet publish a canonical RoutingDecision schema or exported type. Promotion requires working-system validation, conformance fixtures, and an explicit migration path.

{
  "routing_decision_id": "route_01j9",
  "request_id": "req_01j9",
  "run_id": "run_88",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "policy_version": "route.support.standard.v4",
  "selected_profile": "profile_reasoning_standard_v7",
  "candidate_profiles": [
    "profile_reasoning_standard_v7",
    "profile_reasoning_premium_v3"
  ],
  "rejected_profiles": [
    {
      "model_profile_id": "profile_general_fast_v9",
      "reason": "missing_structured_output"
    }
  ],
  "usage": {
    "input_tokens": 18340,
    "output_tokens": 612,
    "estimated_cost_usd": 0.041
  },
  "fallback_index": 0,
  "created_at": "2026-05-09T10:30:00Z"
}

Security controls

The AI Gateway should enforce these controls before any provider call:

• Redact secrets and disallowed data classes from provider-bound payloads.
• Deny requests whose data residency cannot be satisfied by an eligible profile.
• Keep provider credentials inside the gateway boundary.
• Disable provider-side state unless the active profile and tenant policy explicitly permit it.
• Store prompts, completions, and chunks according to classification and retention policy.
• Refuse prompt-response caching for sensitive, tenant-private, or user-private payloads unless policy explicitly permits it.
• Treat provider-native tool calls as proposals, not executed actions.
• Recheck current authority and actual inference/telemetry/cache routes on every fallback; require verified source-label and approved transformation evidence rather than trusting requested residency or a redaction flag.

Deployment model

Evaluation loop

Routing quality should be evaluated like any other ContextOS implementation behavior.

Routing policy changes should pass replay before promotion. Canary routing is acceptable only when the Trust plane can compare outcomes against the pinned baseline.

Acceptance checklist

Before shipping an AI Gateway or router into production:

• A single provider-neutral invocation envelope exists.
• Model profiles are versioned and separate from provider names in business logic.
• Routing policy is versioned, pinned, and environment-scoped; it is signed when the supply-chain profile requires signatures.
• Route decisions are recorded and joinable to Decision Records.
• Provider-side state is explicit and policy-controlled.
• Structured output validation failures are typed.
• Fallback cannot cross residency, data classification, or capability boundaries.
• OTEL spans and token/cost metrics are emitted for every call.
• Golden replay covers route changes before promotion.
• Current runtime evidence demonstrates denied routes, budget exhaustion, provider outages, unknown-state recovery and stop behavior for the exact workflow/release/environment; a routing scorecard or reference test pass alone is insufficient.

Research basis

This design follows patterns from primary docs and keeps provider-specific details behind adapters:

• OpenAI Responses API migration guide: typed response items, function calling, structured outputs, state, and tool-capable model interactions.
• Amazon Bedrock intelligent prompt routing: routing for quality and cost, fallback models, route criteria, and traceable responses.
• Microsoft Foundry model router: deployable model-router pattern, routing modes, model subsets, data-zone boundaries, and versioned underlying model sets.
• OpenTelemetry GenAI semantic conventions: span, metric, and event naming guidance for generative AI systems.

Related docs

• Cognitive Core
• Agentic Context Engineering
• Evaluation and Observability
• Reference Architecture