Evaluation Engine

Reference Architecture

The Evaluation Engine is the scoring and release-gate reference component of the Trust plane. It computes per-run scorecards, runs applicable replay suites on each runtime-affecting change, and emits locally versioned release-gate verdicts. RunScore and ReplayPacket are canonical; GoldenSet and ReleaseVerdict are documented profile artifacts, not exported types.

Scorecards and release verdicts are distinct from ReferenceValidationReport and ConformanceReport. Deterministic reference checks can pass while actual runtime controls remain not_evaluated. Conformance requires the exact workflow/implementation/release/environment, all mandatory useful/denied cases and independently verified current runtime evidence. A high average score cannot hide a failed mandatory control, missing evidence or an unacceptable effect.

Definition

A coordinated set of evaluators—deterministic rules, model-based judges, and cost/latency aggregators—running exhaustively in CI and under an explicit sampling policy in production. Outputs are typed RunScore records bound to a trace_id and a Decision Record.

Why it exists

LLM-driven systems drift. Without continuous, structured evaluation, regressions ship undetected. The Evaluation Engine is the gate that catches them — both online (sampled production) and offline (golden-set replays for every change to the prompt, pack, model, or policy).

Evaluators

scorecard deltas are tracked per intent, per tenant, and per pack version so regressions are localizable.

Inputs

• Trace bundles from the Observability component
• Tool transcripts and evidence manifests
• Operator corrections and approval-gate decisions
• Golden sets per intent
• Release-gate thresholds per eval_targets entry from the active pack

Outputs

• Per-run RunScore record bound to trace_id and record_id
• Canonical ReplayPacket values grouped into locally versioned evaluation datasets
• Candidate-profile release-gate verdict (pass / block) with deltas
• Candidate improvement proposals (consumed by Insight Synthesizer / Strategy Compiler / Autotune)

How it works

1. Score — runs selected by the declared production sampling policy get a scorecard record; applicable CI fixtures are evaluated exhaustively.
2. Replay — golden runs use pinned packs, snapshots, and transcripts; deterministic artifacts compare exactly while model-dependent stages use declared properties, scorecards, or tolerances.
3. Gate — compare against the pinned baseline and explicit hard floors; block mandatory-control failures, unmet evidence obligations or unacceptable regressions. Utility, cost and latency cannot trade away required authority, containment, privacy or safety controls.
4. Propose — failed and corrected runs become candidate proposals routed through the documented improvement profile.

Failure modes

• Sampling not stratified by risk / tenant — blinds rare-but-high-risk regressions.
• LLM-as-judge drift between judge model versions — mitigated by pinning judge model + rubric and re-baselining on judge upgrade.
• Replay mismatch from an unpinned dependency or undeclared probabilistic comparison — fail visibly and block the release gate.
• Trace gaps when a custom adapter forgets W3C propagation — caught by trace-coverage assertion.
• Synthetic or stale evidence represented as live control proof — report insufficient evidence rather than conformance.
• Provider acknowledgement missing or compensation partial — keep effect uncertainty visible; a positive model judgment cannot establish completion or authorize blind retry.

Operational concerns

• Sampling stratified by intent, native risk dimensions, v1 compatibility tier, and tenant.
• Tail-based sampling forced for policy gates, material external effects, security events, or failed thresholds.
• Online evaluator/repair/model work consumes the original run’s aggregate reservations and deadline. A separate offline evaluation job has its own explicitly admitted root budget and data authority; it cannot replenish the production run.
• Quarterly rebaselining of golden sets and judge rubrics.
• Govern actual evaluator/reviewer inference, telemetry and exported artifacts as data flows with current readers/processors and retention. Pin model/configuration, rubric, inputs and observed comparison policy; disclose unpinnable external dependencies.
• Aggregate cost includes failed, retried, repaired, fallback and evaluator work over accepted decisions; retain uncertain usage rather than dropping failures from the numerator.

Evaluation metrics

• Coverage (fraction of production runs with full scorecard).
• Detection lead time (regression introduced → release-gate block).
• Deterministic-artifact exact-match rate and probabilistic property/scorecard pass rate.
• Improvement adoption rate (proposals that ship after review).