Evaluation
28 essays tagged with Evaluation.
Proof-Carrying Context: Why AI Agents Need More Than a Context Window
A research-grounded revision of context engineering: from relevant tokens to a replayable view of evidence, conflicts, omissions, and decision sufficiency.
Production Prompt Engineering in 2026: From Instructions to an Evaluated Contract
A deep production guide to prompt engineering: lean instruction design, stable cacheable prefixes, tool and output contracts, reasoning budgets, datasets, robustness tests, versioning, rollout gates, and a 90-day implementation roadmap.
Persistent Memory Poisoning: The Attack That Outlives the Session
A production security architecture for attributable, revalidated, authority-bounded, traceable, and selectively reversible agent memory.
Multi-Agent Consensus Is Not Correctness: How Debate Manufactures Confidence
Why agent agreement can hide correlated error—and how to build governed deliberation with effective agent count, independent verification, and release gates.
Agent Evaluation After the Leaderboard: The KDD 2026 Production Assurance Playbook
KDD 2026 shows why agent evaluation must cover protocols, trajectories, external state, evaluator operating profiles, causal rollout, and drift.
OpenAI's Latest Science Work: From Benchmarks to Verifiable Discovery
What OpenAI's newest mathematics, scientific-computing, biology, and evaluation work says about AI as a research collaborator.
Graph Engineering for AI Agents: The X Hype Cycle and a Production Playbook
Separate graph-engineering signal from X hype, then compile agent workflows with governed transitions, graph invariants, durable effects, and exact replay.
Reverse Prompting: Work Backward from the Output, Then Prove the Prompt
Reverse prompting is useful when it turns a desired output into a candidate instruction contract and validates that contract forward. Here is the research, the workflow, the failure modes, and the security boundary.
Human Oversight for Agent Fleets: Confidence Is Not an Audit Policy
A new audit-allocation paper shows that self-reported confidence can make limited human review worse than random and that tiny audit budgets can become rubber-stamping. Production oversight needs risk gates, stratified random coverage, correlation-aware learning, and a measured non-vacuity test.
Passive Awareness Is the Missing Multi-Agent Primitive—But It Needs a Control Plane
AgentRadio shows that agents improve when discoveries can cross lane boundaries during execution. Production teams should keep the asynchronous channel, then add typed evidence, bounded delivery, authority isolation, and replay.
Adaptive Agent Harnesses: Learn From Experience Without Letting Production Rewrite Itself
MemoHarness shows that execution experience can improve the control layer around an LLM. Production systems need a stricter pattern: adaptive performance inside an immutable safety envelope.
Agent Memory Is a Systems Workload: What SelfMem Changes—and What It Does Not
SelfMem improves long-horizon recall by letting an agent optimize its memory strategy. New systems and security research shows the production contract must also cover cost, freshness, provenance, and poisoning.
Your Agent Benchmark Is Measuring the Harness
Two new evaluation systems separate benchmark, harness, model, and environment. Their results show why one leaderboard score cannot tell you which agent to ship.
Multi-Agent Systems Need Replay Before More Agents
Multi-agent systems multiply coordination, cost, and causal ambiguity. Prove one agent is replayable before adding lanes and handoffs.
Red-Team Agent Hijacking: Build a Security Eval Gate for Repeat Attacks
A practical agent-hijacking evaluation harness: scenario design, adaptive and repeated attempts, path-aware metrics, deterministic release gates, and production replay.
Agent Harness: An Architectural Framework for Production AI Agents
A whitepaper on typed contracts, policy gates, traces, verification loops, and release control for production AI agents.
ContextOS: A Research-Grounded Architecture for Governed Agent Runtimes
A research-grounded framing of ContextOS as a governed runtime for context, tools, memory, security, evaluation, replay, and optimization.
Harness Improvement Loops Need Replayable Environments
Why harness improvement needs replayable episodes, bounded mutations, scorecards, source closure, and promotion gates.
How to Judge AI Work: Scorecards, Not Vibes
A practical guide for business teams evaluating AI agents with scorecards, examples, traces, human corrections, and launch gates instead of demos and vibes.
Scorecards Before Screens: Evals and Launch Gates for PMs Building Agents
A PM guide to defining agent quality with datasets, trace reviews, scorecards, release gates, and business metrics before building the agent UI.
Autotune the Harness: Baking the Improvement Loop into ContextOS
How ContextOS treats autotune as a gated loop over traces, scorecards, replay sets, bounded candidates, approval, and rollout.
Dataset-First Agent Engineering: The Golden Sets Behind Reliable Agents
A practical guide to golden sets, task distributions, corrected runs, held-out releases, and production slices for agent engineering.
How Great AI Engineers Build Agents: Datasets, Scores, and Harnesses That Improve
Why strong AI engineers build datasets, scorecards, traces, and improvement loops instead of treating agents as prompts plus tools.
Scorecards Over Vibes: The Five Metrics That Keep Agents Honest
The five metrics that keep agents honest: policy, utility, latency, safety, and economics.
Trace Review Is the Agent Debugger: Grade the Path, Not Just the Answer
How trace review grades the path, not just the answer, by inspecting context, plans, tools, guardrails, critic verdicts, and corrections.
Agentic AI Systems Before and After ContextOS
A table-first guide to why agentic systems need bounded context, governed tools, typed decisions, replay, evaluation, and controlled improvement.
Building a Reliability Reviewer Agent: 70 Lines Past the Compliance One
How to extend the reviewer pattern for reliability: timeouts, retries, idempotency, fallback behavior, and rollback declarations.
Building a Compliance Reviewer Agent in 60 Lines and a Golden Set
How to build a compliance reviewer agent with a typed verdict envelope, rubric, golden set, and change-control queue.