Evaluation
42 essays tagged with Evaluation.
Claude 5.5: Efficiency Gains Need a System-Level Evaluation
A research assessment of Opus 5.5 and Sonnet 5.5: workload economics, safeguard fallbacks, benchmark comparability, and a migration experiment.
Cohere Embed 5: Retrieval Quality Is Only the Start of an Evidence Pipeline
A research assessment of Embed 5 and North Small Translate: shared embedding spaces, index migration, multilingual evidence, deployment limits, and retrieval evaluation.
DeepSeek V4.1 Flash: Cache Economics Meet Model Migration Risk
A research assessment of DeepSeek V4.1 Flash's asymmetric architecture, cache claims, native vision, and conflicting V4 Pro migration notices.
Muse for Small Business: The Approval Is Part of the Product
A deeper assessment of Meta's September 29 Muse expansion: connector scope, meaningful approvals, cross-app recovery, and the economics of owner attention.
Microsoft's New MAI Voice Stack: A Partial Transcript Is Not a Final Instruction
A research assessment of MAI-Transcribe-2-Streaming and MAI-Voice-2.1: latency, revisable speech, speculative tool use, and a voice-agent evaluation plan.
Mistral Vibe: Editable Memory and the Difference Between an App and a Snapshot
A research assessment of Mistral's September Vibe updates: Agentic Knowledge Base, spreadsheets, Mini App Canvas, migration, and artifact verification.
OpenAI After DevDay: Cheaper Intelligence, Persistent Agents, and the Cost of Accepted Work
A research assessment of GPT-6.1 Sol, Dots, and the DevDay platform changes: what the evidence supports, where costs move, and how to evaluate persistent work.
Qwen-Planner-Agent: What Model–Harness Co-Evolution Can Actually Prove
A critical assessment of Alibaba's September 24 Qwen-Planner-Agent preprint: hybrid environments, development-set feedback, attribution, and a stronger transfer experiment.
Agent Harness Benchmark: An Experimental Protocol Beyond Model Scores
A proposed benchmark protocol for isolating harness effects, injecting operational faults, measuring repeated reliability, and publishing auditable scorecards.
From Feature Stores to Agent Memory: What Production ML Teaches Agent Systems
Production ML lessons for agent memory: availability-time correctness, historical replay, versioned materialization, promotion, correction, and deletion across derived stores.
Welcome, Astra: What GPT-6 Changes About Intelligence at Work
A research analysis of OpenAI's GPT-6 Astra: benchmark evidence, scientific discovery, tool use, safety, economics, and an evaluation plan for real work.
The AI Agent Exception Economy: Why the Long Tail Decides ROI
Agent ROI is won or lost in ambiguous, failed, and high-risk cases. Price the exception queue before automating the happy path.
The AI Agent Review Tax: When Automation Creates Verification Debt
Measure the review, repair, and integration work behind agent output—and design verification that catches errors without erasing the gain.
AI Agents in the Real World: Five Problems Capability Demos Hide
A research-backed map of the delegation, review, permission, memory, and exception failures that decide whether AI agents help or harm.
Proof-Carrying Context: Why AI Agents Need More Than a Context Window
A research-grounded revision of context engineering: from relevant tokens to a replayable view of evidence, conflicts, omissions, and decision sufficiency.
Production Prompt Engineering in 2026: From Instructions to an Evaluated Contract
A deep production guide to prompt engineering: lean instruction design, stable cacheable prefixes, tool and output contracts, reasoning budgets, datasets, robustness tests, versioning, rollout gates, and a 90-day implementation roadmap.
Persistent Memory Poisoning: The Attack That Outlives the Session
A production security architecture for attributable, revalidated, authority-bounded, traceable, and selectively reversible agent memory.
Multi-Agent Consensus Is Not Correctness: How Debate Manufactures Confidence
Why agent agreement can hide correlated error—and how to build governed deliberation with effective agent count, independent verification, and release gates.
Agent Evaluation After the Leaderboard: The KDD 2026 Production Assurance Playbook
KDD 2026 shows why agent evaluation must cover protocols, trajectories, external state, evaluator operating profiles, causal rollout, and drift.
OpenAI's Latest Science Work: From Benchmarks to Verifiable Discovery
What OpenAI's newest mathematics, scientific-computing, biology, and evaluation work says about AI as a research collaborator.
Graph Engineering for AI Agents: The X Hype Cycle and a Production Playbook
Separate graph-engineering signal from X hype, then compile agent workflows with governed transitions, graph invariants, durable effects, and exact replay.
Reverse Prompting: Work Backward from the Output, Then Prove the Prompt
Reverse prompting is useful when it turns a desired output into a candidate instruction contract and validates that contract forward. Here is the research, the workflow, the failure modes, and the security boundary.
Human Oversight for Agent Fleets: Confidence Is Not an Audit Policy
A new audit-allocation paper shows that self-reported confidence can make limited human review worse than random and that tiny audit budgets can become rubber-stamping. Production oversight needs risk gates, stratified random coverage, correlation-aware learning, and a measured non-vacuity test.
Passive Awareness Is the Missing Multi-Agent Primitive—But It Needs a Control Plane
AgentRadio shows that agents improve when discoveries can cross lane boundaries during execution. Production teams should keep the asynchronous channel, then add typed evidence, bounded delivery, authority isolation, and replay.
Adaptive Agent Harnesses: Learn From Experience Without Letting Production Rewrite Itself
MemoHarness shows that execution experience can improve the control layer around an LLM. Production systems need a stricter pattern: adaptive performance inside an immutable safety envelope.
Agent Memory Is a Systems Workload: What SelfMem Changes—and What It Does Not
SelfMem improves long-horizon recall by letting an agent optimize its memory strategy. New systems and security research shows the production contract must also cover cost, freshness, provenance, and poisoning.
Your Agent Benchmark Is Measuring the Harness
Two new evaluation systems separate benchmark, harness, model, and environment. Their results show why one leaderboard score cannot tell you which agent to ship.
Multi-Agent Systems Need Replay Before More Agents
Multi-agent systems multiply coordination, cost, and causal ambiguity. Prove one agent is replayable before adding lanes and handoffs.
Red-Team Agent Hijacking: Build a Security Eval Gate for Repeat Attacks
A practical agent-hijacking evaluation harness: scenario design, adaptive and repeated attempts, path-aware metrics, deterministic release gates, and production replay.
Agent Harness: An Architectural Framework for Production AI Agents
A whitepaper on typed contracts, policy gates, traces, verification loops, and release control for production AI agents.
ContextOS: A Research-Grounded Architecture for Governed Agent Runtimes
A research-grounded framing of ContextOS as the portable decision-control and evidence architecture around governed agent runtimes.
Harness Improvement Loops Need Replayable Environments
Why harness improvement needs replayable episodes, bounded mutations, scorecards, source closure, and promotion gates.
How to Judge AI Work: Scorecards, Not Vibes
A practical guide for business teams evaluating AI agents with scorecards, examples, traces, human corrections, and launch gates instead of demos and vibes.
Scorecards Before Screens: Evals and Launch Gates for PMs Building Agents
A PM guide to defining agent quality with datasets, trace reviews, scorecards, release gates, and business metrics before building the agent UI.
Autotune the Harness: Baking the Improvement Loop into ContextOS
How ContextOS treats autotune as a gated loop over traces, scorecards, replay sets, bounded candidates, approval, and rollout.
Dataset-First Agent Engineering: The Golden Sets Behind Reliable Agents
A practical guide to golden sets, task distributions, corrected runs, held-out releases, and production slices for agent engineering.
How Great AI Engineers Build Agents: Datasets, Scores, and Harnesses That Improve
Why strong AI engineers build datasets, scorecards, traces, and improvement loops instead of treating agents as prompts plus tools.
Scorecards Over Vibes: The Five Metrics That Keep Agents Honest
The five metrics that keep agents honest: policy, utility, latency, safety, and economics.
Trace Review Is the Agent Debugger: Grade the Path, Not Just the Answer
How trace review grades the path, not just the answer, by inspecting context, plans, tools, guardrails, critic verdicts, and corrections.
Agentic AI Systems Before and After ContextOS
A concrete before-and-after guide to agentic AI: what context, authority, evidence, replay, and release controls change in a refund run—and what they cannot guarantee.
Building a Reliability Reviewer Agent: 70 Lines Past the Compliance One
How to extend the reviewer pattern for reliability: timeouts, retries, idempotency, fallback behavior, and rollback declarations.
Building a Compliance Reviewer Agent in 60 Lines and a Golden Set
How to build a compliance reviewer agent with a typed verdict envelope, rubric, golden set, and change-control queue.