Human Oversight for Agent Fleets: Confidence Is Not an Audit Policy

The most dangerous audit dashboard is one that looks prioritized.

It sends the “least confident” agent outputs to a human, clears a fixed percentage every day, and reports reviewer throughput. The queue feels intelligent because the agents supplied a confidence score.

But what if their confidence is nearly constant? What if wrong answers are confidently wrong? What if every model fails on the same hard case? The dashboard can satisfy its review quota while systematically missing the errors that matter.

One Human, N Agents, submitted July 30, formalizes this problem. It studies a supervisor with budget to audit only a small fraction of an agent fleet, where confidence is miscalibrated and errors are correlated. In the authors’ synthetic model, confidence-ranked auditing crosses a measurable threshold beyond which it becomes worse than random. In their six-model measurements, five open-weight models produced nearly constant verbalized confidence, making that signal operationally useless under the tested protocol.

The paper also gives a more uncomfortable result: at very small budgets, “human oversight” can become vacuous—no studied deployable policy reduces undetected errors enough to distinguish review from rubber-stamping.

This should change production practice:

Human oversight is not the presence of a reviewer. It is a measurable reduction in residual risk under a declared workload, budget, and detection process.

Self-reported confidence may be one input after calibration. It must never be the policy.

This is a review of a new preprint, not regulatory advice or deployment proof. The paper’s fleet model, benchmarks, and thresholds are experimental results, not universal constants.

The allocation problem in plain language

Suppose N agents emit one output each round and one reviewer can inspect only B, where B is much smaller than N.

An audit reveals whether an output is wrong, subject to reviewer error. The question is which B outputs to inspect so the number of undetected errors is minimized.

The paper compares:

Its synthetic confidence model mixes two kinds of errors: errors that tend to report lower confidence and a growing share that mimic the high confidence of correct outputs. The mixture strength is δ.

For each audit fraction, the authors locate δ*: the point beyond which lowest-confidence-first produces more residual risk than random sampling.

Scarce review protects the extreme tail—and still may be useless

The threshold moved in a counterintuitive direction:

Larger budgets reached the failure point earlier because they exhausted the informative extreme-low-confidence tail and began auditing poisoned middle ranks. A tiny budget stayed focused on the tail for longer.

That does not mean tiny budgets are safer.

The paper defines oversight as vacuous at level τ when the best studied deployable policy reduces expected residual risk by less than τ relative to no review. With τ = 10%, 10 of 44 synthetic cells were vacuous. At a 2% audit fraction, oversight was vacuous across almost the entire miscalibration range in the tested setting.

So two statements can be true at the same time:

• a very small audit budget is less vulnerable to confidence-rank poisoning;
• that budget catches too little harm to count as effective oversight.

This is the distinction audit dashboards usually hide. “We reviewed 2%” is an activity metric. It says nothing about the fraction of risk the review process removes.

The model measurements are a warning, not a universal model ranking

The authors asked six models to answer 500 GSM8K and 500 HotpotQA items at temperature zero under a locked prompt, returning an answer and verbal confidence from 0 to 100.

For the five tested open-weight models, confidence clustered near 1.0 whether the answer was correct or wrong:

An AUROC near 0.5 means the reported confidence did not usefully rank correct above incorrect answers under this protocol. The proprietary robustness point was informative, though far from perfectly calibrated.

The paper is careful about uncertainty. Point estimates placed several open models at or beyond the synthetic flip threshold, but confidence intervals often straddled it. The right conclusion is not “open models cannot express uncertainty.” It is:

A confidence field is not audit-ready until its ordering value has been measured on the exact model, elicitation method, task slice, and operating point.

Calibration error and ranking value are also different. A model can be numerically overconfident while still ranking its mistakes usefully. For audit allocation, the precision of the actually reviewed tail matters more than a global calibration chart.

Correlated error changes what one audit teaches you

The paper expected models from different families to fail more independently. The measurements did not support that expectation.

Shared item difficulty dominated lineage. Inter-family error correlation was substantial and overlapped intra-family correlation; after controlling for item difficulty, the extra correlation attributable to lineage was not measurable in the two tested datasets.

This cuts both ways.

Correlation is a risk because one difficult item can create more simultaneous errors than the audit budget can cover. It is also a signal because finding an error in one agent should raise suspicion about other agents handling the same task class.

In the paper’s synthetic model, correlation-aware posterior transfer improved more as persistent correlation increased. On a replay over a 15-agent fleet built from recorded model traces, the ordering was:

perfect oracle
  < correlation-aware Diversity-Bayes
  < confidence-ranked ≈ random
  < no oversight

The differences on the real traces were modest in one benchmark and larger in the other. They support a mechanism, not a finished audit scheduler.

The production implication is broader than model families:

• same input or evidence source;
• same Context Pack;
• same tool failure;
• same prompt template;
• same policy version;
• same retrieval index;
• same environment incident;
• same model route.

These are correlation groups. If one audited run fails because an invoice API returned an empty payload, every run that consumed the same payload shape becomes a more valuable review candidate—even if it came from a different model.

Do not let confidence decide audit eligibility

Production oversight needs two separate decisions:

1. Eligibility: which runs must be reviewed or gated regardless of score?
2. Priority: among the remaining runs, which should consume the limited audit budget first?

Eligibility should be deterministic.

Examples that should bypass sampling:

• destructive or irreversible effects;
• delegated actions above a monetary or legal threshold;
• regulated decisions that require human review;
• tenant or classification boundary exceptions;
• first use of a new tool, model route, or policy bundle;
• failed approval gates, loop guards, or security evaluators;
• unresolved evidence conflicts;
• incidents under active containment.

No confidence score should move these runs out of review.

Priority can combine several measured signals:

priority =
  consequence severity
  + irreversibility
  + data sensitivity
  + novelty and distribution shift
  + evaluator disagreement
  + correlated-incident proximity
  + calibrated uncertainty
  + random exploration allocation

This is a conceptual ordering, not a universal numeric formula. The weights should be derived from incidents, evaluator precision, and domain cost—not invented once in a dashboard.

Use a layered sampling policy

A defensible review queue has multiple lanes:

1. Mandatory risk gates

Review every run whose approval mode, regulation, effect type, or policy requires a human. These are not part of the sampling budget.

2. Forced tail capture

Retain every run with:

• policy or safety evaluator failure;
• tool denial or malformed result;
• large budget overrun;
• unusual retry or re-plan count;
• contradiction or missing evidence;
• user correction;
• high-blast-radius action.

This is tail-based sampling, not confidence ranking.

3. Stratified random coverage

Sample within intent, tenant, risk tier, model route, tool, and release version. Randomness protects the system from blind spots in its own evaluators and gives an unbiased base-rate estimate.

Keep a nonzero random holdout even when adaptive policies look better.

4. Correlation-aware expansion

When an audit finds an error, expand review to runs that share the likely cause:

same evidence digest
same tool-result schema
same task slice
same context-pack version
same environment window
same candidate release

The expansion rule should use recorded attributes and incident hypotheses, not a vague “same model family” label.

5. Validated uncertainty targeting

Use self-confidence only if the reviewed tail has demonstrated lift over random on recent production data. Revalidate whenever the model, prompt, task mix, or elicitation protocol changes.

If confidence collapses to a constant, remove it from the allocator instead of adding jitter and pretending the queue is ranked.

Measure whether oversight is non-vacuous

The paper’s vacuity criterion is its most important governance contribution.

Every oversight program should report:

residual_risk_without_review
residual_risk_with_current_policy
relative_risk_reduction
reviewer_false_negative_rate
confidence_interval

The exact risk model can be domain-specific:

• expected financial loss;
• unsafe action count weighted by severity;
• unsupported regulated decisions;
• data-exposure incidents;
• customer harm or correction cost.

Do not use “number reviewed” as the numerator of safety.

If a policy cannot demonstrate material risk reduction, one of four things must change:

1. increase review capacity;
2. narrow agent autonomy or traffic volume;
3. improve automatic deterministic controls;
4. reduce the task’s consequence surface.

Keeping the same fleet and adding a human name to an impossible queue is not a fifth option.

Reviewer quality belongs in the model

The paper includes audit noise ε: the chance a reviewed error is not successfully caught. Correlation-aware transfer loses value when the reviewer is too noisy and correlation is not strong enough.

Production teams should therefore evaluate reviewers and review interfaces:

• Does the reviewer receive the decision, evidence, policy, and effect preview?
• Can they distinguish tool failure from model failure?
• Is the evidence snapshot frozen while they decide?
• Are approvals specific, expiring, and non-transferable?
• Do reviewers see prior correlated incidents without being anchored by the agent’s conclusion?
• Are reviewer reversals and missed errors measured?
• Does queue pressure reduce dwell time below the minimum needed for the risk class?

A human cannot catch what the interface hides. “Human in the loop” is an architecture claim, not a user-interface widget.

Record an audit decision as evidence

An audit record should bind:

audit:
  audit_id: aud_01J7FLEET2
  run_id: run_a1b2c3d4e5f60718
  selection_policy: fleet_audit@3.2.0
  selection_reasons:
    - stratified_random
    - shared_tool_result_incident:inc_442
  risk_tier: high
  model_confidence:
    value: 0.96
    calibration_profile: support_refund@2026-07-15
  evidence_snapshot: evset_91d
  reviewer_id: opr_risk_17
  reviewer_verdict: incorrect
  error_class: silent_empty_payload
  correlated_expansion_id: exp_884
  completed_at: 2026-07-31T03:42:10Z

This makes the review policy replayable. A team can later ask whether a different allocator would have selected the run, how much risk the queue removed, and which incident expansion found the next error.

The limitations that bound the result

The paper states several important limits:

• its main model assumes honest-but-miscalibrated agents, not strategic evasion;
• model measurements use one locked confidence-elicitation prompt and direct answers without chain-of-thought;
• five measured open-weight models operate at high error rates on some slices;
• gpt-4o-mini is the only proprietary robustness point;
• the real-trace replay contains 15 agents and is preliminary;
• aligned benchmark items likely upper-bound correlation for fleets processing different inputs;
• correlation-aware gains depend on persistent error profiles and weaken under frequent model change;
• noisy reviewers can erase the transfer advantage;
• the studied policies are not claimed to be optimal.

These limits argue for local measurement, not inaction. The paper supplies a falsifiable question every production team can ask: does our allocation policy reduce residual risk more than random and more than no oversight?

The best-practice contract

Human oversight for an agent fleet should satisfy ten rules:

1. Separate mandatory review from sampled review.
2. Never let an agent’s self-confidence decide audit eligibility.
3. Validate uncertainty on the exact reviewed tail, not only with global calibration.
4. Keep stratified random coverage to detect evaluator blind spots.
5. Expand from discovered errors using causal correlation groups.
6. Measure reviewer false negatives and interface quality.
7. Version the allocator and record why every run was selected.
8. Report residual-risk reduction, not review throughput alone.
9. Test for vacuous oversight and narrow autonomy when review capacity is insufficient.
10. Recalibrate after every model, prompt, tool, policy, or task-distribution change.

The human is not the control if the agent decides what the human sees.

Real oversight begins when the system can prove that review finds consequential errors, covers the blind spots of automated evaluators, and reduces the risk left after deterministic controls. Everything else is queue management.

What to read next

• Evaluation and Observability
• Approval Modes Are a Risk Taxonomy
• Business Teams Need AI Scorecards, Not Vibes
• The AI Agent Accountability Matrix