Part 3 of the AI Agents in the Real World series.
An AI agent produces a week’s worth of work in an afternoon.
Then someone has to determine whether the work is correct.
They read the research, run the code, compare the spreadsheet, inspect the sources, check the recipients, reconcile the records, resolve contradictory choices, and repair the pieces that are almost right.
The organization counts the afternoon. The worker experiences the week.
This is the review tax: the human and machine effort required to convert agent output into an outcome someone is willing to stand behind.
When output grows faster than verification capacity, the unpaid balance becomes verification debt. It appears later as defects, reopened cases, unexplained decisions, fragile code, stale documents, customer complaints, and reviewers who approve work they no longer have time to understand.
Productivity research does not support one universal story
Credible evidence points in different directions because AI value is conditional on the work.
An NBER field study of 5,179 customer-support agents found a 14% average increase in issues resolved per hour. The gains were much larger for novice and lower-skilled workers and minimal for experienced, higher-skilled workers. The system offered real-time guidance inside a comparatively structured workflow.
A METR randomized study found that experienced open-source developers working on mature repositories took 19% longer with early-2025 AI tools. Before the study they predicted AI would make them 24% faster; afterward they still believed it had made them 20% faster.
That result is a snapshot, not a timeless verdict. METR’s February 2026 update found weak evidence that later tools likely improved speed, but the researchers judged the new estimate unreliable because developers who would not work without AI selected out and multi-agent use made time measurement harder.
Google’s 2025 DORA reporting adds a perception signal: 90% of surveyed technology professionals used AI at work and more than 80% reported productivity gains, while 30% reported little or no trust in AI-generated code.
These studies do not contradict one another. They measure different people, tasks, tools, periods, and outcomes. Together they reject the shortcut “more output equals more productivity.”
Four costs hide behind the generated artifact
The full cost of an agent-produced outcome is:
total_work = setup + generation + verification + repair + integration + exception_handlingMost dashboards count generation and model spend. The missing terms decide the result.
Verification
Was the artifact compared with the governing evidence, requirements, and current system state?
Examples:
- opening every cited source in a research memo,
- running tests and reviewing a code diff,
- matching invoice fields to the source document,
- checking that an email has the right recipient and attachment,
- confirming that the proposed refund follows the current policy.
Repair
How much work was required after a failure was found?
Repair includes rewriting, rerunning, regenerating, reverting, clarifying, and reconstructing the agent’s incomplete context. A defect that takes two minutes to detect may take two hours to understand.
Integration
Did the artifact fit the surrounding work?
An isolated code function can be correct and still break an interface. A polished market analysis can use different definitions from the finance model. A support response can be factually accurate and violate the team’s escalation policy.
Exception handling
What happens when the system is uncertain, a tool fails, evidence conflicts, or the user changes the goal?
The exception queue is usually staffed by the people most qualified to do the original work. Their time must be charged to the automated workflow.
Why people misperceive the gain
The subjective speedup from AI is often real. Waiting feels shorter than typing. A complete-looking artifact reduces blank-page anxiety. Parallel agents create visible motion. The user can work on something else during generation.
But perceived effort is not elapsed completion time, and elapsed completion time is not trusted throughput.
Use three separate measures:
| Measure | Question |
|---|---|
| Perceived relief | Did the work feel easier or less tedious? |
| Cycle time | How long from accepted request to accepted outcome? |
| Trusted throughput | How many outcomes passed quality, policy, and user checks per total person-hour and machine cost? |
All three matter. Collapsing them into “productivity” makes an honest pilot impossible.
Review is not one activity
Different artifacts require different verification designs.
| Review class | Verification method | Good agent use | Main danger |
|---|---|---|---|
| Deterministic | Schema, tests, reconciliation, constraints | Data transforms, bounded code changes | Tests cover the wrong contract |
| Evidence-based | Compare claims with authoritative sources | Research, case summaries, eligibility checks | Citation exists but does not support claim |
| Judgment-based | Qualified human applies standards | Hiring support, policy exceptions, strategy | Reviewer anchors on plausible output |
| Unreviewable | Reviewer lacks evidence, time, or expertise | None for consequential use | Approval becomes theater |
The fourth class is the critical one. If a reviewer cannot independently assess the output, “human in the loop” is not a control.
The EU AI Act’s human-oversight requirements make the same point for high-risk systems: oversight should be proportionate, and assigned people should understand system limitations, remain aware of automation bias, interpret output, override or reverse it, and intervene safely. Presence is not enough; capability and authority matter.
The verification ledger
Before and during a pilot, record the full work at the task level.
task_id: case_4821
task_class: refund_eligibility
cohort: agent_assisted
time_minutes:
prepare_input: 3
agent_run: 2
wait_not_working: 1
human_review: 6
repair: 4
integration: 1
exception_handling: 0
quality:
accepted_first_pass: false
defects_found_before_action: 2
defects_escaped: 0
policy_pass: true
evidence_pass: true
review:
reviewer_role: support_lead
reviewer_qualified: true
evidence_available: true
decision: accept_after_repair
outcome:
resolved: true
reopened_within_7_days: false
customer_harm: noneDo not require employees to hand-enter this forever. Instrument the workflow and sample the fields that need judgment. During the pilot, however, detailed collection is how hidden work becomes visible.
Measure against a real baseline
A before/after launch comparison is weak because demand, staffing, seasonality, and task mix change.
Where feasible:
- Define the unit of work and accepted end state.
- Stratify by task complexity and worker experience.
- Randomly assign comparable tasks to the existing and agent-assisted workflow.
- Measure total cycle time and active human time.
- Score quality blind to the treatment when practical.
- Track delayed failures: reopens, rollbacks, complaints, and incident cost.
- Report distributions, not only the average.
An enterprise does not need an academic publication. It needs a comparison strong enough to prevent enthusiasm, selection, and easier task mix from deciding the result.
For an individual, the same method can be lighter: compare ten recurring tasks with and without the agent, including review and correction. A personal agent that drafts emails quickly but doubles checking time may still be worthwhile for accessibility or cognitive relief; the user should know the trade.
Build verification before increasing generation
The common sequence is backwards:
- buy more agent capacity,
- generate more work,
- discover a review bottleneck,
- ask the same model to review itself,
- treat reviewer agreement as correctness.
Instead, design a verification path for every consequential claim or action before scaling output.
Make important claims addressable
The reviewer should be able to move from a claim to its source, policy clause, database row, test, or tool result. A long prose rationale without evidence anchors increases review time.
Verify invariants, not eloquence
For a refund, test eligibility, amount, currency, duplicate status, and approval requirement. Do not ask only whether the explanation sounds reasonable.
For code, run the contract tests, inspect security-sensitive paths, and compare the diff with the stated intent. Do not reward the agent for a long summary.
Route uncertainty instead of hiding it
An abstention with a precise missing-evidence reason is a productive outcome. A plausible guess that creates ten minutes of forensic review is not.
Separate maker and checker evidence
A second model can find defects, but shared model families, prompts, and context can produce correlated blind spots. The checker should use independent evidence and deterministic controls wherever possible. The Critic pattern is strongest when it verifies a contract, not when it merely offers another opinion.
Sample the easy-looking cases
If humans review only low-confidence outputs, the agent can become confidently wrong outside the sample. Keep risk-triggered review, but add random coverage across confidence, task type, user group, and tool path. Human oversight for agent fleets develops this audit-allocation problem in depth.
A launch gate that includes review debt
Do not widen the pilot unless all of these are true:
- trusted throughput improves, not only raw output,
- review time is stable or falling with experience,
- qualified reviewer capacity covers the expected volume,
- escaped-defect severity remains inside a stated tolerance,
- the artifact is more inspectable than the baseline,
- exception and rollback time are included in the economics,
- reviewers can reject without schedule or performance pressure,
- a random sample confirms that confident cases are not escaping review.
If the system creates more artifacts than the organization can responsibly verify, pause generation. That is not anti-automation. It is capacity management.
Personal rules for high-stakes work
- Never substitute polish for evidence.
- Open the source behind consequential factual claims.
- Review the exact recipients, attachments, amount, and destination before action.
- Use AI for a second view in medical, legal, or financial matters—not as the accountable professional.
- Prefer drafts and comparisons when you cannot independently verify an action.
- Count the time spent checking and fixing; your attention is part of the price.
The metric that matters
An agent has not saved time when it generated faster.
It has saved time when the accepted outcome arrived faster, with no hidden transfer of error, risk, or exhaustion to the reviewer or the person affected.
That is why the right numerator is trusted outcomes and the right denominator includes human attention.
Everything else is output accounting.