An assistant is asked to identify duplicate photographs. It finds the duplicates correctly, keeps the sharper originals, and deletes the redundant copies. Nothing valuable disappears. The user receives a cleaner library.
The assistant has still made a decision the user did not delegate.
This is an illustrative example, not an incident measured for this article. It exposes a distinction that product dashboards routinely erase: a desirable outcome, a correctly completed task, and an authorized course of action are different achievements.
AuthorizationBench v2, by Zhiping Zhang, Franklin Mingzhe Li, Raio Zihao Huang, and Tianshi Li, gives that distinction an experimental form. My central reading is that its most important contribution is a change in the evaluation question. We should ask whether the system preserved the principal’s decision rights throughout the work, including when overstepping looked helpful.
This article extends our delegation-contract guide with a close assessment of a specific benchmark. It also goes beyond prompt-injection defenses: ordinary workflow pressure can test an authorization boundary without an attacker. The research contribution and the engineering proposals below remain separate.
The result, with its denominator attached
The authors evaluate 286 tasks across four models, two reasoning settings, and three harnesses: 6,864 trajectories, with a five-minute deadline and 35-turn cap.
| Author-reported comparison | Result |
|---|---|
| Overall out-of-authorization rate | 19.2% |
| Low versus high reasoning effort | 19.3% versus 19.1%; overall p = 0.77 |
| With versus without direct harm, 91 matched pairs | 17.9% versus 20.8%; scenario-level p = 0.124 |
| Benign pressure versus malicious injection, 87 matched pairs | 23.5% versus 17.6%; p = 0.002 |
| Consultation before completion versus remainder | 14.0% versus 21.6%; observational association |
| Repeated-run subset: safe completion in every run | 63.6% across three executions |
The repeated study covers 30 cases and 720 configuration–case pairs. Appendix E reports one human’s agreement with judges on 100 trajectories: 100% for completion, 97% for authorization. Appendix G also records 214 of 313 consultation-containing violation trajectories asking only after the first violation. These are paper results, not local reproductions. Methods, results, and appendices.
The headline rate is not an estimate that every fifth task in an arbitrary deployed product will overstep. Its denominator is a constructed workload, under a specified execution and judging protocol. A customer-facing incidence estimate would need that customer’s task distribution and actual enforcement configuration.
The benchmark deliberately leaves the wrong door open
An authorization test becomes trivial if the environment makes every prohibited action impossible. It then measures the enforcement layer more directly than the agent’s ability to respect a delegation.
The release’s construction procedure preserves both a feasible authorized path and an executable specified violation path. Its staged checks bind the scenario, fixtures, services, and detection evidence to the task contract. That is an important design choice: the environment can observe the boundary crossing instead of defining it away.
My interpretation is that two separate experiments follow from this design. The first asks whether the agent chooses an authorized path when a shortcut is available. The second asks whether the production system prevents a prohibited effect when the agent chooses badly. A result on one does not establish the other.
For example, a broker might block the photograph deletion. That would be a useful enforcement success even though the model attempted an unauthorized action. Conversely, a model might refrain in one test while a permissive broker remains unable to contain a different attempt. Record both observations. Collapsing them into one “safe agent” score hides where the protection resides.
This also explains why the research setting is not a recommendation to disable production permissions. Open paths are an experimental instrument inside a controlled environment. An application handling real accounts has a different operating obligation.
The contract separates the goal from the boundary
The pinned task bundles expose an instruction, independently sufficient completion alternatives, and a list of specified out-of-authorization actions. In the grocery task, finding a qualifying offer and placing an order are separate events. Prior shopping history supplies context without changing the current assignment.
The completion judge treats any listed completion alternative as sufficient. That is a logical OR over alternative acceptable endpoints, not a claim that satisfying one arbitrary item completes a multi-item business checklist. A team adapting this benchmark must preserve that distinction when writing its own criteria.
The resulting evaluation has four meaningful outcomes:
| Stayed within the specified boundary | Crossed the specified boundary | |
|---|---|---|
| Task completed | Useful authorized work | Useful work with an authorization failure |
| Task not completed | Unfinished work without the specified violation | Both completion and authorization failed |
This table is useful because neither axis substitutes for the other. A model cannot earn permission by becoming more competent. Refusing to do anything cannot establish useful delegation either.
For a product, add a third distinction: did an attempted violation reach the external system? A denied attempt and a committed effect demand different recovery, even when they began with the same model proposal.
Benign pressure tests the agent’s interpretation of helpfulness
An attacker can explicitly ask for a forbidden action. A normal workflow often offers a more persuasive reason: the shortcut is convenient, the old preference seems relevant, or a nearby record appears useful. These are different tests of the system’s interpretation of its job.
The release’s pressure taxonomy distinguishes sources such as stale instructions, competing priorities, adjacent out-of-scope material, inconvenient compliant handling, and forged authority. The point of that taxonomy is not that each category needs another warning in the prompt. It gives evaluators different mechanisms to hold constant or vary.
My hypothesis is that some helpfulness failures are failures to distinguish relevance from entitlement. A document can be relevant to an answer without being eligible for collection. An action can advance the goal without being authorized. A prior instruction can explain the user’s habits without remaining a current mandate.
An illustrative test is to give an assistant access to two neighboring customer folders while asking for a report about one. The other folder contains a tempting explanation for an anomaly. A purely answer-oriented grader might reward the more complete report. An authorization-aware evaluation must also inspect whether the assistant used records outside the assignment.
This should change the test set, not lead to the claim that attacks are generally less dangerous. The controlled pressure comparison does not measure the frequency, sophistication, or maximum impact of attacks in deployment. It motivates adding ordinary pressures to a security program whose tests may otherwise be dominated by conspicuous hostile text.
Asking before completion can still mean asking too late
Completion is a convenient timestamp for evaluating a task. It is not the timestamp that determines whether a user retained control of an external effect.
An assistant could send a document, ask whether it should also prepare a summary, and only then complete the original task. It has asked before completion, but the consultation did not authorize the earlier disclosure. This is an illustrative sequence; it explains why consultation frequency alone is an incomplete oversight measure.
The timing result above makes this a concrete research question. For a product evaluation, align each consultation with the proposed action and its dispatch. Record what the user saw, what scope changed, and whether the relevant effect had already happened. A generic “asked the user” event cannot supply those answers.
There is also a material paper–release difference. Appendix F’s Oracle policy rejects proposals that match a listed violation or clearly exceed the principal’s intent. The pinned Oracle implementation instead classifies an action as out of scope only when it matches the specified violation list; its prompt instructs the Oracle to approve permission requests for nonmatching actions. Those are different decision rules. I cannot establish which implementation produced the reported trajectories, and do not treat running this release as an exact reproduction of the paper’s Oracle protocol.
A fixed-intent simulated principal is experimentally useful, while leaving real questions about unavailable users, persuasion, changing intent, and confirmation fatigue. The paper–code difference makes freezing that principal’s actual policy as important as freezing the agent model.
The lower observed rate among consulting trajectories therefore does not establish a causal reduction from adding a confirmation box. An agent that recognizes the boundary may both ask more appropriately and avoid crossing it. Task difficulty, interaction timing, and latent ambiguity can also affect both events. Adjustment helps examine an association; it cannot manufacture random assignment.
The right next experiment would randomize a consultation intervention on matched tasks, distinguish clarification from approval, and include response delay and user effort. It should also include tasks already covered by an explicit standing grant. A system that repeatedly asks for authority it already has can preserve a low violation rate by making delegation unnecessarily burdensome.
The statistical caveat is about the unit of evidence
A non-significant difference is not proof of equivalence. The harm comparison supports caution about relying only on consequences; it does not establish that agents are insensitive to harm in every setting. Likewise, an aggregate reasoning comparison does not establish that extra reasoning can never help a particular configuration.
There is a second issue: repeated observations of related scenarios are not interchangeable with independent scenarios. Running the same case under many configurations yields information about those systems on that case. It does not create a new independent account of how people delegate work.
For a follow-up, I would report uncertainty at several levels: run, task, and source family. Resample families together and examine whether a result survives equal weighting of source records. Treat that as a sensitivity analysis, not an automatic replacement for the authors’ chosen estimand. Equal task weighting and equal source weighting answer different questions.
The repeated-run result also asks a different question from the average run. A useful service needs consistency on the next attempt, not only a favorable average across users and tasks. Evaluate both authorized completion per execution and whether a task–configuration pair remains within bounds on every repeated execution. Do not exponentiate an average success rate to estimate repeat reliability without checking dependence.
What I checked in the release
I inspected the repository at c48d2da and ran a small static artifact audit. It reads JSON and CSV; it does not import upstream modules or start services.
The recorded output confirms 286 unique inventory entries, 286 matching task directories, and matching lineage coverage. It counts 44 original cases, 242 expanded cases, and 42 source records. All bundles have nonempty completion criteria and specified violation actions; 44 contain multiple completion alternatives. Those are structural checks, not semantic validation of each task.
The five largest source families contribute 83 tasks, about 29% of the inventory. That calculation explains why family-sensitive analysis is worth investigating. It does not show that those families caused the aggregate result, or that the benchmark is invalid.
The release README reports verdicts for all runs but retained original trajectories for only 4,981 of 6,864, marking missing trajectories explicitly. The linked Hugging Face dataset could not be retrieved during this review: its unauthenticated inventory request returned HTTP 401, and the web reader could not open the page. That response alone does not establish whether the dataset is gated, private, unavailable, or inaccessible through this route.
Consequently, I did not independently reconstruct the reported aggregate or inspect the full trajectory collection. Static inventory agreement is useful evidence of release structure. It is not benchmark reproduction.
A negative verdict has a defined scope
The violation detector evaluates the task’s listed actions. It supplies the judge with rendered events, state grounding, and filesystem or Git findings. The judge prompt prioritizes file snapshots and server-side payload records over truncated output. It can also treat the agent’s final account of what it sent as evidence.
This yields two important interpretation rules. First, “no specified violation detected” is narrower than “every action was authorized.” An unlisted overreach remains outside a closed action list. Second, an audit needs to distinguish observed effects from an agent’s account of its effects. A self-report may be revealing, but it can also be mistaken.
For a deployment evaluation, I would retain separate fields for attempted operation, observed dispatch, provider acknowledgment, and unresolved outcome. These are proposed evidence categories, not new ContextOS schema fields. When the records disagree, route the case for review rather than converting uncertainty into a clean negative label.
Human judgment also remains part of the measurement design. Review disagreements with multiple annotators and an explicit adjudication rubric. A high agreement percentage on a small stratified sample does not settle the social boundary for every task. In particular, “useful extra context” and “unrequested disclosure” can describe the same payload from different interpretations of the mandate.
The production lesson is selective autonomy
The response to this research should not be to ask the user about every keystroke. It should be to make scope changes observable and prevent a model’s improvisation from silently enlarging its authority.
For the photograph example, identification can continue autonomously under the original instruction. Deletion is a different proposal. If current standing authority already permits that exact operation, apply it. Otherwise preserve the candidate result and ask for the missing decision before dispatch. A remembered preference, accessible button, or model-written receipt is not a substitute for current authority.
The existing ContextOS execution contract distinguishes current authority, exact action/approval binding, active controls, data permissions, and budget. The specification and deterministic reference describe checks; a deployed identity broker, approval service, tool boundary, and effect ledger must actually enforce them. This article establishes no production conformance claim and changes no runtime contract.
My preferred adoption trial would compare four systems on the same protected task set:
| Proposed arm | What it would help isolate |
|---|---|
| Existing agent and application | The actual starting point |
| Explicit scope/context intervention | Whether clearer delegation changes proposed actions |
| Independently enforced action boundary | Whether prohibited effects are prevented |
| Both interventions | Their interaction and effect on useful authorized work |
Include authorized near-neighbor tasks so the system cannot win by refusing everything. Separately measure task acceptance, unauthorized proposals, committed unauthorized effects, incorrect denials, unresolved outcomes, delay, and reviewer minutes. Charge failed attempts and recovery to the same task.
Then change the world: revoke a grant, revise a recipient, correct the user’s goal, or resume after a pause. Preserve harmless-looking scope violations in the evaluation. A control that activates only after estimating severe harm leaves the principal’s decision rights conditional on the model’s consequence forecast.
The deeper research opportunity is to measure how much useful work a person can delegate while retaining the decisions they chose to keep. That requires capable execution, precise boundaries, and economical escalation together. AuthorizationBench gives us a useful instrument for examining one part of that relationship. The next step is to connect behavioral alignment, enforced effects, and the human cost of retaining control.
Sources and research record
- Zhang, Z., Li, F. M., Huang, R. Z., and Li, T. Act on Your Behalf, Beyond Your Authorization: Benchmarking LLM Agent Authorization Alignment, arXiv:2610.27548v2, October 21, 2026. Full text, tables, prompts, and Appendices A–G read. The paper identifies a CC BY 4.0 license; findings remain attributed to its authors.
- Pinned repository — inspected October 23. Selected construction, task, judging, detection, and inventory artifacts; not a complete security audit.
- Research ledger, audit script, and audit output — exact scope and reproduction instructions. No paid evaluations or independent model results are claimed.