# AuthorizationBench v2 — research and reproduction ledger

Research date: October 23, 2026. Accompanies “AuthorizationBench: Success Does Not Grant Authority.”

## Evidence identity and scope

- Paper: Zhiping Zhang, Franklin Mingzhe Li, Raio Zihao Huang, and Tianshi Li, [*Act on Your Behalf, Beyond Your Authorization: Benchmarking LLM Agent Authorization Alignment*](https://arxiv.org/html/2610.27548v2), arXiv:2610.27548v2, October 21, 2026. The HTML identifies CC BY 4.0 licensing. Attribution here does not imply author endorsement of this assessment.
- Read the main text, all tables, figure captions, references, and Appendices A–G, including construction, taxonomy, validation, Oracle/judge prompts, statistical robustness, consultation timing, and repeated execution.
- Downloaded HTML SHA-256: `a5bac1cf015ebbb323f07de4fcfc91c08edbac0d0f5298543d58edf107874a2c`. This identifies the retrieved HTML representation, not a canonical semantic hash of the paper.
- Code: [PEACH-Research-Lab/AuthorizationBench](https://github.com/PEACH-Research-Lab/AuthorizationBench/tree/c48d2da60f207ad2664feebda516b1cc72f4fa98), inspected commit `c48d2da60f207ad2664feebda516b1cc72f4fa98`.
- The release manifest separately records `source_revision: 7e57b7201c06dcb516454169a36c6915dfc8536c`. That field is not silently substituted for the inspected Git revision or asserted to identify the paper's exact experimental checkout.
- Repository inspection was selective. It was not an exhaustive code, service, security, or dataset audit.

## Claim-to-source map

Repository paths below are relative to the pinned code tree.

| Subject | Evidence |
|---|---|
| Experimental population, deadlines, headline comparisons | Paper Sections 3–4 and Appendix F.1 |
| Independent completion/authorization judgments | Paper Sections 3.1 and 4.1; `evaluation/judges/completion_judge.py`, `evaluation/detectors/violation.py` under `authorization_eva_sandbox/` |
| Completion alternatives are independently sufficient, not checklist partial credit | Paper Appendix C.1 and F.4; completion judge; sampled task configuration; static audit counts only list structure |
| Authorized and violation paths remain executable | Paper Section 3.3 and Appendix C.2; `docs/CONSTRUCTION.md` describes the release's construction and validation route |
| Benign and malicious pressure mechanisms | Paper Appendix B.2; `taxonomies/situational_pressures.md` |
| Scenario-level inference, not independent replicated configurations | Paper Appendix G.4; family-sensitive analysis is an editorial proposal, not an author-reported result |
| Consultation association and temporal qualification | Paper Section 4.2, Finding 6; Appendix G.5, Tables 14–16 |
| Repeated execution uses configuration–case pairs | Paper Appendix G.6, Table 17; distinct from the full benchmark population |
| Sampled human agreement | Paper Appendix E: two reviewers for 35 task-quality cases; one human for 100 trajectory judgments. These are different validation exercises |
| Paper Oracle policy | Paper Appendix F.2: listed violations **or** actions clearly beyond intent are refused; never expand authorization |
| Release Oracle policy | `authorization_eva_sandbox/oracle_user.py`, especially `ORACLE_USER_SYSTEM_PROMPT`: out of scope only when matching a listed violation; prompt instructs approval for nonmatching action requests |
| Agent system-prompt conditions and fingerprints | `authorization_eva_sandbox/evaluation/harness/prompt_composition.py`; this is not the Oracle policy file |
| State evidence and interpretation of self-reported effects | `authorization_eva_sandbox/evaluation/judges/violation_judge.py`; `evaluation/detectors/violation.py` |
| Inventory, lineage, and grouping | `benchmark/manifest.json`, `benchmark/lineage.csv`, all released `tasks/*/task_config.json` files |
| Retained trajectory count | Pinned `README.md`, “Dataset and reproducibility.” A publisher disclosure, not independently counted downloaded trajectories |
| Production authority and enforcement distinctions | Existing ContextOS [execution-contract documentation](/docs/implementation/execution-contract). No runtime/spec changes or deployed conformance assertion |

## Material paper–release difference

The Oracle policy in the inspected code is not the same decision rule as the paper's Appendix F.2. The paper's policy can refuse an action that clearly exceeds intent without matching a listed violation. The current implementation defines out-of-scope actions through the specified list and instructs the Oracle to approve nonmatching action requests.

This matters to the interpretation and reproduction of consultation. It does not show which policy generated the paper's reported runs, whether the difference changes a particular score, or that the authors' numerical findings are incorrect. No experiment was run to quantify its effect. Both descriptions remain explicit instead of being blended into one protocol.

## Local execution

Executed the article's own static audit against the pinned checkout:

```text
python3 public/research/authorizationbench-2026-10-23/audit_release.py /private/tmp/contextos-authorizationbench/upstream
```

The checked-in [audit script](/research/authorizationbench-2026-10-23/audit_release.py) uses the Python standard library, reads release JSON/CSV artifacts, and asks Git for the checkout's revision. It does not import upstream Python or invoke its bootstrap, agents, judges, containers, services, or external APIs. It exits nonzero if an inventory/structure check fails.

The full [recorded output](/research/authorizationbench-2026-10-23/release-audit.json) contains input hashes and eight passing checks. It confirms 286 unique manifest tasks, matching bundle directories and lineage IDs, 44 original/242 expanded cases, 42 source records, nonempty criteria/actions, and 44 task configurations with multiple completion alternatives.

The five largest source families contain 21, 18, 16, 16, and 12 tasks. Their sum, 83/286 = approximately 29.0%, is local descriptive arithmetic. It does not estimate their contribution to the observed violation rate or prove a statistical defect. Family-weighted sensitivity analyses are proposed future work.

To repeat, obtain the official repository, check out the inspected commit above, and run the script from this site's repository with that checkout as its only argument. Compare both revision and file hashes. Running against a changed release produces evidence about that changed release, not the pinned one.

## Results that were not reproduced

- No 6,864-run model experiment, 30-case repeated-run study, paired significance test, regression, or judge-agreement study was rerun.
- The paper's numerical table is attributed to v2. The local script validates release inventory and list structure, not model behavior or semantic quality.
- The README reports retained original trajectories for 4,981 runs and verdict/metadata records for the remaining 1,883. No missing trajectory is treated as an independently inspectable trace.
- An unauthenticated request to `https://huggingface.co/api/datasets/peach-lab/AuthorizationBench/tree/main` returned HTTP 401. The web reader could not open the linked dataset page either. No account access was attempted. Those observations do not establish why access failed.
- Dataset verdicts and trajectories were therefore not downloaded or independently aggregated. This limitation is visible in the article and editorial note.
- “Within authorization” in the benchmark refers to the specified action set and judge, rather than a guarantee about every possible operation in a trajectory.
- The article's product scenarios, proposed intervention study, deployment scorecard, and interpretation of decision rights are original editorial analysis. They are not additional experimental findings by the paper's authors.