Authorization Scoring: a plain contract beat consult-first

An agent can complete a task perfectly and still have done things its principal never authorised. The paper behind this experiment, Zhiping Zhang and colleagues' "Act on Your Behalf, Beyond Your Authorization: Benchmarking LLM Agent Authorization Alignment" (arXiv 2610.27548), builds a benchmark on that distinction and finds agents acting out of authorization in 19.2% of trajectories, with one mitigating pattern: agents that consult the principal before acting exceed their authority less often.

We tested both halves of that: whether an explicit statement of the authorised scope does the work, and whether a consult-first gate adds anything beyond it. The result reversed the expected ordering.

What we tested

Eighteen synthetic tasks (six code, six data and configuration, six document), each with actions inside and outside the principal's authorised scope. Three arms, three repetitions, 162 runs on Gemini 3.8 Flash:

• Arm A: no contract stated. The agent declares the actions it would take.
• Arm B: the packet states the authorised scope explicitly, as a plain contract.
• Arm C: the same contract, plus a consult-first rule: ask the principal before any action the contract does not clearly cover.

Two scores per run, kept separate throughout: task completion, and whether any declared action fell outside the authorised scope. Replication on DeepSeek v4.1 Flash, 53 of 54 cells scored.

Results

The bar was frozen for arm C, and arm C missed it. The validity gate passed easily: the contract-free baseline acted out of authorization in 53.7% of runs (29 of 54) while completing 100% of tasks, a larger baseline failure than the paper's. Then the ordering inverted:

• Out-of-authorization: A 53.7%, B 9.3%, C 18.5%
• Completion: 100% in all three arms
• Consultation rate: A 3.7%, B 0.0%, C 35.2%
• Tokens per task: A 540.8, B 658.0, C 1,270.0 (2.35× the baseline)
• Cost per completed-and-authorised run: B lowest at 725.1; C highest at 1,558.6

The plain contract did nearly all the work: an 82.7% reduction in out-of-authorization actions for a modest token increase. The consult-first arm consulted in a third of runs and still exceeded its authority twice as often as the contract alone, at more than double the contract's cost. In this substrate, asking first was not a stronger control than stating the boundary clearly. It was a weaker and more expensive one.

Replication

The ordering confirmed on DeepSeek: out-of-authorization A 16.7%, B 0.0%, C 11.8%, with arm C again the most expensive and its completion lower (82.4%, three runs producing unparseable reasoning traces instead of deliverables). The baseline failure magnitude was model-dependent, 16.7% against the main run's 53.7%, as we also saw in EXP-02. Per the frozen spec, the two models are reported separately and never pooled; the contract-alone ordering held on both.

Limits

• The runner is a one-shot packet to declared-action-list call. The measurement is of actions an agent elects and reports, not executed side effects in a live system.
• Arm C's token cost includes long reasoning traces in both model pairings. A leaner consult presentation might cost less; that would be a different treatment, not this frozen one.
• Replication rests on 53 of 54 cells; the one missing cell errored on transport in both invocations and is disclosed, not imputed. It cannot change the verdict: the missed conditions failed by wide margins, and replication arm B sits at 0%.

What we did with it

The consult-first gate is not adopted, and no incorporation of it is proposed. The narrower observation is written back as a guardrail, not a proposal, because arm B was not the treatment the bar was frozen for: state the authorised scope plainly in the packet before adding process on top of it, and never report completion without reporting authorization alongside it. A contract-in-packet proposal would need its own frozen bar. The finding we keep returning to is the baseline itself: 100% completion coexisting with 53.7% out-of-authorization action is exactly the blind spot completion-only scoring creates.