Call-Site Contracts: nothing got through, even unguarded

Wherever a program calls a language model, the program still relies on guarantees about what comes back: allowed values, ranges, cross-references, ordering. The paper behind this experiment, Shihao Weng and colleagues' "When Code Becomes an LLM Call: An Empirical Study of Call-Site Contracts in Agentic Systems" (arXiv 2610.24545), reconstructs the expected contracts at 496 call sites across 40 open-source agentic systems and finds 63% of expected constraints unmet and only 5.3% of sites satisfying their whole contract. Its sharpest claim: the presence of checks says little about their adequacy, because most constraints are merely requested from the model in the prompt rather than enforced in code.

We tested the remedy ordering that claim implies: prompt-stated contract, contract plus a shallow presence check, contract enforced by a deterministic validator. The substrate gave the remedy nothing to remedy.

What we tested

Eighteen synthetic scenarios, each presenting model-output candidates against a stated contract: some compliant, some violating in semantic ways (allowed-value, range, cross-reference, window-ordering, no-extra-field) that all pass a shallow presence and format check, verified 72 of 72 at fixture build. Three arms, three repetitions, 162 runs on Gemini 3.8 Flash:

• Arm A: prompt-only. The contract is stated in the packet; the agent decides which candidates may pass.
• Arm B: the contract plus the verdict of a shallow presence and format check.
• Arm C: the contract plus a deterministic validator's per-constraint verdicts.

Replication on DeepSeek v4.1 Flash, 54 of 54 cells scored.

Results

Pass-through was 0.0% in every arm, on 0 of 108 violating-candidate decisions in the main run, 0 of 36 in the validity smoke, and 0 of 36 in replication arm A: 0 of 144 across both model pairings. Retention of compliant candidates was 100% everywhere, and the whole-contract rate was 100% in every arm. The validity gate, requiring the prompt-only baseline to pass through at least 20% of violating candidates, missed at zero.

With safety identical across arms, the only measured differences were costs, and they ran against enforcement:

• Tokens per task: A 466.4, B 560.6, C 586.7 (arm C is 1.26× arm A, narrowly missing the bar's 1.25× cost condition)
• Cost per correctly handled candidate: A 116.6, B 140.1, C 146.7

Arm B's zero is worth isolating: the shallow check catches none of the planted violations by construction (they are semantic, and all passed it at fixture build), so the models blocked them from the stated contract alone. The paper's gap between requested and enforced constraints did not appear when the contract and the candidates sit together in one packet in front of the model.

Limits

• Arm C measures an agent honouring a validator's verdicts presented in a packet, not a deployed code boundary where model output flows into control flow whether the model cooperates or not. The paper's call sites are the deployed kind.
• Two readings remain open: this fixture set's violations are too legible when the contract is stated alongside the candidates, or both model pairings are in a class that does not show the failure at this scale. The data cannot distinguish them.
• The replication agreed exactly on pass-through, retention, and whole-contract rates, and disagreed only on token costs. Per the frozen spec, per-model results are reported separately; here they happen to coincide.

What we did with it

No incorporation. Code-enforced call-site contracts are not adopted on this evidence, and neither is the check-presence arm: both were solutions to a failure the baseline did not exhibit. The guardrail written back: the paper's 63%-unmet figure is not an assumed local effect. A real test of enforcement needs a substrate where model output reaches an actual control-flow boundary and the prompt-only path can pass violations through, which is a new fixture task, not a relaunch. This report is one of five in this batch (with EXP-07, EXP-10, EXP-12, and EXP-13) where the published failure did not reproduce in a one-shot packet substrate; read together, they mark where that substrate stops being evidence about live systems.