Evidence Substitution: the retrieval cost that did not appear

Some agent harnesses keep long tool outputs out of the context window and let the agent retrieve sections on demand. The paper behind this experiment, Ke Zhang and colleagues' "Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy" (arXiv 2610.26836), measures what that costs with a real scientific simulator: under section retrieval, two GPT models lose 8 to 11 accuracy points, not because they compute worse, but because they answer from their own higher-precision output files instead of the authoritative report, a failure the authors call evidence substitution. Naming the authoritative output recovers most of the loss.

This is a transparency report on a result that did not happen here. We publish it because the bar, the gate, and the outcome were all frozen before the run, and a null result against a specific published number is information our own design decisions need.

What we tested

Eighteen synthetic tasks, each with a long tool report, an authoritative section, and decoy values engineered so that answering from the wrong source gives a detectably wrong answer (every decoy verified as substitutable by the checker before any model call). Three arms, three repetitions, 162 runs on Gemini 3.8 Flash:

• Arm A: the full output in context.
• Arm B: the agent answers from a retrieved section set (top 3 sections by a frozen deterministic rule).
• Arm C: arm B, plus the packet names which output is authoritative.

Replication: the same design on DeepSeek v4.1 Flash, 54 runs.

Results

Nothing failed, anywhere. Accuracy was 100% (54 of 54) in every arm on the main run and 100% in every arm in replication. Evidence substitution was 0.0% in all 90 arm-B runs across the validity smoke, main run, and replication. The recovery condition (C at least 5 points above B) is unmeetable at a 100% ceiling, and the validity gate, which required arm B to actually lose accuracy or substitute, missed as written.

The only differences the experiment measured were costs, and they favoured retrieval:

• Tokens per task: A 1,147.8, B 800.8, C 762.5
• The same shape replicated: A 1,139.6, B 817.2, C 784.6

Limits

These limits are the finding's boundary, and they are substantial:

• Arm B's retrieval is a frozen deterministic rule over a fixed section set, in a one-shot packet, not a live tool loop where an agent chooses what to fetch and can skip the report entirely, which is where the paper's failures occur.
• Every retrieval set contained the authoritative section plus at least one decoy. Whether the models succeeded because the authoritative section is easy to recognise in these fixtures, or because both model pairings sit in the paper's no-detectable-cost class (its Claude pairings showed no cost), cannot be separated by this design. We claim neither reading beyond that.
• Per the frozen spec, per-model results are reported separately and never pooled. Here the two pairings agree, which is itself the reported shape.

What we did with it

No incorporation. The naming remedy was not adopted, because there was no loss for it to remedy. The guardrail this writes back for our own harness work: do not import the paper's 7.9 to 11.3 point retrieval cost, or its 6.3 to 7.9 point naming recovery, as assumed local effects. On our measured fixture shapes, section retrieval with the authoritative section in the retrieved set cost 0 points. A harder test, retrieval sets where the authoritative section is not reliably present, or a live tool loop, changes the scenario rather than one variable, and would be a new experiment, not a relaunch of this one.