Source-Judged Compaction: pruning summaries by their sources

When a long-running agent compacts its history, the summary it keeps asks for decisions and next steps, so it mixes facts the environment established with reasoning the compressor wrote itself. Afterwards the agent cannot tell the two apart, and it acts on invented entries as if they were observed. The paper behind this experiment, Guanghui Min and Chen Chen's "Prune the Summary, Not the History: Source-Judged Compaction for Long-Horizon Agents" (arXiv 2610.20927), traces 83% of failed runs in one widely used compressor to actions carrying out summary entries no environment output supports, and proposes a post-processing step that scores each entry for source support and removes the unsupported ones.

We run compaction constantly, in session handoffs and context management. This experiment tested the pruning step on our own fixture histories, against a bar frozen before the run.

What we tested

Eighteen synthetic histories (six code, six data and configuration, six document), each ending in a compaction whose summary contains a mix of supported facts and unsupported entries written to be acted on. Two arms continued from each compacted history, three repetitions each, 108 runs on Gemini 3.8 Flash:

• Baseline: continue from the summary as the compressor wrote it.
• Source-judged: a judge pass scores every summary entry for whether an environment output or the prior summary supports it; unsupported entries are removed before the continuation.

Scoring: whether the continuation enacted an unsupported entry, how many supported facts survived, whether the task succeeded, and tokens per completed continuation (judge tokens counted against the treatment).

Results

The bar was met on all four conditions, and the validity gate passed comfortably: the baseline enacted an unsupported entry in 81.5% of runs, close to the paper's failure rate.

• Unsupported enactment: 81.5% baseline (44 of 54), 0.0% source-judged (0 of 54)
• Supported-fact retention: 90.7% under source-judging
• Task success: 13.0% baseline, 77.8% source-judged
• Cost per completed continuation: 3,649.3 tokens baseline, 616.9 source-judged (0.169×)

The cost figure needs its definition: it is total arm tokens divided by successful completions. Per-run tokens were nearly identical (479.8 treatment vs 473.1 baseline). The treatment is cheaper per finished task because it finishes far more tasks, not because each run is shorter.

Replication

On DeepSeek v4.1 Flash (36 runs), the direction held with one caveat. Enactment fell from 66.7% to 11.1%, success rose from 27.8% to 66.7%, and cost per completed continuation was 0.615×. Retention improved over baseline (83.3% vs 57.4%) but did not reach the 90% absolute threshold the main run met. We report that as a caveat, not a clean second pass: with a different judge model, the pruning cut closer to the bone.

Limits

• The fixtures are 18 synthetic histories. The numbers are not a production estimate.
• Retention cleared the 90% bar on the main run by 0.7 points and missed it in replication. Over-pruning with a weaker or different judge is the open risk for any deployment of this step.
• The substrate is a one-shot packet to deliverable call, with no live tool loop. The continuation never touches a real environment.

What we did with it

The result supports an incorporation proposal, made to the lab's owner with the retention caveat attached: a source-judging pass on one compaction surface first, our own session and handoff compaction, behind a flag. This experiment is also half of a pair. Its sibling, EXP-13, tested the same compaction surface as a security boundary and found the opposite kind of result: the paper's attack did not reproduce at all. Both reports stand as written; the substrate notes in each say why they are not in tension.