False Frontiers: grading without the proposer's packet

A grader that reads the proposer's own summary of the work is grading the summary as much as the work. The paper behind this experiment, Meijia Chen and colleagues' "False Frontiers: Diagnosing and Mitigating Co-Cheating in Self-Evolving Search Agents" (arXiv 2609.39102), studies that failure inside self-evolving search agents: a proposer and a solver drift into agreeing on shared errors, so the internal score improves while external correctness does not. The mitigation family is independence: grade against something the proposer did not frame.

Our grading path has the same shape. Work in our delivery chain is graded from a packet that includes the proposer's framing of what was done and why it is right. This experiment asked what that framing costs, and what removing it buys.

What we tested

Twenty-four items drawn from our own grading history, 12 that passed and 12 that failed on the record, six of the failures carrying a planted erroneous rationale written to be persuasive. Two arms graded every item three times under a frozen rubric, taste excluded:

• Coupled: the grader sees the output together with the proposer's packet, the way grading normally runs.
• Source-excluded: the grader sees the output alone. The packet is removed from the grading input; nothing else changes.

Main grader: Gemini 3.8 Flash, 144 gradings. Replication: DeepSeek v4.1 Flash as a second grader, 39 of 48 cells covered (the missing cells are transport failures on the largest packets, disclosed in Limits).

Results

The bar was met on all three conditions. Total grading error:

• Coupled arm: 38.9% of gradings wrong (28 of 72)
• Source-excluded arm: 2.8% wrong (2 of 72)
• False agreement (passing work that failed): 36.1% coupled, 2.8% source-excluded
• False rejections (failing work that passed): 41.7% coupled, 2.8% source-excluded
• Grading cost per item: 985 tokens coupled, 1,012 source-excluded (1.03×)

The packet corrupted grading in both directions. It talked graders into passing bad work, and it talked them into rejecting good work: four genuinely approved items were rejected in all three coupled gradings and passed once the packet was removed.

The validity gate missed, and the miss is part of the result. The gate required the coupled arm to false-agree on at least 25% of the planted failures; it managed 22.2%, one grading short. The graders mostly resisted the planted rationales, because those defects are visible in the outputs themselves. What misled them was the ordinary, unplanted packet: coupled false agreement on unplanted failures was 50.0%. The failure mode is real in our grading path; its dominant vector here is genuine proposer framing, not planted text.

Replication

The second grader reproduced the structure with the corruption's sign flipped. DeepSeek's coupled arm degenerated into rejecting everything (17 of 17 rejections, false rejection 100% of pass items, false agreement 0%), while its source-excluded arm graded near-perfectly (one error in 22 gradings). The paper's specific signature, false agreement, is grader-dependent. The structural claim held on both graders: the proposer packet corrupts grading, and removing it restores accuracy at roughly equal cost.

Limits

• The bank's item texts are not published with this report. The items derive from real operations history; a sanitization review cleared the aggregate numbers in this report, generalised 15 of the 24 items for the private dataset, and withheld 9 items' text entirely (they are verbatim design records) in every external artifact. What you are reading uses aggregates and method only.
• The items are record excerpts and faithful summaries, not full artifacts, and several are thin (a few hundred characters). The paired design keeps the arm comparison valid; absolute rates would shift on full artifacts.
• Ground truths are our own recorded verdicts, independent of the proposer but not independent of the platform that produced them.
• Replication coverage is 39 of 48 cells; the missing cells correlate with packet size, not with verdicts, but they are not random.
• n is small: 24 items. Rates move in steps of about 3 points per grading. The effects here are large relative to that grain.

What we did with it

Source-excluded grading is the most direct operational finding in the programme so far, and the cheapest: packet assembly only, no new model calls, 1.03× cost. Because the validity gate missed as written, the result did not auto-advance under our pipeline rules; the incorporation call was put to the lab's owner with these numbers in front of him, including the gate miss. The standing lesson either way: a grading packet assembled by the proposer is an interested document, and grading that reads it is partly grading the sales pitch.