Probe-and-resume: an executing critic for claimed completions

Experiment run 2026-10-18. Pass bar frozen before any run.

Our delivery pipeline builds software in stages, and one stage verifies a builder's claim that its work is done. In the shape we run today, verification executes the tests the builder says it wrote and reads the evidence the builder recorded. That shape has a known hole: one real change once passed two full verification runs while carrying defects across six patterns, and only a reviewer who probed the running code, instead of reading its records, caught them.

The paper behind this experiment, Jiakai Xu and colleagues' “Probe-and-Resume: When and How to Evaluate Agent Execution” (arXiv 2610.21088), proposes closing that hole with an executing critic: a separate agent that writes and runs its own probes against the claimed work, sends failures back for repair, and rechecks the repair. We tested that proposal against the hole we actually have.

The fixtures reproduce our verification hole before they test anything

We built 24 small synthetic workdirs: six defect patterns drawn from the real incident, each in a clean control version and three defective versions. The patterns are error-reconcile, failed-query, custody reference, admission race, configuration hash, and crash window. One subsystem per fixture, ground truth by construction.

The frozen bar had three conditions. Treatment false-pass rate at most half the baseline's. True-pass retention at least 90%. Cost per verified task at most 1.5 times baseline. A validity gate came first: the baseline had to false-pass at least 6 of the 18 defective fixtures, or the fixtures would not be evidence about anything.

The baseline false-passed 18 of 18 in the gate run, and every defective run in both main runs after it. Verification in the current shape accepted every false claim we handed it.

The critic caught every defect on first check, and the repair loop let a third back in

The main run was 144 runs (24 fixtures, two arms, three repetitions), critic model Gemini 3.8 Flash. A replication with regenerated fixtures and a fresh seed ran the same design once more.

Figure 1. False-pass rate: the share of defective fixtures an arm accepted as done. The treatment number is what survived the full critic, repair, and recheck loop.

The same numbers in text form:

• Baseline false-pass rate: 100% on the main run, 100% on the replication.
• Treatment false-pass rate: 31.5% on the main run, 33.3% on the replication.
• True-pass retention: 100% on both runs. The critic never rejected a clean control.

Split the treatment arm at the repair boundary and the result changes shape. Before any repair, the executing critic's probes caught every defect: 0 false passes in 54 main-run checks and 0 in 18 replication checks. Every post-repair false pass arrived through the repair and recheck round, and they concentrated in the timing and state patterns: admission race accepted after repair in 8 of 9 runs, crash window in 5 of 9, failed-query in 3 of 9. Error-reconcile and custody reference leaked in 0 of 9 each, configuration hash in 1 of 9. Sampled rechecks show the mechanism plainly: the recheck critic sometimes accepted on the claimed tests passing instead of re-running its race probe under real contention. Detection is not the weak point. Re-verification of timing and state-machine defects is.

The cost number needs its accounting basis attached to it

Cost per verified task, under the frozen definition, is the builder-token constant for a fixture plus the arm's verification tokens. The constant prices a builder at the text it emits for the fixture: code, tests, and evidence at roughly four characters per token, averaging 589 tokens. Baseline verification is scripted and consumes no model tokens, so the baseline costs 589 per task. Treatment verification averaged 56,271 tokens per run across the critic, repair, and recheck loops, which lands at 56,860 per task and the 96.6x ratio. Total experiment spend was about 5.77 million tokens.

That basis is honest for a synthetic fixture and misleading anywhere else. A real build stage consumes orders of magnitude more than 589 tokens, in exploration, tool output, and retries. Against a real build cost, a verification stage of about 56,000 tokens would be a small fraction of the total, not 96 times the build. The bar missed under the accounting we froze, and we report it as missed. Anyone quoting the ratio without the basis is quoting a number we do not stand behind. What survives the caveat: at flash-model prices the absolute critic cost is cents per task, the first pass is cheap, and the token appetite lives in the repair and recheck loop, which averaged about 35 model calls per session.

One deviation, recorded because it affects how much weight the numbers carry. The treatment arm was built for a different model, served through a relay that returned truncated or errored responses under this workload; three of twelve early sessions errored and throughput was about one session per fifteen minutes. We moved the arm to Gemini 3.8 Flash on the provider's direct API with the same prompts, the same tool contract, and the same fixtures. The five completed runs on the first model are retained in the dataset and excluded from the headline numbers, and the ones that completed showed the same behaviour: defects caught by execution.

Limits

• The fixtures are small synthetic workdirs, one subsystem each. Real candidates are larger and carry several defects at once. Detection rates here are an upper-bound shape, not a production estimate.
• The repair agent's final message was often empty, because it ended on tool calls. Repair quality is judged only by the recheck outcome, and in the sampled race-pattern accepts the recheck evidence was weak. We cannot prove no genuine fix was applied in every accepted run.
• One critic model family was measured at scale. Whether a different critic shows the same repair-loop leak is untested.
• The baseline faithfully emulates the current verification shape, running the claimed suite and reading recorded evidence. It is not the harness's literal verification code.

What we did with this

Nothing was incorporated. A missed bar is a missed bar, and this one missed on the cost condition as frozen. The guardrail we keep is narrower than the paper's claim: executing probes discriminate claimed completions perfectly at first check, and a repair loop whose recheck does not re-run the original failing probe will hand some of those defects back.

The follow-up this points at, if it is ever run, is a smaller thing than the full treatment: an executing critic as a first-pass-only verification step, with any recheck required to re-run the probe that failed, priced against real build-stage token costs. That is a new experiment with its own frozen bar, not a rescue of this result.