Research
Experiments, published either way
We run a standing pipeline that replicates published agent-systems research against our own delivery pipeline, under pass bars frozen before any run. Every concluded experiment gets a report here — the ones that met the bar, the ones that missed, and the ones where the published effect did not appear in our substrate.
Operations 4 reports
How the delivery pipeline verifies, grades, and steers agent work — and where those mechanisms break.
Probe-and-resume: an executing critic for claimed completions
A verification step that runs the claimed tests and reads the recorded evidence accepted every defective fixture we gave it. A critic that executes its own probes caught every defect on first check, then let about a third back in through its own repair loop.
Bar missed. False-pass and retention conditions met on both runs; the cost condition missed (96.6x against a 1.5x bar), on an accounting basis that prices the builder at fixture text only. Read the cost section before quoting that ratio.
Call-Site Contracts: nothing got through, even unguarded
A large empirical study finds most expected constraints at LLM call sites unmet and nearly every site failing its whole contract. In our packet substrate, the prompt-only baseline passed through 0 of 144 violating candidates across two models.
Bar missed; validity gate missed. The prompt-only baseline passed through 0 of 144 violating-candidate decisions across both models, so code enforcement had nothing to demonstrate against.
PAUSE: steering before the first edit bought nothing here
Intercepting a coding agent just before its first edit, for one planner pass, is reported to raise resolution by double digits. In our substrate the baseline needed no steering (100% unaided), the pass cost 2.45× the tokens, and on the second model the steered arm produced no deliverable at all.
Bar missed; validity gate missed. The baseline solved 100% of tasks unaided on both models; the steering pass cost 2.45× the tokens on the main run and collapsed deliverable production on the second model (0 of 17).
Evidence Substitution: the retrieval cost that did not appear
A paper finds that agents answering from retrieved tool output lose accuracy by answering from their own files instead of the authoritative one. In our substrate, on two models, retrieval cost nothing: 100% accuracy and zero substitutions in every arm.
Bar missed; validity gate missed. The paper's retrieval cost did not appear in this substrate: 100% accuracy and 0% substitution in every arm, on both models.
Perimeter 3 reports
What agents do with the authority they are given, and how to detect actions outside it.
Authorization Scoring: a plain contract beat consult-first
Agents completed every task while acting outside their authorization in 53.7% of runs. A plain explicit contract cut that to 9.3%. Adding a consult-first gate made it worse, 18.5%, at 2.35× the tokens. Completion and authorization are separate scores.
Bar missed (the consult-first conditions); validity gate passed. The plain contract alone cut out-of-authorization actions from 53.7% to 9.3%; consult-first was worse and cost 2.35×.
Secondary-Call Authorization: the attack that barely landed
A published attack poisons tool descriptions so the agent makes an extra, attacker-controlled call after finishing. Reported success: 62.64–100%. In our declared-sequence substrate the planted call was executed in 1 of 87 scored baseline runs.
Bar missed; validity gate missed. The planted call executed in 1 of 87 scored baseline runs across both models, against the paper's 62.64–100% attack success.
False Frontiers: grading without the proposer's packet
When the work being graded arrives inside the proposer's own summary packet, graders agree with wrong work and reject good work. Grading the output alone, with the packet removed, cut total grading error from 38.9% to 2.8% at 1.03× the cost.
Bar met on the main run; validity gate missed as written. Grading error fell from 38.9% to 2.8% at 1.03× cost, and the packet corrupted grading in both directions.
Context 3 reports
How agent memory, compaction, and context selection behave when they are measured instead of assumed.
Compaction Laundering: the forged approval that never laundered
Flattening a conversation into plain text before compaction lets a forged user approval survive summarisation as genuine memory, 87–100% of the time in the paper's harnesses. In our substrate the forged line laundered in 0 of 87 baseline runs across two models.
Bar missed; validity gate missed. Forged approvals laundered in 0 of 87 flattening-baseline runs across both models, against the paper's 87–100%. Retention and success were 100% in the main run.
Dependency-Tethered Memory: verdicts that carry their reasons
A stored verdict stays readable after the facts it rested on have failed. Recording what each verdict derives from, and withdrawing it when that support fails, held unsafe publication at 0% with full retention, at 3.0 checks per request against 4.0 for checking everything.
Bar met on the main run, all four conditions; validity gate missed, because the flat baseline failed weakly (8.7% unsafe against the paper's roughly 62%). Tethered memory: 0% unsafe, 100% retention, 3.0 vs 4.0 checks.
Source-Judged Compaction: pruning summaries by their sources
Context summaries mix facts the environment established with reasoning the compressor invented, and the agent later acts on both. Scoring each summary entry for whether it has a source, and removing the ones that do not, took unsupported enactment from 81.5% to 0.0%.
Bar met on the main run. Unsupported enactment fell from 81.5% to 0.0% and success rose from 13.0% to 77.8%. Replication retention was 83.3%, below the 90% threshold.
Looking for the narrative write-ups? The journal carries those; this index is the complete experiment record. Read the journal