The harness is the lever, until it is not

Research track

We build on fixed-weight models. The models we use are someone else’s, frozen at whatever they shipped, and we are not going to retrain them. So everything we can improve lives around the model: the harness — the program that decides what the model sees, which tools it can call, in what order, with what feedback, and what happens when it fails.

So one question decides a lot of our roadmap: how far does harness work actually go?

Two papers from this week’s research intake point at the same answer from opposite directions, and the answer is more useful than “a long way.”

Split the failure before you pick the fix

The first paper asks when harness evolution hits its ceiling. Its move is to label every failed run by the first thing that went wrong, and that splits failures into two kinds:

• Process failures: a blocked call, a loop, a step budget spent before

the work was done. The plan never really got its chance.

• Content failures: the run completed and delivered, and what it

delivered was poor.

A self-evolving harness loop repaired the process kind, and the gains were real: on their planning benchmark, held-out scores roughly doubled on a small model, and held-out delivery rose from 55% to 90%. Then the interesting part. Adapters trained on the trajectories the evolved harness produced kept much of that gain even under the original harness, and content failures fell from about a quarter of runs to about one in twenty. A placebo adapter, trained on shuffled answers, fell below the base model, which is what makes the claim believable rather than rhetorical. The loop also transferred to a second benchmark it had not been tuned on.

Harness evolution is very good at fixing how work proceeds. The gains it produces can later be trained into the weights. But the ceiling is real, and it arrives exactly where the failures stop being about process.

Figure 1. What changed in the ceiling paper's own numbers. Delivery and score rose under harness evolution alone. The content-failure drop came only after adapters were trained on the evolved trajectories.

The same numbers, as a table:

The same numbers in text form:

• Held-out delivery rose from 55% to 90% under the self-evolving harness loop.
• Held-out score on the small model rose from 0.16 to 0.30 under the self-evolving harness loop.
• Content failures fell from about 1 in 4 runs to about 1 in 20 runs after adapters were trained on evolved trajectories.

The harness itself can be the learned thing

The second paper treats the harness as the object being learned. A proposer revises another agent’s harness using execution feedback, the way gradient updates revise weights. The thing changing is a program, so a fixed-weight model can still improve at test time, on a new task, over successive runs. The revision ability transferred to tasks the proposer had not seen.

We rate this one slightly lower on rigour, and we say so in our own report: the experiments use small models on puzzles and question answering, not the long tool-using runs our work consists of. That caveat is part of the point of this journal. A result can be directionally right and still unproven where you need it.

What this changes in our work

Classify before you fix

Every failed run in our delivery pipeline now has to answer the process-or-content question before anything retries it. A process failure gets a harness change: a budget, a boundary, a better error (see this week’s development post, where a timeout spent two repair attempts being treated as a parsing problem). A content failure gets a different conversation entirely, and no amount of pipeline tuning will substitute.

The feedback loop is the product

Reviewing an earlier week’s papers, the conclusion we wrote down was that a feedback loop is what is missing from whatever workflow you use. These two papers say why in a testable form: execution feedback is the training signal for the harness, whether a human or a proposer applies it. So the loop has to be explicit: outcomes recorded, revisions proposed against them, revisions kept only when held-out performance improves. A pile of lessons in someone’s context window is not a loop.

We do not train weights, and that constraint cuts the other way too. The ceiling paper’s discipline still applies to us in mirror image: when our failures are mostly content failures, the honest conclusion is that the harness has done what it can, and the remaining options are a better model, a better task definition, or a human. Pretending another workflow tweak will get there is how teams burn months.

Next week on this track: if the harness is where improvement lives, its boundaries are where the risk lives: what the research says about defending the execution boundary instead of chasing attack patterns.

Sources: “Harness Evolution Hits a Ceiling: When Weight Training Should Begin” (arXiv 2610.11655) and “Harness Learning Enables Generalizable Test-Time Adaptation” (arXiv 2609.35738), both in our Weekly AI Research Report for October 7–13, 2026, where each is scored for relevance and technical merit.