Operations · EXP-04
Task Underspecification: which missing piece hurts most
Remove one piece of a task, the goal, the context, or the constraints, and each gap is supposed to damage the work in its own recognisable way. In our substrate the ranking inverted: missing context destroyed completion outright, hidden constraints broke compliance, and the missing goal, the papers' headline gap, did the least characteristic damage.
Verdict against the frozen bar
Bar missed, partial pattern: 1 of 3 clauses on the main run, 0 of 3 formally on replication with two narrow misses. The gap-class taxonomy is not adopted. Missing context destroyed completion (success down 72.2 points) instead of burning tokens; the goal gap did the least characteristic damage.
Real task briefs are incomplete, and the kind of incompleteness is supposed to be diagnosable from the damage. Two papers anchor that idea. Kaiyu Huang and colleagues' "AM-Bench: Understanding Agents under Task Underspecification across Models and Harnesses" (arXiv 2610.18090) varies goals, context, and constraints across 20,160 runs and finds distinct signatures: goal gaps hinder completion, context gaps increase token use, and constraint gaps weaken compliance without necessarily preventing completion. Dongxu Yang and colleagues' "FDE-Gym: The Last Mile of AI Delivery" (arXiv 2610.22350) finds undiscovered constraints the primary bottleneck in realistic delivery work. If the signatures are reliable, a reviewer can read a failed run backwards to the gap that caused it, and a task pipeline can tag and route underspecified work by class. That diagnostic use is what we tested, against a bar frozen on 2026-10-18, before any run.
What we tested
Eighteen known-completable tasks (six code, six data and configuration, six document), every reference completion verified against a deterministic checker before any model call. Four arms remove exactly one element from an otherwise complete brief:
- Full: the complete task.
- Goal-missing: the objective statement removed.
- Context-missing: the background material removed.
- Constraint-missing: the stated constraints removed; the checker still enforces them, so compliance is measurable against rules the agent never saw.
Three scores per run, from the deterministic checker, no model judge: success, completion, and compliance. Main run: Gemini 3.8 Flash, three repetitions per cell, 216 of 216 runs scored, no errors.
Results
EXP-04 — verdict against the frozen bar (main run, 216 runs)
| Condition | Measured | Verdict |
|---|---|---|
| Clause 1: goal-missing success at most half of Full | 57.4% vs 47.2% required | Missed |
| Clause 2: constraint signature (compliance −20+, completion −15 max) | −57.4 pts / −13.0 pts | Met |
| Clause 3: context signature (tokens 1.3x, or the stated fall pattern) | 1.06x tokens | Missed |
| Validity gate: Full-arm success at least 60% | 94.4% | Met |
Source: NonceLogic EXP-04 RESULTS, bar frozen 2026-10-18, run 2026-11-11.
Main run, per arm (n=54 each):
- Full: success 94.4%, completion 94.4%, compliance 100%, 273.5 tokens
- Goal-missing: success 57.4%, completion 57.4%, compliance 79.6%, 285.1 tokens
- Context-missing: success 22.2%, completion 22.2%, compliance 100%, 289.3 tokens
- Constraint-missing: success 37.0%, completion 81.5%, compliance 42.6%, 274.8 tokens
Source: NonceLogic EXP-04, 216 runs (Gemini 3.8 Flash), 2026-11-11.
One clause of three met, and the pattern underneath is more interesting than the score:
Missing context was the most destructive gap, through the opposite channel from the paper's. AM-Bench's context signature is token burn: the agent spends effort rediscovering what it was not told. Here success fell 72.2 points while token cost stayed flat (1.06× Full). In this substrate there is no rediscovery loop to burn tokens in, a one-shot packet in, a deliverable out, so missing context kills the deliverable instead. The paper's mechanism is not refuted for live harnesses; it is not expressible in this one, a limit recorded before the run, not after it.
Hidden constraints broke compliance hard, the most paper-consistent signature, but the clean shape only partially held. Compliance fell 57.4 points while completion fell 13.0, inside the bar's allowance on the main run. A secondary signal cuts the same way: on constraint-missing runs the deliverable still satisfied at least half of the hidden checks in 83% of cases. The work gets done; it gets done against rules the agent could not see. That is FDE-Gym's bottleneck, visible here.
Goal damage was real but shallow. Goal-missing success was 57.4% against Full's 94.4%: the model infers a plausible goal from the surviving context and constraints and completes about half the time. AM-Bench's sharper collapse (12.5% against 76.5% in its headline comparison) does not appear on this pairing.
Replication, and its caveat
The replication caveat here is structural, and we state it as plainly as the results file does. The replication of record is a same-model replication: Gemini 3.8 Flash over the same direct route as the main run. It establishes run-to-run stability of the signatures on this pairing only. It does not test transfer across model pairings, which is the very variation AM-Bench reports, so it is weaker evidence than the frozen spec intended. The cross-model replication (DeepSeek v4.1 Flash via OpenRouter) died twice on transport errors with 63 of 72 cells scored; that partial series is reported separately and is never pooled with the Gemini series.
On the replication of record, the arm numbers nearly reproduced the main run (Full 94.4%, goal-missing 50.0%, context-missing 22.2%, constraint-missing success 38.9% with compliance 44.4%), and formally 0 of 3 clauses met, with two narrow misses: goal-missing success 50.0% against the 47.2% clause-1 threshold, and the constraint signature's completion fall of 16.7 points against the 15 allowed. The partial DeepSeek series is weaker still as evidence: its Full-arm baseline is poor (success 44.4%, compliance 44.4%), which compresses every fall by floor effects. We treat it as an anecdote, in the results file's own word: the signatures do not cleanly transfer to that pairing, and the series is incomplete.
Limits
- The substrate is a one-shot packet to deliverable call. It has no rediscovery loop, so the context token-burn channel cannot exist in it, and no clarification channel, so the discovery-versus-asking trade-off both papers discuss is untested here. Two of the papers' three mechanisms are only partially expressible in this harness.
- The replication of record is same-model, for the transport reasons stated above. Cross-model transfer, the property that would make a diagnostic taxonomy useful across a fleet, rests on one incomplete series with a weak baseline.
- Eighteen tasks, one main model pairing. The compliance signature replicated in magnitude across both Gemini series; everything else is sized to that grain.
What we did with it
The gap-class taxonomy is not adopted, and no incorporation is proposed: under the pipeline's rules a partial pattern does not adopt. The recorded finding for packet work in this substrate is narrower and, we think, still useful: the two damage classes that actually showed up were missing context and hidden constraints, while goal-completeness, the class the papers rank first, did the least characteristic damage. A retest of the full taxonomy belongs on a live tool-loop harness, where an agent can spend tokens rediscovering context and can ask questions, because those are the channels two of the three signatures live in.
Source paper
Kaiyu Huang, Yiting Li, Angyang Gu, Pinjia He, Minchen Yu, Qingjiang Shi. “AM-Bench: Understanding Agents under Task Underspecification across Models and Harnesses.” arXiv:2610.18090
Dongxu Yang, Hongda Yan, Renxin Wang, Baoli Zhang, Haojia Zuo, Ying Zhao, Youmin Chen, Xiaojian Liao, Congming Gao, Tianyang Jiang, Yixiao Chen, Qing Wang, Zhiguo Yu, Xiaqing Li, Yuhao Zhang, Jiawen Sun, Renzhe Xu, De Mi, Tiancheng Wu, Xianya Fu, Xuanlu Zhao, Bo Feng, Meixuan Ren, Guanming Lin, Ziheng Zhou, Xinhua Wang. “FDE-Gym: The Last Mile of AI Delivery.” arXiv:2610.22350