Empirical Basis

Numbers from the literature that inform the four primitives. Peer-reviewed sources marked with venue. Others marked arXiv.

The repair loop problem

• LLMs read 14x more than they write (2B input tokens, 140M output in the Anthropic C compiler case study)
• Failed resolutions cost 5-13x more tokens than successes (SWE-Effi)
• Most models lose 60-80% of debugging capability within 2-3 repair attempts. Decay is exponential: E(t) = E_0 * e^(-lambda*t) (Adnan & Kuhn 2025, Nature Scientific Reports)
• GPT-3.5-turbo: lambda=1.33 (useless by attempt 3). GPT-4: lambda=0.76
• Longer error messages correlate with faster decay (same study)
• 65% of patches in iterative repair are duplicates (Chen et al. 2025, ICSE 2025)

Where the errors are

• 83% of LLM errors are logic, not syntax. Compilers catch 17%.
• 94% of LLM compilation errors are type errors, not syntax errors (Mundler et al. 2025, PLDI 2025)
• Type-constrained generation resolves 74.8% of compilation errors but improves functional correctness by only 3.5-5.5% (same study)

What helps

• Self-correction converges only with unambiguous external signals -- compiler errors, test pass/fail, formal verification. Without external signals, self-correction often degrades performance (Kamoi et al. 2024, TACL)
• Structured test feedback (61%) outperforms human explanations (50.5%). Minimal feedback ("the code is wrong") scores 56.9%, nearly as good as the best (Dai et al. 2025, arXiv)
• Scalar reward outperforms Reflexion and Self-Refine with 1000x less compute (Song et al. 2025, arXiv)
• Strategic restart (Thompson Sampling) saves 10-80% of LLM queries while solving more problems (Tang et al. 2024, NeurIPS)
• SWE-agent's LM-optimized interface outperforms human-designed interfaces for agent tasks (Yang et al. 2024, NeurIPS)

What hurts

• AGENTS.md context files reduce success rates while increasing cost 20%+ (Gloaguen et al. 2026, arXiv, ETH Zurich)
• All 8 frontier models degrade monotonically as context increases. 2000-token structured diff beats 2500-token full context on every model (Kumar 2026, arXiv)
• Generating 20 independent attempts beats generating 10 and repairing each (Olausson et al. 2024, ICLR)
• Strict generation-time constraints degrade reasoning (Banerjee et al. 2025, ICML)
• Model capability gap (21pp) dwarfs format/feedback effects (not significant) across 9,649 experiments (McMillan 2026, arXiv)

Our canary pilot

8 Python repair tasks, 4 feedback treatments (2x2: verbose/brief x vague/precise), tested across gpt-3.5-turbo, gpt-4o-mini, gpt-4o, claude-haiku-4-5.

On gpt-3.5-turbo (the only model that didn't ceiling):

Precision matters more than brevity for accuracy. Brevity matters for token cost (47% savings). Current-gen models ceiling at 100% on these tasks regardless of feedback treatment.

Caveat: n=1 per condition, 8 tasks. This is a pilot, not a study. The harness works; the results need more runs for statistical power.