English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Looping Is Not Reliability: Coding Agents Regress 16% of Fixed Bugs Across Revision Rounds

Forum topic · ✨步子哥 · 2026-08-03

Summary

A July 2026 arXiv paper, 'Looping Is Not Reliability' (arXiv:2607.24604), from Alibaba Cloud and HKUST (Qiang Yang), shows that iterative bug-fixing by coding agents does not accumulate correctness. In a controlled study of 900 three-round forced-revision trajectories (30 HumanEval problems, 5 seeds, 7B model), current correctness fell from 82.0% to 67.3% by round three, while ever-correct rose to 85.3% — meaning 16% of trajectories fixed a bug and then broke it again. Stale test evidence was the worst offender: it caused already-fixed bugs to re-break in 25.2% of cases versus 3.0% with current evidence (statistically significant, Holm-corrected p=0.0337). The paper argues correctness is a reversible state, not an absorbing one, and that verifier quality does not imply verifier independence. It proposes StateSeal, an evidence-bound typed loop contract featuring state-bound evidence, typed revision actions, last-known-good checkpoints, risk-aware stopping rules, and bundled admission. Implications span agent evaluation benchmarks, architecture design, and trace-level optimization granularity.

Looping Is Not Reliability: Agents Fix a Bug, Then Fix It Back

> This post analyzes the paper "Looping Is Not Reliability" (arXiv:2607.24604), which reports that coding agents, under repeated bug-fixing revisions, sometimes undo their own correct fixes.

You give a coding agent a bug to fix. It writes a patch, runs the tests, tests pass. You say "do another revision round." The second round changes something else, tests still pass. You check the final code and discover the originally fixed bug is broken again.

You thought it was making progress. In reality it was spinning in circles, losing a bit each lap.

This is not hypothetical — it is an experimentally verified finding from the July 2026 arXiv paper *Looping Is Not Reliability* (arXiv:2607.24604) by a team from Alibaba Cloud and Hong Kong University of Science and Technology (Qiang Yang). The controlled study used 900 three-round revision trajectories (5 seeds) to test one question: when coding agents repeatedly fix bugs, are they accumulating correctness or losing it?

The counterintuitive, precise answer: under repeated revision, the rate of "was ever correct" rose from 82.0% to 85.3%, but the rate of "currently still correct" fell from 82.0% to 67.3%. The agent had fixed the bug before — and later fixed it back.

1. The Core Experiment: 900 Trajectories

The design: 30 HumanEval problems, 5 random seeds, a 7B coding agent performing three forced-revision rounds each — 900 three-round trajectories total. Each round the agent sees test results and continues revising.

Two key metrics:

  • Current correctness: whether the *final* version passes tests
  • Ever-correct: whether *any* round passed tests
  • The gap between them is the fraction of "fixed correctly once, then broken again."

    2. Verdict: Correctness Is Not an Absorbing State

    | Revision round | current correctness | ever-correct | |---------|---------------------|--------------| | Round 1 | 82.0% | 82.0% | | Round 2 | 67.3% | 84.7% | | Round 3 | 69.3% | 85.3% |

    By round 3, 16.0% of trajectories had produced a correct patch and then lost it. With current binary evidence (only telling the agent whether the current version passes), regression dropped to 8.0% — still not zero.

    This directly refutes the default assumption that "once the agent fixes it correctly, the correctness is preserved." Correctness in agent systems is not an absorbing state — it is a reversible state that the next revision round can undo.

    RQ2: Evidence Helps Revision but Hurts Retention

    Three evidence conditions on the 7B model:

  • Current traces: 113/135 error starts repaired (83.7%)
  • Stale traces: 105/135 (77.8%)
  • Unavailable evidence: 97/135 (71.9%)
  • Evidence genuinely helps revision — but the cost appears elsewhere.

    RQ3: Stale Evidence Resurrects Fixed Bugs

    In a 14B replication experiment starting from *already-fixed* states:

  • Current traces: only 4/135 re-broke fixed bugs (3.0%)
  • Stale traces: 34/135 re-broke fixed bugs (25.2%)
  • A 22.2 percentage-point gap, 95% CI [8.9, 37.0], Holm-corrected p=0.0337 — statistically significant.

    Stale evidence resurrects fixed bugs. The agent reads old test outputs, believes a bug still exists, "fixes" it, and reverts already-correct code. This is dangerous because many production memory mechanisms store historical test outputs for agents — if those outputs don't update with the code, they are time bombs.

    RQ4: Verifier Quality ≠ Independence

    A common belief is that stronger verifiers make loops more reliable. This paper disagrees: using three verifiers from different families (Qwen 7B, Qwen 14B, DeepSeek 6.7B), the authors found cross-family diversity does not equal verifier quality. A strong verifier may share blind spots with the coder model. Verifier quality and verifier independence are distinct attributes.

    3. The Solution: Evidence-Bound Typed Loop Contract

    The paper proposes StateSeal, a reference implementation of an evidence-bound typed loop contract with five core designs:

    1. State-bound evidence: every test evidence is bound to a code-state hash; stale evidence auto-invalidates 2. Typed revision actions: revisions are typed actions (fix tests, fix implementation, fix interfaces), each with distinct admission and retention rules 3. Last-known-good checkpoint: correct versions are saved automatically, enabling rollback 4. Risk- and dependence-aware stopping: stopping evaluates "risk vs. benefit of continuing," not just "tests pass" 5. Bundled admission: admission considers impact on retention and other components, not just the current test

    StateSeal is not another agent framework — it is a mechanically enforceable contract any existing coding agent can adopt.

    4. Why It Matters

    For Agent Evaluation

    Benchmarks like SWE-bench and HumanEval only measure final correctness — a fatal blind spot that cannot see code fixed correctly mid-trajectory and later broken. The 85.3% vs. 69.3% gap is another instance of the evaluation blind-spot law: what you measure gets optimized; what you don't measure hides problems.

    For Agent Architecture

    Mainstream coding agents (SWE-agent, OpenHands, CodeAct) assume the generate-test-revise loop converges to correctness. This paper shows loops themselves provide no reliability guarantee — the transition kernel may converge upward or diverge downward. A loop without retention mechanisms and stopping rules is little better than a random walk.

    Granularity Isomorphism

    The paper echoes prior work (Heddle, CodeRescue): agent system optimization must move from single-call granularity to trajectory granularity. "Tests pass" on one call ≠ fixed at trajectory level; "strong verifier" ≠ independent verifier; "useful evidence" ≠ harmless evidence.

    5. Honest Limitations

  • 30 HumanEval problems are entry-level; production SWE-bench tasks are far more complex
  • Forced revision is artificially triggered; real agents decide when to stop. A prospective 540-rollout policy experiment eliminated correct-start harm but also reduced wrong-start repair, failing the joint criterion
  • Repository experiments (24 real bugs, 4 coder stacks) hit a floor effect — bugs too hard for all agents
  • StateSeal is a reference implementation, not a production system
Still, with 900 trajectories, 5 seeds, and Holm-corrected p=0.0337, this is a controlled experiment, not a case study.

6. Final Thoughts

Human engineers also revert fixes — but we have git, diff, and rollback. An agent memory that stores only the latest version is an engineer without git. StateSeal's last-known-good checkpoint is essentially git for agents. The reliability problem in the agent era is not an AI problem — it is an engineering problem.

> Paper: arXiv:2607.24604 > Code: StateSeal reference implementation mentioned in the paper; no public link on the arXiv page yet

FAQ

Q1: Who is this for? Practitioners, researchers, and students interested in AI, machine learning, and deep learning.

Q2: Key takeaways? Correctness in agent loops is a reversible state: 16% of correctly fixed bugs were re-broken; stale evidence significantly resurrects fixed bugs (3.0% → 25.2%).

Q3: Is there open-source code? See the links in the body.

Tags

#coding-agents#llm-reliability#bug-fixing#stateful-loops#agent-evaluation#arxiv-2607-24604#stateseal#software-engineering

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178503904