Looping Is Not Reliability: Agents Fix a Bug, Then Fix It Back
> This post analyzes the paper "Looping Is Not Reliability" (arXiv:2607.24604), which reports that coding agents, under repeated bug-fixing revisions, sometimes undo their own correct fixes.
You give a coding agent a bug to fix. It writes a patch, runs the tests, tests pass. You say "do another revision round." The second round changes something else, tests still pass. You check the final code and discover the originally fixed bug is broken again.
You thought it was making progress. In reality it was spinning in circles, losing a bit each lap.
This is not hypothetical — it is an experimentally verified finding from the July 2026 arXiv paper *Looping Is Not Reliability* (arXiv:2607.24604) by a team from Alibaba Cloud and Hong Kong University of Science and Technology (Qiang Yang). The controlled study used 900 three-round revision trajectories (5 seeds) to test one question: when coding agents repeatedly fix bugs, are they accumulating correctness or losing it?
The counterintuitive, precise answer: under repeated revision, the rate of "was ever correct" rose from 82.0% to 85.3%, but the rate of "currently still correct" fell from 82.0% to 67.3%. The agent had fixed the bug before — and later fixed it back.
1. The Core Experiment: 900 Trajectories
The design: 30 HumanEval problems, 5 random seeds, a 7B coding agent performing three forced-revision rounds each — 900 three-round trajectories total. Each round the agent sees test results and continues revising.
Two key metrics:
- Current correctness: whether the *final* version passes tests
- Ever-correct: whether *any* round passed tests
- Current traces: 113/135 error starts repaired (83.7%)
- Stale traces: 105/135 (77.8%)
- Unavailable evidence: 97/135 (71.9%)
- Current traces: only 4/135 re-broke fixed bugs (3.0%)
- Stale traces: 34/135 re-broke fixed bugs (25.2%)
- 30 HumanEval problems are entry-level; production SWE-bench tasks are far more complex
- Forced revision is artificially triggered; real agents decide when to stop. A prospective 540-rollout policy experiment eliminated correct-start harm but also reduced wrong-start repair, failing the joint criterion
- Repository experiments (24 real bugs, 4 coder stacks) hit a floor effect — bugs too hard for all agents
- StateSeal is a reference implementation, not a production system
The gap between them is the fraction of "fixed correctly once, then broken again."
2. Verdict: Correctness Is Not an Absorbing State
| Revision round | current correctness | ever-correct | |---------|---------------------|--------------| | Round 1 | 82.0% | 82.0% | | Round 2 | 67.3% | 84.7% | | Round 3 | 69.3% | 85.3% |
By round 3, 16.0% of trajectories had produced a correct patch and then lost it. With current binary evidence (only telling the agent whether the current version passes), regression dropped to 8.0% — still not zero.
This directly refutes the default assumption that "once the agent fixes it correctly, the correctness is preserved." Correctness in agent systems is not an absorbing state — it is a reversible state that the next revision round can undo.
RQ2: Evidence Helps Revision but Hurts Retention
Three evidence conditions on the 7B model:
Evidence genuinely helps revision — but the cost appears elsewhere.
RQ3: Stale Evidence Resurrects Fixed Bugs
In a 14B replication experiment starting from *already-fixed* states:
A 22.2 percentage-point gap, 95% CI [8.9, 37.0], Holm-corrected p=0.0337 — statistically significant.
Stale evidence resurrects fixed bugs. The agent reads old test outputs, believes a bug still exists, "fixes" it, and reverts already-correct code. This is dangerous because many production memory mechanisms store historical test outputs for agents — if those outputs don't update with the code, they are time bombs.
RQ4: Verifier Quality ≠ Independence
A common belief is that stronger verifiers make loops more reliable. This paper disagrees: using three verifiers from different families (Qwen 7B, Qwen 14B, DeepSeek 6.7B), the authors found cross-family diversity does not equal verifier quality. A strong verifier may share blind spots with the coder model. Verifier quality and verifier independence are distinct attributes.
3. The Solution: Evidence-Bound Typed Loop Contract
The paper proposes StateSeal, a reference implementation of an evidence-bound typed loop contract with five core designs:
1. State-bound evidence: every test evidence is bound to a code-state hash; stale evidence auto-invalidates 2. Typed revision actions: revisions are typed actions (fix tests, fix implementation, fix interfaces), each with distinct admission and retention rules 3. Last-known-good checkpoint: correct versions are saved automatically, enabling rollback 4. Risk- and dependence-aware stopping: stopping evaluates "risk vs. benefit of continuing," not just "tests pass" 5. Bundled admission: admission considers impact on retention and other components, not just the current test
StateSeal is not another agent framework — it is a mechanically enforceable contract any existing coding agent can adopt.
4. Why It Matters
For Agent Evaluation
Benchmarks like SWE-bench and HumanEval only measure final correctness — a fatal blind spot that cannot see code fixed correctly mid-trajectory and later broken. The 85.3% vs. 69.3% gap is another instance of the evaluation blind-spot law: what you measure gets optimized; what you don't measure hides problems.For Agent Architecture
Mainstream coding agents (SWE-agent, OpenHands, CodeAct) assume the generate-test-revise loop converges to correctness. This paper shows loops themselves provide no reliability guarantee — the transition kernel may converge upward or diverge downward. A loop without retention mechanisms and stopping rules is little better than a random walk.Granularity Isomorphism
The paper echoes prior work (Heddle, CodeRescue): agent system optimization must move from single-call granularity to trajectory granularity. "Tests pass" on one call ≠ fixed at trajectory level; "strong verifier" ≠ independent verifier; "useful evidence" ≠ harmless evidence.5. Honest Limitations
6. Final Thoughts
Human engineers also revert fixes — but we have git, diff, and rollback. An agent memory that stores only the latest version is an engineer without git. StateSeal's last-known-good checkpoint is essentially git for agents. The reliability problem in the agent era is not an AI problem — it is an engineering problem.
> Paper: arXiv:2607.24604 > Code: StateSeal reference implementation mentioned in the paper; no public link on the arXiv page yet
FAQ
Q1: Who is this for? Practitioners, researchers, and students interested in AI, machine learning, and deep learning.
Q2: Key takeaways? Correctness in agent loops is a reversible state: 16% of correctly fixed bugs were re-broken; stale evidence significantly resurrects fixed bugs (3.0% → 25.2%).
Q3: Is there open-source code? See the links in the body.