Looping Is Not Reliability: Coding Agents Undo Their Own Bug Fixes

> You ask a coding agent to fix a bug. It writes a patch, runs the tests, tests pass. You say, "do one more revision round." In round two it changes something else; the tests still pass. You inspect the final code — the originally fixed bug is back. > > You thought it was making progress. In reality it is spinning in circles, dropping a little correctness with every turn.

This is not a hypothetical. It is the empirical result of a paper titled "Looping Is Not Reliability" posted to arXiv in July 2026. The author team is from Alibaba Cloud and HKUST (Qiang Yang). They ran a sealed, five-seed controlled study with 900 three-round revision trajectories to answer one question: when coding agents repeatedly fix bugs, are they accumulating correctness or leaking it?

The answer is counterintuitive but precise: under repeated revision, the rate of "ever-correct" rose from 82.0% to 85.3%, while the rate of "currently correct" fell from 82.0% to 67.3%. The agent once fixed the bug — then fixed it back into brokenness.

1. Core Experiment: 900 Sealed Trajectories

The design is rigorous: 30 HumanEval problems, 5 random seeds, and a 7B coding agent performing three rounds of forced revision per problem — 900 trajectories in total. Each round the agent sees the test results and keeps editing.

Two metrics matter:

• Current correctness: whether the final code passes tests
• Ever-correct: whether any round passed tests

The gap between them is the fraction of "fixed once, broken again."

2. The Verdict: Correctness Is Not an Absorbing State

| Revision round | Current correctness | Ever-correct | |---|---|---| | 1 | 82.0% | 82.0% | | 2 | 67.3% | 84.7% | | 3 | 69.3% | 85.3% |

By round three, 16.0% of trajectories had produced a correct patch and then lost it. With "current binary evidence" (telling the agent only whether the current version passes), attrition dropped to 8.0% — still not zero.

This directly refutes a default assumption: once an agent fixes a bug, that correctness persists. It does not. Correctness in agent systems is not an absorbing state — it is a reversible state that the next revision round can undo.

RQ2: Evidence Helps Revision, Hurts Retention

Three evidence regimes on the 7B model:

• Current traces: 113/135 error starts repaired (83.7%)
• Stale traces: 105/135 (77.8%)
• No evidence: 97/135 (71.9%)

Evidence genuinely helps repair — but the cost appears elsewhere.

RQ3: Stale Evidence Resurrects Fixed Bugs

In the 14B replication, starting from already-correct states:

• Current traces: only 4/135 broke a fixed bug (3.0%)
• Stale traces: 34/135 broke a fixed bug (25.2%)

A 22.2-point gap, 95% CI [8.9, 37.0], Holm-corrected p=0.0337 — statistically significant.

Stale evidence resurrects fixed bugs. The agent sees old test output, believes the bug still exists, "fixes" it, and reverts already-correct code. The danger: many production memory mechanisms feed historical test outputs to agents. If those outputs are not updated with code changes, they are ticking time bombs.

RQ4: Verifier Quality ≠ Independence

The paper challenges the consensus that "stronger verifiers make loops more reliable." Cross-validating with three verifier families (Qwen 7B, Qwen 14B, DeepSeek 6.7B), the authors find cross-family diversity is not verifier quality. A strong verifier on an independent test set may still share blind spots with the coder model. Verifier quality and verifier independence are distinct properties and cannot substitute for each other.

3. The Fix: Evidence-Bound Typed Loop Contract (StateSeal)

The paper proposes StateSeal, a reference implementation of an "evidence-bound typed loop contract" with five core elements:

1. State-bound evidence: every piece of test evidence is bound to a specific code-state hash; stale evidence auto-invalidates 2. Typed revision actions: revisions are typed actions (fix tests, fix implementation, fix interface), each with distinct admission and retention rules 3. Last-known-good checkpoint: correct versions are automatically checkpointed and can be rolled back to 4. Risk- and dependence-aware stopping: stopping weighs "risk vs. benefit of further revision," not just "tests pass" 5. Bundled admission: admission considers effects on retention and other components, not just the current test result

StateSeal is not another agent framework — it is a mechanically enforceable contract that existing coding agents can plug into.

4. Why It Matters

For agent evaluation: benchmarks like SWE-bench and HumanEval only score the final version, blind to correctness that was achieved mid-trajectory and then destroyed. The 85.3% ever-correct vs. 69.3% current-correct gap is an evaluation blind spot — what is measured gets optimized; what isn't measured hides the problems.

For agent architecture: mainstream agents (SWE-agent, OpenHands, CodeAct) assume the generate-test-revise loop converges with enough rounds. This paper shows loops provide no reliability guarantee — the transition kernel determines direction, and without retention mechanisms and stopping rules, looping is a random walk.

On optimization granularity: "tests passed on one call" ≠ "fixed at trajectory level"; "strong verifier" ≠ "independent verifier"; "useful evidence" ≠ "harmless evidence." The granularity of optimization should match the granularity of the object being optimized.

5. Honest Assessment

Limitations:

• 30 HumanEval problems are entry-level; real SWE-bench tasks are far more complex
• Forced revision is externally triggered; a prospective 540-rollout adaptive policy experiment eliminated correct-start harm but reduced wrong-start repair and failed the joint criterion (Section 5.6)
• Repository experiments (24 real bugs, 4 coder stacks) hit a floor effect — all agents failed, so revision dynamics couldn't be observed
• StateSeal is a reference implementation, not a production system

Still, as a paper that punctures a field-wide default assumption with 900 trajectories, 5 seeds, and Holm-corrected significance, it sets a standard for controlled experiments.

6. Closing Thought

"Once achieved" and "currently retained" are different properties. Human engineers also break fixed code — but we have git: diff, revert. An agent memory that stores only the latest version is an engineer without git. StateSeal's last-known-good checkpoint is essentially git for agents. Reliability in the agent era is not an AI problem; it is an engineering problem.

> Paper: arXiv:2607.24604 > TL;DR: Under repeated revision, 16% of once-correct fixes were re-broken by round three; stale evidence raised the resurrection rate of fixed bugs from 3.0% to 25.2%.