> You ask a coding agent to fix a bug. It writes a patch, runs the tests, tests pass. You say "do another revision round." In the second round it changes something else, tests still pass. You check the final code and find the originally fixed bug is back. > > You thought it was making progress. Actually it was spinning in circles, dropping a bit more each lap.
This is not a thought experiment — it is the measured result of a paper called *Looping Is Not Reliability* on arXiv (July 2026). The team is from Alibaba Cloud plus HKUST (Qiang Yang). They ran a sealed five-seed controlled experiment with 900 three-round revision trajectories, asking one question: when coding agents repeatedly fix bugs, are they accumulating correctness or leaking it?
The answer is counterintuitive and precise: under repeated revision, the "ever correct" rate rose from 82.0% to 85.3%, but the "currently still correct" rate fell from 82.0% to 67.3%. The agent fixed the bug once — then fixed it right back into existence.
1. The Core Experiment: 900 Sealed Trajectories
The design is rigorous: 30 HumanEval problems, 5 random seeds, a 7B coding agent performing three rounds of forced revision per problem — 900 three-round trajectories total. Each round the agent sees test results and keeps editing.
Two key metrics:
- current correctness: does the final version pass tests
- ever-correct: did any round pass tests
- current traces: test output of the current code
- stale traces: test output from an earlier version (not refreshed)
- unavailable evidence: no test output given
- with current traces: only 4/135 re-broke a fixed bug (3.0%)
- with stale traces: 34/135 re-broke a fixed bug (25.2%)
- "Tests passed" at call level ≠ "fixed correctly" at trajectory level
- "Strong verifier" at call level ≠ "independent verifier" at trajectory level
- "Useful evidence" at call level ≠ "evidence that doesn't harm" at trajectory level
- 30 HumanEval problems are entry-level; production SWE-bench tasks are far more complex.
- Forced revision is human-triggered; real agents decide when to stop themselves. Section 5.6 admits this and runs a prospective 540-rollout policy experiment — it eliminated correct-start harm, but wrong-start repair also dropped, failing a joint criterion.
- Repository experiments (24 real bugs, 4 coder stacks) hit a floor effect — bugs too hard for any agent, so revision trajectories were indistinguishable.
- StateSeal is a reference implementation, not a production system; no large-scale production numbers are given.
The gap between the two is exactly the "fixed once, broken later" fraction.
2. Verdict: Correctness Is Not an Absorbing State
RQ1: Correctness is not absorbing
| Round | current correctness | ever-correct | |-------|--------------------:|-------------:| | 1 | 82.0% | 82.0% | | 2 | 67.3% | 84.7% | | 3 | 69.3% | 85.3% |
By round 3, 16.0% of trajectories had produced a correct patch and then lost it. Adding "current binary evidence" (only telling the agent whether the current version passes) reduced the loss rate to 8.0% — still not zero.
This directly refutes the default assumption that "once the agent fixes it correctly, that correctness is preserved." It is not. In agent systems, correctness is a reversible state, and the next revision can undo it.
RQ2: Evidence helps revision but hurts retention
RQ2 splits "evidence" (what test feedback the agent sees) into three kinds:
On the 7B model, current traces fixed 113/135 erroneous starting points (83.7%), stale traces fixed 105/135 (77.8%), no evidence fixed 97/135 (71.9%). Evidence does help revision — but the cost shows up on the other side.
RQ3: Stale evidence resurrects fixed bugs
The sharpest finding. In a 14B-model replication starting from already-correct states:
A 22.2 percentage-point gap, 95% CI [8.9, 37.0], Holm-corrected p=0.0337 — statistically significant.
Stale evidence resurrects fixed bugs. The agent looks at old test output, believes the bug still exists, "fixes" it, and reverts already-correct code.
This matters because many production systems' "memory" mechanisms store historical test outputs for the agent. If those outputs are not refreshed as code changes, they are time bombs.
RQ4: Verifier quality ≠ independence
A community consensus says: "the stronger the verifier, the more reliable the loop." This paper says: no.
Using three verifiers from different families (Qwen 7B, Qwen 14B, DeepSeek 6.7B) for cross-validation, the authors find that cross-family diversity does not equal verifier quality. A verifier that performs well on an independent test set may still share blind spots with the coder model, failing to provide independent judgment inside the loop.
"Verifier quality" and "verifier independence" are different properties; neither substitutes for the other.
3. The Solution: Evidence-Bound Typed Loop Contract
The paper goes beyond diagnosis with StateSeal, a reference implementation of an "evidence-bound typed loop contract." Five core designs:
1. State-bound evidence: every piece of test evidence is bound to a specific code-state hash; stale evidence is automatically invalidated. 2. Typed revision actions: revisions are typed actions (fix tests, fix implementation, fix interface, etc.), each with its own admission and retention rules. 3. Last-known-good checkpoint: once a correct version is produced, it is saved as a checkpoint that later revisions can roll back to. 4. Risk- and dependence-aware stopping: stop not merely when tests pass, but after weighing "risk vs benefit of continuing." 5. Bundled admission: admission considers not just whether this round's tests pass, but effects on retention and other components.
StateSeal is not another agent framework — it is a mechanically enforceable contract that any existing coding agent can plug into.
4. Why It Matters
For agent evaluation
Coding-agent benchmarks (SWE-bench, HumanEval) almost universally measure only final correctness. The paper exposes a fatal blind spot: they cannot see code that was fixed mid-trajectory and then broken again. The ever-correct 85.3% vs current 69.3% gap is another instance of the evaluation blind-spot law — you optimize what you measure, and problems hide where you don't measure.
For agent architecture
Mainstream coding agents (SWE-agent, OpenHands, CodeAct) assume the generate-test-revise loop converges to correctness given enough rounds. This paper says: the loop itself provides no reliability guarantee. The transition kernel decides the direction — it can converge upward or diverge downward. A loop without retention mechanisms and stopping rules is no better than a random walk.
The granularity-matching principle
The paper echoes earlier work (Heddle, CodeRescue): the optimization granularity of agent systems must move from "single call" to "trajectory" level.
The optimization granularity should match the granularity of the object being optimized — a general design principle this paper nails down with 900 trajectories.
5. Honest Assessment
Limitations are real:
6. A Closing Thought
The paper suggests a broader question: in agent systems, "once achieved" and "currently retained" are different properties.
Human engineers also reintroduce bugs — but we have git, we can diff, we can revert. An agent memory that stores only the latest version is an engineer without git: when it breaks something, there's no way back. StateSeal's last-known-good checkpoint is essentially git for the agent. The minimum bar for production coding agents may not be a stronger model, but basic engineering hygiene — state binding, typed actions, checkpoints, stopping rules. Not fancy AI, just Software Engineering 101.
And precisely because it's SE 101, it's worth saying: the reliability problem in the agent era is not an AI problem — it's an engineering problem.
> Paper: arXiv:2607.24604 > Code: StateSeal reference implementation mentioned in the paper; no public link yet on the arXiv page > TL;DR: Agents make bugs multiply — 16% of once-correct fixes were reverted by round 2, and stale evidence pushed already-fixed-bug resurrection from 3.0% to 25.2%.