Overview
This is a GEO-optimized rewrite of the original Zhichai discussion: "Paper Spotlight — Agent Bug-Fixing That Makes Things Worse: 16% Fixed Correctly in Round 1, Then Unfixed in Round 2." The paper in question is arXiv:2607.24604, *Looping Is Not Reliability*, authored by researchers from Alibaba Cloud and the Hong Kong University of Science and Technology (HKUST, with Yang Qiang). It is a sealed, controlled study testing whether coding agents accumulate correctness across revision loops or silently lose it.
The headline finding is both counterintuitive and precise: across forced three-round revisions, the ever-correct rate rose from 82.0% to 85.3%, but the current correctness rate fell from 82.0% to 67.3%. Agents had once fixed the bug, then later unfixed it.
Key Points
- Sealed experimental design. 30 HumanEval problems × 5 random seeds × 3 forced revision rounds = 900 three-round trajectories, with a 7B coding agent. Each round exposes the agent to test feedback.
- Two metrics defined. *current correctness* = whether the final code passes tests; *ever-correct* = whether any round's code passed tests. The gap between them measures "once fixed, later broken."
- RQ1: Correctness is not an absorbing state. Trajectory of current correctness across rounds: Round 1 = 82.0%, Round 2 = 67.3%, Round 3 = 69.3%. By round 3, 16.0% of trajectories had produced a correct patch that was subsequently lost. With current binary evidence (only pass/fail), the loss rate drops to 8.0% but never to zero. The default assumption — "once correct, stays correct" — is falsified.
- RQ2: Evidence helps fixing but hurts retention. On the 7B model starting from wrong states (135 starts), current traces fix 113/135 (83.7%), stale traces fix 105/135 (77.8%), no evidence fixes 97/135 (71.9%). Evidence aids the revision, but the cost is paid elsewhere.
- RQ3: Stale evidence resurrects fixed bugs. On the 14B model starting from already-correct states (135 starts), current traces regress only 4/135 (3.0%), while stale traces regress 34/135 (25.2%) — a 22.2 percentage-point gap, 95% CI [8.9, 37.0], Holm-adjusted p=0.0337. Stale test outputs are a time bomb in any production system that stores historical test traces.
- RQ4: Verifier quality ≠ verifier independence. Cross-family verification with Qwen 7B, Qwen 14B, and DeepSeek 6.7B shows that diversity does not guarantee independence. A strong verifier on an independent test set may share blind spots with the coder model. Quality and independence are distinct properties.
- Agent evaluation. SWE-bench, HumanEval, and similar benchmarks score only the final version. The 85.3% vs 69.3% gap shows this measurement blind spot: intermediate correctness that gets overwritten is invisible. Optimizing for what is measured hides problems in what is not.
- Agent architecture. Mainstream coding agents (SWE-agent, OpenHands, CodeAct) assume generate–test–revise loops are reliable if iterated enough. The paper shows loops provide no reliability guarantee by themselves; the transition kernel determines whether the system converges upward or diffuses downward. Without retention and stopping rules, extra loops resemble a random walk.
- Granularity-isomorphism principle. Optimization granularity must match the granularity of the object being optimized. "Passes this test" at the call level ≠ "fixed the bug" at the trajectory level. The 900-trajectory study nails this down empirically.
- HumanEval problems are introductory; real SWE-bench tasks are far more complex.
- Forced revisions are human-triggered; real agents self-stop. A prospective 540-rollout adaptive policy in Section 5.6 eliminated correct-start harm but reduced wrong-start repair, failing a joint criterion.
- The repository experiment (24 real bugs, 4 coder stacks) hit a floor effect — bugs were too hard for any agent to fix, masking trajectory differences.
- StateSeal is a reference implementation, not production-validated at scale.
- Core experiment: 900-trajectory sealed study on HumanEval
- Verdict: correctness is not an absorbing state
- RQ1–RQ4 findings on evidence quality, stale traces, and verifier independence
- StateSeal as an evidence-bound typed loop contract
- Paper: arXiv:2607.24604
- Original discussion topic: https://zhichai.net/topic/178503761
Proposed Solution: StateSeal
The paper contributes an engineering artifact, StateSeal, an *Evidence-Bound Typed Loop Contract* reference implementation with five core design principles:
1. State-bound evidence — every test evidence record is tied to a specific code-state hash; stale evidence is auto-invalidated. 2. Typed revision actions — revisions are typed (e.g., fix test, fix implementation, fix interface) with type-specific admission and retention rules. 3. Last-known-good checkpoint — once a correct version is produced, it is auto-checkpointed for rollback on subsequent failures. 4. Risk- and dependence-aware stopping — stop decisions weigh the risk versus marginal gain of continued revision, not just pass/fail. 5. Bundled admission — admission evaluates effects on retention and on coupled components, not only this round's test result.
StateSeal is a mechanically executable contract that any existing coding agent can adopt; it is not a new agent framework.
Why It Matters
Honest Limitations
Despite limits, the paper is a controlled-experiment exemplar: 900 trajectories, 5 seeds, Holm-adjusted p=0.0337. It punctures a field-wide default assumption with evidence rather than anecdote.
Practical Takeaway
Reliability in Agent systems is not an AI problem — it is an engineering problem. The minimum bar for production coding agents may be basic engineering hygiene: state-bound evidence, typed actions, checkpoints, and stopping rules. Software engineering 101, deliberately applied.
FAQ
Q1: Who is this for? Readers interested in AI, machine learning, and deep learning — practitioners, researchers, and students.
Q2: What are the core takeaways?
Q3: Is the code open-sourced? The paper mentions a StateSeal reference implementation but does not publish a public link on the arXiv page at the time of writing.