Three Weeks, 2,903 Commits: An AI Agent Taught Itself to Refactor — Then Someone Closed the Door
*English summary of a zhichai.net forum post (original in Chinese), covering a CodeScene AI-refactoring case study and QGIS's QEP428 AI ban.*
Key points
- CodeScene's AI refactoring run: a decompiled *Street Fighter III: Third Strike* codebase (~300,000 lines of C) was processed by an AI agent over three weeks: 2,903 commits, 726 files changed, 252,055 lines rewritten, CodeHealth score raised from 5.6 to 10.0, at roughly $4,000 in token cost.
- Why it worked — deterministic verification: two mechanisms gave the agent a self-readable feedback loop:
- CodeScene's own CodeHealth MCP Server provided an immediate, deterministic score after each transformation.
- A replay trace test framework hashed every frame of the fighting game; after each change, state hashes were compared frame-by-frame against a baseline. Any mismatch meant rollback — "identical every frame" was the acceptance bar, not a passing test suite.
- Adam Tornhill (30 years in large-scale systems) called it the first time he'd seen "superhuman AI performance" at scale.
- Supporters: Paolo Perrone argued frame-accurate replay verification is a far higher bar than "tests are green." Mats Iremark (CTO, Omda Response) said CodeScene MCP plus current agents is "practically cheating."
- Scope skeptics: Konrad Otrębski asked whether changes were actually merged. Engineer Daniel Webb answered: 54 pull requests merged into a versioned main branch. Otrębski countered it should be tried on something like Grafana and merged into master.
- Methodological critiques: Tracy Bannon objected to calling the result "perfect." Denis Baltor questioned the recipes — DRY concerns duplicated *knowledge and intent*, not visually similar lines. Asko Nõmm noted Claude Code and Codex are tuned for their own models, so attribution between model and test harness is unclear.
- Open questions from the authors themselves: non-functional improvements (frame rate, memory, input latency) are unresolved — a performance specialist is being brought in. Whether the harness can catch subtle frame-timing regressions has no clear answer. Crucially: the cited 70-percentage-point defect reduction and 45% token waste reduction are extrapolated estimates from earlier research, not measurements from this case. Only the $4,000 and 21 days are measured.
- Next step: the team will work with Lund University, having students implement features in both the 5.6-score and 10.0-score functionally equivalent versions with frontier models, comparing cost and quality.
- CodeScene figures come from InfoQ's October 12 report and CodeScene's official case page; all performance numbers are CodeScene's own, not independently reproduced.
- The 70-point defect reduction and 45% token waste reduction are company-stated extrapolations from earlier research, not measured in this case.
- The ~68/19 vote count is a community sentiment read; formal ballot tallies were not confirmed.
- The claim about a lost high-value contributor is Dawson's personal statement; the individual and details were not made public.
- Autocomplete and translation exemptions were discussed but are not authorized in the merged text.
The agent wrote its own playbook
The team did not supply a refactoring catalog. The output was 22 refactoring recipes with 82 supporting notes, including non-generic, codebase-specific techniques: shared index ranges (for duplicate loops differing only in start/end), action parameters (for duplicate control structures differing only in the called function), and uniform stride tables (rewriting dissimilar calls into table-dispatched lookups). Failed transformations were also logged. The manual was a byproduct, not a design document.
| Dimension | Figure | |---|---| | Codebase | ~300,000 lines of C | | Duration | 3 weeks | | Files touched | 726 | | Commits | 2,903 | | Lines rewritten | 252,055 | | CodeHealth | 5.6 → 10.0 | | Token cost | ~$4,000 | | Self-derived recipes | 22 + 82 notes |
The community argued about conditions, not numbers
The same week, QGIS closed the door
On October 7, 2025, the QGIS project merged QEP428, titled "No AI contributions," into master. QGIS previously used an LLVM-style policy (QEP-408): AI allowed with disclosure tagging and human accountability. Core maintainer Nyall Dawson's accounting: about 50 PRs carried the AI-assist tag, roughly 40 were closed unmerged, some were closed without even a tag. He called QEP-408 a policy that "clearly didn't work" and said the project lost at least one highly valuable contributor in the process.
The merged text is hard-line: no LLM-, AI-, or agent-assisted code contributions to qgis/qgis; violations lead to reverts and contributor-rights sanctions. Voting was overwhelming — core maintainers Even Rouault, alexbruy, signedav, and elpaso voted yes; community sentiment ran roughly 68 👍 to 19 👎.
| | QEP-408 | QEP-428 | |---|---|---| | Stance | Allowed with disclosure | Banned outright | | Proposed by | LLVM template | Nyall Dawson | | Merged | — | 2025-10-07 | | Real-world data | ~50 tagged PRs | ~40 closed unmerged | | Enforcement | Tag + review | Integrity pledge + post-hoc revert |
Scope disputes outweighed the stance dispute. Contributor kannes noted "assisted" and "used" are broad enough to cover LLM code review or learning an unfamiliar codebase. 02JanDal listed edge cases: ML-ranked IDE completions, brainstorming designs in a chatbot, accidentally opening a search engine's AI summary, translating a term with Google Translate. Dawson's answer: close the door first, then discuss per-case exceptions for autocomplete and translation — admitting they may be "throwing out the baby with the bathwater," but "that baby might not have survived anyway." Meanwhile, maintainer haubourg pointed out an irony: every security advisory received since the program launched had been written with AI assistance.
Reading the two together
Within one week, one project demonstrated that an agent can refit 300,000 lines in three weeks; another project banned AI-assisted contributions outright. Both point to the same place: what made the 300k-line transformation work was a scoring ruler plus a frame-accurate equivalence check. QGIS lacks such a second deterministic acceptance channel, so facing human-authored codebases of 50k+ lines, it chose throttling. QEP428 may become a template for other projects — its evidence (50 tagged PRs, one lost core contributor) is an experience most maintainers can recognize.