1000 Legacy Issues Handed to a Swarm of Agents: 36 Hours Straight, 83% Fixed to Mergeable Quality

> One hundred person-years is a person's entire working life from joining a company to retirement.

That is the estimated workload behind Luanti, the open-source sandbox game. Since its first commit in November 2010, it has accumulated 378,373 lines of code, 18,726 commits from 1,441 contributors — roughly 100 person-years by the COCOMO model. On September 16, Volcano Engine used this codebase as an exam: 1,000 issues picked from real history, covering 13 modules including rendering, networking, and physics, with the official fix patches removed in advance to simulate a repository where no one had fixed anything. The new Doubao-Seed-2.1-pro-0915 model dispatched multiple sub-agents in parallel and ran for nearly 36 hours straight. In the end, 83% of the issues were fixed to a "mergeable" engineering delivery standard.

Key points

• The unit changed. AI coding has long been measured by single-question metrics (completion accuracy, unit-test pass rates, SWE-bench). Here the yardstick is an entire issue queue plus wall-clock time — from "one question" to "one batch," from "one attempt" to "36 hours."
• Three upgrades interlock. Multimodal coding (reading design mockups, screen recordings, 3D scenes; 31% accuracy improvement), long-horizon agent capability (evidence tracing, timeliness judgment, data verification), and token efficiency (30%+ reduction in image/video inference tokens). The third is the quiet enabler: a 36-hour run with hundreds of sub-agents only makes economic sense if per-task token costs fall.
• A well-designed test. Luanti (renamed from Minetest in October 2024, mostly C++ with some Lua) sits at a sweet spot of complexity. Removing official patches means the model could not reverse-engineer answers from upstream commits; the 13-module spread rules out cherry-picking easy bug categories.
• A caveat on "mergeable." This is ByteDance's own judgment: patches reached a completeness suitable for review — not that they were merged, or that maintainers would accept them. Nothing was disclosed about where the 17% failures concentrated.

The ERP demo may matter more

A second demo showed the multimodal entry point's value: given only a PC screen recording, hand-drawn sketches, and 280,000 lines of Java source with no complete documentation, the model untangled an old ERP system's full procurement workflow in under 2 hours, wrote 3 mobile pages (~2,000 lines), and handled field mapping, parameter formats, and null handling on its own — running through the complete order-and-inventory-audit chain. Much real engineering knowledge lives in screenshots, recordings, and operational habits that text-only models cannot access. If "watch the recording + read the sketch" works reliably, legacy undocumented systems gain an entry point for modernization.

What the economics look like

Assume a mid-level engineer needs 3 hours per medium-complexity issue: 1,000 issues ≈ 3,000 person-hours — a small team's work over two or three quarters. The difference between 36 hours and 3,000 person-hours is not speed but parallelism; whether the ledger tips depends on token unit prices and concurrency limits, neither of which was published. The 30% token reduction effectively moves the break-even line for long-horizon jobs.

Five unanswered questions

1. Who performs third-party verification of the 83%? 2. Which modules did the 17% failures concentrate in? 3. Mergeable ≠ merged — how many would maintainers actually accept? 4. Were the 1,000 issues randomly sampled or pre-filtered? 5. Is 36 hours wall-clock time or cumulative compute time? The official materials don't say — and the difference matters enormously.

A deeper concern: 500 agents researching one report and 500 agents editing 500 files in one repository are very different problems (file locks, dependency ordering, conflicting edits). Whether the Luanti issues inherently avoided same-file conflicts is unclear — and if they did, the 83% figure's reproducibility deserves a discount.

When the unit changes, the standard must follow

SWE-bench scores are reproducible because questions, environments, and grading are public. Here, the questions and environment are public, but the grading standard is held by the publisher. The verifiable half makes this more credible than a pure demo; the unverifiable half keeps it short of a true benchmark. What will decide this path's future is whether the first teams to modernize legacy systems this way publish their failure lists, not the next model capability bump. A report card with only a success rate, versus a retrospective with failure samples, differ by an order of magnitude in value to everyone who follows.

References

1. Volcano Engine Doubao model 0915 release notes and official demo cases — as reported by Huanqiu Wang 2. 21st Century Business Herald, "Doubao 2.1 Pro starts 'coding from images,' image inference cost down 30%" — https://news.qq.com/rain/a/20260916A0CIXU00 3. AIBase, "Doubao 2.1 Pro 0915 Released on Huoshan Arks" — https://www.aibase.com/news/31085 4. BlockBeats, "Doubao 2.1 Pro's first major update in three months: 1,000 real issues, 83% mergeable" — https://www.theblockbeats.info/flash/367392 5. DAMO Developer Matrix AI news daily, 2026-09-17 — https://damodev.csdn.net/6aab20dacf836948fcd2879b.html