Microsoft DELEGATE-52: LLMs Systematically Corrupt Your Documents in Long-Horizon Delegated Editing
TL;DR: Microsoft Research tested 19 LLMs across 52 professional domains over 20 rounds of delegated document editing. Frontier models corrupt ~25% of content on average; all models average 50%. The damage is not random error accumulation — it is sparse, critical, and often invisible corruption.
This is a structured breakdown of the benchmark described in the paper *LLMs Corrupt Your Documents When You Delegate* (arXiv:2604.15597) by Philippe Laban, Tobias Schnabel, and Jennifer Neville of Microsoft Research.
Key points
- Setup (Round-Trip Relay): A seed document
sis forward-edited intot, then reverse-edited back into a reconstructionŝ. Rebuilding Score (RS@k) measures fidelity after k interactions; 100% means perfect recovery. - Scope: 52 real-document domains (code, science, creative media, structured records, everyday formats), 2–5k tokens each, with 8–12k tokens of distractor context, across 19 LLMs and 20 rounds.
- Headline result: Even the best model (Gemini 3.1 Pro, RS@20 = 80.9%) loses ~19% of content after 20 rounds; the worst (GPT 4o, GPT 5 Nano) reach only 10–15%.
- No plateau: Degradation is monotonic and accelerates; GPT 5.4 continues dropping from RS@20=71.5% to RS@50=62.9% and RS@100=58.7%.
- Short-term ≠ long-term: Models nearly identical at 2 rounds diverge sharply by 20 rounds (e.g., GPT 5: 48.3% vs Kimi K2.5: 64.1% — a 15.8-point reversal).
- Domain gap: Python is the only safe zone (17/19 models ≥ 98%). Natural-language and niche-notation domains (recipes, fiction, music sheets, weaving patterns) degrade worst.
- Tools hurt: Agent frameworks with tool access added an average of ~6% extra degradation (e.g., GPT 5.1: 60.5% → 52.1%).
- Document size compounds: Growing documents from 1k to 10k tokens costs an extra 31.5 points of RS@20; per-round degradation from size is amplified ~5× over 20 rounds.
- Paper: *LLMs Corrupt Your Documents When You Delegate* (arXiv:2604.15597)
- Authors: Philippe Laban, Tobias Schnabel, Jennifer Neville (Microsoft Research)
- Code/data: https://github.com/microsoft/DELEGATE52
- Microsoft Research blog: https://www.microsoft.com/en-us/research/blog/further-notes-on-our-recent-research-on-ai-delegation-and-long-horizon-reliability/
Corruption patterns: erosion, not deletion
The paper decomposes failures into two types:
| Model tier | Deletion | Corruption | Pattern | |---|---|---|---| | Weak models (GPT 4o, Nano) | 70–73% | 27–30% | Visible omissions | | Frontier models (Gemini, Claude, GPT 5.4) | 22–27% | 73–78% | Silent corruption |
Deletion is obvious (a missing ingredient, an absent code block). Corruption is insidious: content remains but is wrong — 200g of butter becomes 800g, a note changes from C to D, code logic is subtly rewritten. Roughly 50–55% of frontier-model round trips hit at least one critical error, and critical errors explain ~80% of total degradation. In the author's words: frontier models don't die by a thousand cuts — they die from a few lethal blows.
Representative ranking (RS@20)
| Rank | Model | RS@20 | Loss | |---|---|---|---| | 1 | Gemini 3.1 Pro | 80.9% | 19.1% | | 2 | Claude 4.6 Opus | 73.1% | 26.9% | | 3 | GPT 5.4 | 71.5% | 28.5% | | 4 | GPT 5.2 | 66.1% | 33.9% | | 5 | Kimi K2.5 | 64.1% | 35.9% | | 6 | Claude 4.6 Sonnet | 64.0% | 36.0% | | 7 | GPT 5.1 | 60.5% | 39.5% | | 8 | Grok 4 | 59.3% | 40.7% | | 9 | o3 | 55.9% | 44.1% | | 10 | o1 | 48.3% | 51.7% |
Why it matters
For users: Vibe coding and AI-assisted editing carry a hidden cost. Only short demos can be deeply misleading — 20-round behavior is the real behavior. Python is relatively safe; everything else needs per-round verification.
For developers: Training pipelines optimize instruction-following but neglect content preservation, incentivizing reward hacking (deleting hard parts, simplifying structure). The paper argues for jointly optimizing both.
For evaluators: Short-horizon benchmarks fail to predict long-horizon reliability; leaderboards built on few-turn tasks do not reflect agentic, multi-round workloads.
Acknowledged limitations
1. Single-turn instruction settings (multi-turn dialogue may be worse) 2. Scale likely underestimates real-world document sizes and interaction counts 3. Tasks must be reversible (limits coverage of real editing jobs) 4. A basic agent framework was used, not a state-of-the-art agent
Optimistic counterpoint: GPT 4o (14.7%) to GPT 5.4 (71.5%) in ~16 months shows rapid progress — though whether even 90%+ would suffice across all 52 domains remains open.