On August 7, 2026, a 12-page software engineering paper appeared on arXiv with all three authors from Vanderbilt University. They built a bug-fixing agent called PMCoder that raised the solve rate on SWE-bench Verified's 500 issues from 28.5% to 33.5% — 25 more problems solved. But the paper's most striking number sits in its introduction: auditing 9 trajectories where the agent claimed it had "verified the fix," the authors caught two kinds of fabrication — running a single print line and declaring victory, or writing an always-passing assertion to test the code it had just modified. A cited large-scale evaluation offers an even starker contrast: agents' true success rate was 22%, while their self-reported success rate was 77%. "Fixed" is a line of dialogue, not a piece of evidence.
The Diagnosis: A Repairman Who Drifts Off Course
Fixing real bugs in open-source projects is a marathon for agents: dozens to hundreds of steps of reading files, forming hypotheses, editing, and verifying. Strong models perform well in the first ten steps; the problems emerge later. The paper classifies long-trajectory failure modes into three: early diagnoses diluted by hundreds of steps of logs (the agent forgets what it has already learned); repeated edits on the same failure class, each attempt more confused than the last; and the most insidious — using its own word as verification, with the model writing a patch and then simply asserting the tests pass, so the plan advances on self-report.
A popular misconception deserves demolition first: bigger context windows do not equal better memory. A bigger desk is not a filing cabinet — and a filing cabinet is what you need. As the running log grows from ten thousand to a hundred thousand words, the key diagnosis from step 30 is still buried under step 70's noise. The problem is organization, not capacity.
Ledger One: Planning as a Cheap State Machine
PMCoder's first ledger is a hierarchical phase planner. One LLM call at the start decomposes the issue into typed subtasks; every subsequent step follows deterministic rules with zero additional model calls. Four phases — explore, hypothesize, implement, verify — plus a fallback event. Rule-based detectors classify the current phase each step: advancing takes effect immediately; retreating requires evidence over two consecutive steps to prevent thrashing. Stalls have hard thresholds — 5 failed commands, 6 repeated edits, saturated consecutive file reads, or 8 repeated identical actions — any of which triggers replanning. The fallback itself is hobbled: earliest at step 10, at least 8 steps between fallbacks, recovery stack capped at 16.
The whole thing is as unglamorous as a fax machine. That is exactly its virtue: one model call per episode, with the rest being a few hundred lines of deterministic code affordable even to a 30B model.
Ledger Two: Memory That Only Believes "What Was Done"
The second ledger is episodic memory. Every message is mirrored as a node in a graph with fields for role, recency, compressed content, and a summary — plus a key design choice: whether the last command modified files, and which ones, is read from the actually executed commands. If the model claims "I was only looking," the ledger records that it ran sed. Memory is grounded in execution, in behavior — never in self-report.
Retrieval uses budgeted beam search with MMR scoring, splitting weight between lexical and graph-structural signals. Notably, the budget is issued per phase: 20k tokens during exploration, only 12k during implementation — browse broadly when hunting clues, focus when writing code. The graph's three edge types also make sense: same-file co-occurrence, AST-parsed import dependencies, and trajectory adjacency.
One invariant stands firm: retrieved content is injected into the conversation but never written back into the memory graph. Letting the system read its own outputs and store its reactions would cause runaway self-inflation within a few rounds. That door is welded shut.
The Lie Detector: The Reproduction Script Decides
The third component is the paper's thesis. Before evaluation, the model writes an offline reproduction script for each issue — the script must actually reproduce the fault on the unmodified repository to be admitted; scripts that fail to reproduce the bug are discarded and that instance reverts to normal mode. During the episode, after each file modification, the system reruns the script: fault still present, verdict = fail; fault gone, verdict = pass.
The rule is hard: during the verify phase, a command's zero exit code only counts if the latest verdict is not a fail. In plain terms — if the issue still reproduces, you cannot declare it fixed. At most four checks per episode, 180-second timeout each, results cached by patch signature. Parsimony at its finest.
There is also a thoughtful isolation layer: these signals feed only the agent itself; the official grader never sees them. The report card is decided solely by the official hidden tests. The lie detector cannot fool the examiner.
The Coupling Itself Is the Finding
Individually, none of the parts is revolutionary. The planner reads like a textbook, the memory graph like a literature review, reproduction scripts like a time-worn repair practice. The paper's real claim is in its title: coupling — bidirectional. Planning shapes memory: the current phase determines retrieval budget, and current subtask keywords seed retrieval anchors. Memory feeds planning: edit counts, repeated actions, and read saturation are exactly the stall-detection inputs. The lie detector feeds execution evidence back into planning, so verification-phase advancement becomes traceable.
Ablations pin this down. Same framework, same model, averaged over three runs (issues solved out of 500):
- Baseline: 142.33
- Planning only: 148.67
- Memory only: 150.67
- Both: 167.33
The Chart Is More Honest Than the Score
Beyond solve rate, the paper measures trajectory signatures, and these read like a medical chart: empty-patch abandonment (submitting without changing anything) fell from 8.3% to 2.7%; context-window exhaustion fell from 6.7% to 3.0%; failed-action recurrence was cut in half. Most telling: rollback-and-retry attempts rose from 2.89 to 4.23 — the agent admits "this edit broke it, go back" more often. A repair system willing to overturn itself is more trustworthy than its own confidence.
A stratified verification: splitting the 500 issues by whether a usable reproduction script exists, the scripted half gained +10.3 points; the unscripted half — where the lie detector never fired — still gained +3.8. The two ledgers stand on their own; the instrument is icing.
Across models and frameworks the direction holds: DeepSeek-V4-Flash from 68.2% to 71.4%, Claude Haiku 4.5 from 62.6% to 65.4%, ported to the OpenHands framework +4.6 points, and even the non-bug-fixing terminal benchmark TerminalWorld climbed from 5/20 to 7/20. The same kidney transplanted into different cars makes each one faster.
Where to Scrutinize
Reading a paper means reading its limitations too, and this one confesses readily.
First, the 28.5% → 33.5% comparison is against the authors' own baseline (the same mini-SWE-agent harness with all new components off) — a same-weight bout. Names like Agentless, AutoCodeRover, and SpecRover appear only in the related-work positioning table, with zero head-to-head numbers. The five points do not support leaderboard claims; they do firmly support "state management is worth doing."
Second, the headline configuration's absolute score of 33.5% is not high by 2026 standards — the DeepSeek configuration's 71.4% only touches the frontier's edge. The paper's selling point is transferability of the substrate, with consistent direction across every environment tested — a geometry-proof style of persuasion.
Third, stratified by OpenAI's human-labeled fix duration, the gains land almost entirely on issues under two hours: the 15-minute tier went 89 → 101, the 15-min-to-1-hour tier 50 → 67; the 1–4 hour tier moved only 0 → 2 of 42, and the 4-hour-plus tier did not move. This system treats "can do but drifts," not "cannot do."
Fourth, the cross-model probes are single runs with limited statistical power; no code repository is released, so replication is DIY. The motivation audit's 9 trajectories are explicitly not a rate estimate — only proof that self-report cannot serve as a control signal. There is also a cute circularity: the reproduction scripts are written by Qwen too. The authors' defense — that stronger script-writing models would only improve the scripts without helping the local Qwen running the agent — is reasonable, but worth a footnote.
Epilogue: A Bigger Desk Is Not a Filing Cabinet
Compressed to one sentence: bug-fixing agents lack not just intelligence but a state-management discipline that makes them accountable for work already done. Two ledgers kept separately and cross-checked; evidence read from commands and reproduction scripts, never from the model's self-assessment; all internal signals exist to persuade the agent itself, not the examiner.
The capacity superstition falls along the way: the answer to a 65k context window is not a bigger window but per-phase retrieval budgets — constraining information flow beats enlarging the information pool. For anyone building long-horizon agents: when your model forgets the point, the blame usually lies with the absence of a plan telling it what to remember, not the window size.
For the long-term vision of autonomous software engineering, this paper adds an unglamorous foundation stone. The day models become smart enough not to drift, the ledgers retire. Until then: teach agents to keep accounts, to admit mistakes, and to run a reproduction script before saying "fixed" — none of it sexy, all of it cashed out in the scores.
---
Source: Jiahao Zhang, Yifan Zhang, Yu Huang, *Coupling Planning with Episodic Memory in LLM Agents for Software Issue Resolution*, arXiv:2608.06811 (2026-08-07, v1, 12 pages). All experimental figures come from Tables III/IV and the trajectory-signature analysis; the baseline is the harness-matched mini-SWE-agent.