ROFT: Removing RL From Training Makes Agents Learn Faster Through Self-Retrospection
*Translation and summary of a paper-review post on zhichai.net.*
The Cost Problem With RL
Training a capable agent is expensive: one round of forty GRPO updates costs about $500 in GPU time, and a single SWE-bench Pro evaluation costs $200. Beyond cost, reinforcement learning has a structural weakness: gradients come from within-group comparisons—sampling multiple attempts per task and learning from the reward gap. If every attempt on a task fails, the gap is zero, the gradient is zero, and nothing is learned. The harder the task, the more often this failure mode strikes.
The paper asks a deceptively simple question: humans learn by explaining to themselves what happened after an attempt, regardless of success or failure. Can a model improve purely by writing retrospectives of its own experience—no reward model, no policy gradients?
Answer: yes, and much faster.
The Method
ROFT (Retrospection-Only Fine-Tuning) has four steps:
1. Let the model attempt a software engineering task. 2. Feed it its own trajectory (actions taken, observations made). 3. Have it write a retrospective: key assumptions, mistakes, how to handle similar cases in the future. 4. Use that retrospective as training data for ordinary next-token prediction.
A key design choice: the entire trajectory—task description, code, test outputs—serves only as *context* and never enters the loss function. Gradients flow only through the retrospective text. The model is trained on "explaining this experience," and behavioral improvement is a byproduct of learning to explain.
Results: Less Time, Higher Scores
- Base model Qwen3.5-4B starts at 44.2% on SWE-bench Verified.
- ROFT: 49.2% in 20 updates; GRPO: 48.0% in 40 updates.
- At 40 updates each: ROFT takes 4.32 hours / 6,000 attempts; GRPO takes 8.30 hours / 11,000+ attempts.
- ROFT reached 49.0% in 10 updates (1.29 hours), surpassing GRPO's 40-update result in about one-sixth of the time.
- On Qwen3.5-9B: 58.8% vs 55.8%, 1.93 hours vs 5.33 hours.
- Attribution analysis: comparing per-step likelihoods before and after training, correct steps rise more and incorrect steps fall relatively—retrospection training indirectly performs credit assignment despite never having explicit step-level signals.
- Weighting experiments: upweighting "evidence" and "correction" components (54.4%, 54.2%) beats uniform weighting (50.6%). Content, not form, is doing the work.
Fairness note: the training set was deliberately biased *in favor of GRPO*—only tasks where the base model had both successes and failures were kept, since GRPO needs within-group reward variance for signal. ROFT still won even on GRPO's home turf. Also, GRPO degrades with longer training (48.0% at 40 rounds → 46.0% at 90 rounds, overfitting); ROFT saturates at 20 rounds.
Learning From Total Failure
Some tasks are zero-gradient dead zones for GRPO: the base model failed all 64 attempts. Training ROFT on these tasks individually: SymPy and SymbiFlow went from 0/64 to solvable; Pylint from 2/64 to 12%.
Why does total failure still teach? Because the retrospective's signal comes not from "did it succeed" but from "explain what happened"—which assumption was wrong from the start, where a path broke down. Learning doesn't need success as a seed.
The Signal Was Always in the Experience
Does the model peek at whether its attempt was correct? An ablation closes this loophole: one retrospective version sees the final pass/fail verdict, another sees only the trajectory summary and final patch. Both reach 49.2%. The verdict adds no information—the agent already ran tests during the task, and those outputs sit in the trajectory. RL compresses this into a scalar; ROFT keeps it as language.
Modes matter too: online self-written retrospectives score 49.2%; a frozen retrospective generator with fresh attempts, 47.4%; fully offline with a fixed corpus, lower. Fresh self-interpretation is worth more than stored old interpretation.
Language Is a Steering Wheel
The sharpest experiment changes one sentence in the retrospective prompt: "think about how to solve this problem more directly." Same initialization, data, and objective—no length penalty, no verbosity reward.
Result: 11.7% fewer tokens and 13.1% fewer interaction rounds on downstream tasks, across both solved and unsolved problems. As the paper puts it: retrospection is not just a record of experience—it is a *steerable supervision source*.
Why "Learning to Explain" Becomes "Learning to Act"
Two layers of evidence:
The Bigger Picture
Prior approaches to learning from experience: trajectories in context, replayable simulators, external knowledge bases. ROFT takes a newer path—turning experience into the training target itself, letting explanation enter the weights. The same retrospective text placed in a prompt as reference has mediocre effect; used as gradient data, behavior genuinely changes. Context is inferior to textbook.
For long-horizon agents this is especially valuable: a dozens-of-steps task yields only one sparse outcome that cannot localize which step went wrong. Retrospection preserves that information as language—precisely the medium models learn from best.
---
*Original tags: paper review, zhichai daily, xiaokai.*