Shockingly Simple Self-Retrospection Improves Agentic Models Without Rewards: Retrospection-Only Fine-Tuning (ROFT)

Forum topic · 小凯 · 2026-09-30

Summary

This paper introduces Retrospection-Only Fine-Tuning (ROFT), a deliberately minimal online training procedure for language-model agents. The agent attempts a task, observes feedback, generates a retrospective explanation of its experience, and is fine-tuned with a next-token prediction loss applied only to the explanation tokens—no external teacher and no reward-based policy update are required. In software-engineering experiments with Qwen3.5-4B, ROFT trained on a mix of successful and failed base-model attempts reaches 49.2% and 26.8% resolution rates on SWE-bench Verified and Pro (held-out data, no verifier) after 20 updates, compared with 48.0% and 25.3% for GRPO after 40 updates, while making faster early progress in training time and sampled attempts. ROFT can even learn from tasks where all 64 base-model attempts failed, showing learning can begin without initial success trajectories. Behavioral analysis suggests ROFT implicitly assigns credit to actions, encouraging good ones and discouraging mistakes, and prompting retrospection toward more direct solutions yields shorter trajectories without explicit length penalties. Findings indicate that learning to explain also improves learning to act.

Paper Overview

Field: NLP
Authors: Jonathan Light, Christopher Zhang Cui, Jeonghye Kim, Roger Creus Castanyer, Emiliano Penaloza, Zhengyan Shi, Alessandro Sordoni, Marc-Alexandre Côté, Xingdi Yuan, Minseon Kim
Published: 2026-09-28
arXiv: 2609.35741

Summary

People learn not only by repeating successful actions, but also by recounting and explaining their experiences, revising their understanding to guide future behavior. Can a language-model agent improve its future actions by training only on explanations of its own experience?

The authors investigate this question through Retrospection-Only Fine-Tuning (ROFT) — a minimal online procedure designed to isolate the effect of explanation-only training on subsequent behavior:

1. The agent attempts a task 2. It observes available feedback 3. It generates a retrospective explanation 4. It is fine-tuned with a next-token prediction loss on the explanation tokens alone

The procedure uses neither an external teacher nor a reward-based policy update.

Key Results

In software-engineering experiments with Qwen3.5-4B, ROFT is trained on a mix of successful and failed base-model attempts.
On held-out data, after 20 updates and without a verifier, ROFT achieves 49.2% on SWE-bench Verified and 26.8% on SWE-bench Pro, while GRPO reaches 48.0% and 25.3% after 40 updates. ROFT also shows faster early progress in training time and sampled attempts.
ROFT learns to solve a task on which all 64 base-model attempts failed, showing that learning can begin without any initial success trajectories.
Behavioral analysis finds ROFT implicitly assigns credit to actions, encouraging good actions and discouraging wrong ones.
Prompting retrospection toward more direct solutions leads to shorter trajectories in subsequent attempts, even without an explicit length penalty.

Conclusion

Learning to explain can also improve learning to act: self-generated retrospection is a useful training signal for agentic models, without requiring rewards or a teacher.

--- *Auto-collected on 2026-09-30*

Shockingly Simple Self-Retrospection Improves Agentic Models Without Rewards: Retrospection-Only Fine-Tuning (ROFT)

Summary

Paper Overview

Summary

Key Results

Conclusion

Tags