Learning to Reason by Analogy via Retrieval-Augmented Reinforcement Fine-Tuning (RA-RFT)

Forum topic · 小凯 · 2026-06-14

Summary

A paper on arXiv (2506.10670) by Zilin Xiao, Qi Ma, and Chun-cheng Jason Chen proposes RA-RFT (Retrieval-Augmented Reinforcement Fine-Tuning), a post-training framework that teaches language models to reason by analogy. Conventional RAG retrieval based on lexical or semantic similarity is poorly suited for complex reasoning, since similar problems may require different strategies while dissimilar ones may share reasoning patterns. RA-RFT trains a retriever with gold-relevance distillation to rank contexts by expected reasoning benefit rather than semantic overlap, then fine-tunes the policy model via reinforcement fine-tuning to exploit retrieved analogous reasoning traces under verifiable outcome rewards. The authors show that reasoning-aware retrieval surfaces diverse, complementary solution strategies. On challenging math benchmarks, RA-RFT consistently outperforms standard reinforcement fine-tuning: on AIME 2025 average@32 accuracy, it improves over GRPO by 7.1 and 2.8 points for Qwen3-1.7B and Qwen3-4B respectively, indicating reasoning-aware retrieval is an improvement dimension orthogonal to reward design or training curricula.

Paper Overview

Field: NLP Authors: Zilin Xiao, Qi Ma, Chun-cheng Jason Chen Published: 2025-06-13 arXiv: 2506.10670

Abstract

Retrieval-augmented generation (RAG) has become a standard mechanism for grounding language models in external knowledge, yet conventional retrieval based on lexical or semantic similarity is poorly suited for complex reasoning tasks: a semantically similar problem may demand an entirely different solution strategy, while a superficially different problem may share the same underlying reasoning pattern. We propose Retrieval-Augmented Reinforcement Fine-Tuning (RA-RFT), a post-training framework that teaches language models to reason by analogy. RA-RFT uses gold-relevance distillation to train a retriever that ranks contexts by expected reasoning benefit rather than semantic overlap, and then fine-tunes the policy model via reinforcement fine-tuning methods with retrieved analogous demonstrations, so the model learns to leverage reasoning traces under verifiable outcome rewards. The authors further analyze the diversity of retrieved contexts and find that reasoning-aware retrieval surfaces complementary solution strategies, providing different reasoning scaffolds for a single problem.

Key Results

On challenging mathematical reasoning benchmarks, RA-RFT consistently outperforms standard reinforcement fine-tuning methods.
On AIME 2025 average@32 accuracy, RA-RFT improves over GRPO by 7.1 points (Qwen3-1.7B) and 2.8 points (Qwen3-4B).
This suggests reasoning-aware retrieval is a complementary improvement dimension, orthogonal to advances in reward design or training curricula.

---

*Auto-collected on 2026-06-14.*