Learning to Reason by Analogy: Retrieval-Augmented Reinforcement Fine-Tuning (RA-RFT) Boosts Math Reasoning in LLMs

Forum topic · 小凯 · 2026-06-14

Summary

Researchers Zilin Xiao, Qi Ma, and Chun-cheng Jason Chen propose RA-RFT (Retrieval-Augmented Reinforcement Fine-Tuning), a post-training framework that teaches language models to reason by analogy rather than relying on conventional semantic-similarity retrieval. The method trains a retriever via gold-relevance distillation to rank retrieved contexts by expected reasoning benefit instead of semantic overlap, then fine-tunes the policy model with reinforcement learning using retrieved analogous demonstrations under verifiable outcome rewards. Analysis of retrieved context diversity shows reasoning-aware retrieval surfaces complementary solution strategies, providing different reasoning scaffolds for the same problem. On challenging math reasoning benchmarks, RA-RFT consistently outperforms standard reinforcement fine-tuning: on AIME 2025 (average@32 accuracy), it beats GRPO by 7.1 and 2.8 percentage points for Qwen3-1.7B and Qwen3-4B respectively, suggesting reasoning-aware retrieval is a complementary improvement dimension orthogonal to reward design or training curricula. Paper: arXiv 2506.10670 (June 13, 2025).

Paper Overview

Field: NLP Authors: Zilin Xiao, Qi Ma, Chun-cheng Jason Chen Published: 2025-06-13 arXiv: 2506.10670

Abstract (translated)

Retrieval-augmented generation (RAG) has become a standard mechanism for grounding language models in external knowledge, yet conventional retrieval based on lexical or semantic similarity is poorly suited for complex reasoning tasks: a semantically similar problem may demand an entirely different solution strategy, while a superficially different problem may share the same underlying reasoning pattern.

The authors propose Retrieval-Augmented Reinforcement Fine-Tuning (RA-RFT), a post-training framework that teaches language models to reason by analogy. RA-RFT works in two stages:

1. Reasoning-aware retrieval: Gold-relevance distillation trains a retriever that ranks contexts by expected reasoning benefit rather than semantic overlap. 2. Reinforcement fine-tuning: The policy model is fine-tuned with retrieved analogous demonstrations, learning to exploit reasoning trajectories under verifiable outcome rewards.

The authors further analyze the diversity of retrieved contexts, finding that reasoning-aware retrieval uncovers complementary solution strategies, providing different reasoning scaffolds for a single problem.

Results

On challenging math reasoning benchmarks, RA-RFT consistently outperforms standard reinforcement fine-tuning. On AIME 2025 (average@32 accuracy), RA-RFT surpasses GRPO by:

+7.1 percentage points with Qwen3-1.7B
+2.8 percentage points with Qwen3-4B

This indicates that reasoning-aware retrieval is a complementary improvement dimension, orthogonal to advances in reward design or training curricula.

--- *Auto-collected on 2026-06-14*