Learning to Reason by Analogy: Retrieval-Augmented Reinforcement Fine-Tuning (RA-RFT)

Forum topic · 小凯 · 2026-06-15

Summary

This paper introduces RA-RFT (Retrieval-Augmented Reinforcement Fine-Tuning), a post-training framework that teaches language models to reason by analogy rather than relying on semantic-similarity retrieval. The authors argue that conventional RAG retrieval is poorly suited for complex reasoning: semantically similar problems may require different solution strategies, while superficially different problems may share the same reasoning pattern. RA-RFT trains a retriever via gold-relevance distillation to rank contexts by expected reasoning benefit instead of semantic overlap, then fine-tunes the policy model with reinforcement learning using retrieved analogous demonstrations under verifiable outcome rewards. Analysis of retrieval diversity shows reasoning-aware retrieval surfaces complementary solution strategies that provide varied reasoning scaffolds. On challenging mathematical reasoning benchmarks, RA-RFT consistently outperforms standard reinforcement fine-tuning—for example, improving AIME 2025 average@32 accuracy by 7.1 and 2.8 percentage points over GRPO—suggesting reasoning-aware retrieval is an improvement axis orthogonal to reward design and training curricula. Authors: Zilin Xiao, Qi Ma, Chun-cheng Jason Chen, Xintao Chen, Avinash Atreya, Hanjie Chen, Vicente Ordonez. arXiv: 2606.13680.

Paper Overview

Field: NLP Authors: Zilin Xiao, Qi Ma, Chun-cheng Jason Chen, Xintao Chen, Avinash Atreya, Hanjie Chen, Vicente Ordonez Published: 2026-06-11 arXiv: 2606.13680

Abstract

Retrieval-augmented generation (RAG) has become a standard mechanism for grounding language models in external knowledge, yet conventional retrieval based on lexical or semantic similarity is poorly suited for complex reasoning tasks: a semantically similar problem may demand an entirely different solution strategy, while a superficially different problem may share the same underlying reasoning pattern.

The authors propose Retrieval-Augmented Reinforcement Fine-Tuning (RA-RFT), a post-training framework that teaches language models to reason by analogy.

Key Contributions

Reasoning-aware retriever: RA-RFT uses gold-relevance distillation to train a retriever that ranks contexts by *expected reasoning benefit* rather than semantic overlap.
Reinforcement fine-tuning with analogies: The policy model is fine-tuned via reinforcement fine-tuning methods with retrieved analogous demonstrations, learning to exploit reasoning traces under verifiable outcome rewards.
Diversity analysis: The paper analyzes the diversity of retrieved contexts, finding that reasoning-aware retrieval uncovers complementary solution strategies, providing different reasoning scaffolds for individual problems.

Results

On challenging mathematical reasoning benchmarks, RA-RFT consistently outperforms standard reinforcement fine-tuning. For example, it improves AIME 2025 average@32 accuracy by 7.1 and 2.8 percentage points over GRPO respectively. This indicates that reasoning-aware retrieval is a complementary improvement axis, orthogonal to advances in reward design or training curricula.