Learning to Reason by Analogy via Retrieval-Augmented Reinforcement Fine-Tuning (RA-RFT)

Forum topic · 小凯 · 2026-06-15

Summary

This arXiv paper (2606.13680) introduces Retrieval-Augmented Reinforcement Fine-Tuning (RA-RFT), a post-training framework that teaches language models to reason by analogy. Conventional retrieval based on lexical or semantic similarity is poorly suited for complex reasoning, since similar-looking problems may require different strategies while dissimilar ones may share the same reasoning pattern. RA-RFT addresses this with two components: a retriever trained via gold-relevance distillation to rank contexts by expected reasoning benefit rather than semantic overlap, and a policy model fine-tuned with reinforcement learning that leverages retrieved analogous demonstrations under verifiable outcome rewards. The authors also analyze retrieval diversity, showing reasoning-aware retrieval surfaces complementary solution strategies that provide different reasoning scaffolds per problem. On challenging math reasoning benchmarks, RA-RFT consistently outperforms standard reinforcement fine-tuning: on AIME 2025 it improves average@32 accuracy over GRPO by 7.1 and 2.8 percentage points on respective settings, suggesting reasoning-aware retrieval is an improvement axis orthogonal to reward design or training curriculum advances.

Paper Overview

Field: NLP Authors: Zilin Xiao, Qi Ma, Chun-cheng Jason Chen, Xintao Chen, Avinash Atreya, Hanjie Chen, Vicente Ordonez arXiv: 2606.13680

Abstract

Retrieval-augmented generation (RAG) has become a standard mechanism for grounding language models in external knowledge, yet conventional retrieval based on lexical or semantic similarity is poorly suited for complex reasoning tasks: a semantically similar problem may demand an entirely different solution strategy, while a superficially different problem may share the same underlying reasoning pattern.

The authors propose Retrieval-Augmented Reinforcement Fine-Tuning (RA-RFT), a post-training framework that teaches language models to reason by analogy.

Key Components

Reasoning-aware retriever: Trained via gold-relevance distillation to rank contexts by expected reasoning benefit rather than semantic overlap.
Reinforcement fine-tuning: The policy model is fine-tuned using retrieved analogous demonstrations, learning to exploit reasoning trajectories under verifiable outcome rewards.
Retrieval diversity analysis: The paper shows reasoning-aware retrieval reveals complementary solution strategies, providing different reasoning scaffolds for individual problems.

Results

On challenging mathematical reasoning benchmarks, RA-RFT consistently outperforms standard reinforcement fine-tuning. On AIME 2025, it improves average@32 accuracy over GRPO by 7.1 and 2.8 percentage points in the respective settings.

This indicates that reasoning-aware retrieval is a complementary improvement axis, orthogonal to advances in reward design or training curricula.

--- *Auto-collected on 2026-06-15*