[论文] Learning to Reason by Analogy via Retrieval-Augmented Reinforcement Fi...

论文概要

研究领域: NLP
作者: Zilin Xiao, Qi Ma, Chun-cheng Jason Chen
发布时间: 2025-06-13
arXiv: 2506.10670

中文摘要

检索增强生成(RAG)已成为将语言模型锚定在外部知识上的标准机制，但传统的基于词汇或语义相似度的检索不适用于复杂推理任务：语义相似的问题可能需要完全不同的解决策略，而表面不同的问题可能共享相同的底层推理模式。我们提出检索增强强化微调(RA-RFT)，一个后训练框架，教会语言模型通过类比进行推理。RA-RFT使用gold-relevance蒸馏训练检索器，按预期推理收益而非语义重叠对上下文排序，然后通过强化微调方法使用检索到的类比演示微调策略模型，使模型学习在可验证结果奖励下利用推理轨迹。我们进一步分析检索上下文的多样性，发现推理感知检索揭示了互补的解决策略，为单个问题提供不同的推理支架。在具有挑战性的数学推理基准上，RA-RFT持续优于标准强化微调方法。例如，在AIME 2025 average@32准确率上，RA-RFT比GRPO分别提升7.1和2.8个百分点（Qwen3-1.7B和Qwen3-4B）——这表明推理感知检索是一个互补的改进维度，与奖励设计或训练课程的进步正交。

原文摘要

Retrieval-augmented generation (RAG) has become a standard mechanism for grounding language models in external knowledge, yet conventional retrieval based on lexical or semantic similarity is poorly suited for complex reasoning tasks: a semantically similar problem may demand an entirely different solution strategy, while a superficially different problem may share the same underlying reasoning pattern. We propose Retrieval-Augmented Reinforcement Fine-Tuning (RA-RFT), a post-training framework that teaches language models to reason by analogy. RA-RFT uses gold-relevance distillation to train a retriever that ranks contexts by expected reasoning benefit rather than semantic overlap, and then fine-tunes the policy model via reinforcement fine-tuning methods with retrieved analogous demonstr...

自动采集于 2026-06-14

#论文 #arXiv #NLP #小凯