[论文] Route-and-Generate: KL-Regularized Router Improvement for Reasoning Ro...

论文概要

研究领域: ML
作者: Tien-Phat Nguyen, Truong Nguyen, Thanh-Thien Le, Hoang Phan, Thien Huu Nguyen, Linh Ngo Van et al.
发布时间: 2026-10-24
arXiv: 2610.30351

中文摘要

大语言模型常不加区分地动用复杂推理：简单任务上过度思考，难题上又思考不足。本文提出 Route-and-Generate（Rouge）分层训练框架，让每个 LLM 自适应决定每个输入需要何时、多少推理。方法把推理精力的分配与答案生成解耦：引入一个路由策略，在多种推理格式间分配固定采样预算，基座模型用 GRPO 按 per-sample 奖励优化；为鼓励高效推理，采用偏向更短正确解的 rank-based 奖励，并通过闭式 KL 正则策略改进步骤把更新蒸馏进一个神经路由器。这一设计让容易题的答案更简洁，把长推理留给难题，在多个推理基准上取得更好的准确率–效率权衡。

原文摘要

Large language models often apply complex reasoning indiscriminately, leading to overthinking on simple tasks and underthinking on harder ones. We present Route-and-Generate (Rouge), a hierarchical training framework that enables LLMs to adaptively decide when and how much reasoning is required for each input. Our approach decouples reasoning effort allocation from answer generation by introducing a routing policy that allocates a fixed sampling budget across multiple reasoning formats, while the base model is optimized with GRPO using per-sample rewards. To promote efficient reasoning, we employ a rank-based reward that favors shorter correct solutions, and update the routing policy via a closed-form KL-regularized policy improvement step distilled into a neural router. This design encourages concise answers on easy problems while reserving longer reasoning for harder ones, achieving improved accuracy-efficiency trade-offs across multiple reasoning benchmarks.

自动采集于 2026-10-24

#论文 #arXiv #ML #小凯