[论文] AutoScaffold: AI-Guided Scaffolding for Reinforcement Learning of Lang...

论文概要

研究领域: NLP
作者: Zijian Zhang, Rizhen Hu, Jiangweizhi Peng, Shiyang Li, Sujay Bhatt, Sihan Zeng, Mingyi Hong
发布时间: 2026-11-10
arXiv: 2611.14515

中文摘要

基于可验证奖励的强化学习（RLVR）产生丰富的训练经验，但结果奖励把每次尝试压缩成单个标量，大部分信息被浪费。训练时脚手架（如训练时加入的策略或提示）是把信息转化为引导的自然方式，但其设计与适配目前主要靠人工、且针对特定领域。我们提出 AutoScaffold——统一的 AI-for-AI 框架，把学生训练经验转化为自适应脚手架，几乎不依赖领域信息，训练无需人工介入。冻结的教师模型周期性审查训练经验、诊断瓶颈，以两种形式设计脚手架：(i) 跨任务共享的策略；(ii) 针对单个难题的提示，并随学生学习进度调整支持的内容与强度，评估时移除。我们将同一框架应用于 ALFWorld、搜索式 QA、数学推理与 GPU 内核生成四个领域。无脚手架评估下，AutoScaffold 学生全面超越对应 RL 基线：ALFWorld 成功率 +7.2 分、QA 平均精确匹配 +4.0 分；GPU 内核生成上"正确且快于 PyTorch 参考"任务占比最多 +5.0 分；数学上以约 1/3 的训练步数超过 QuestA 报告的准确率。分析显示教师在每个任务类识别出不同失败模式，随学生进步将脚手架从分步工作流转向针对性修复；修正后的脚手架比沿用旧脚手架或不使用更能帮助学生。

原文摘要

Reinforcement learning with verifiable rewards (RLVR) produces rich training experience, yet outcome rewards reduce each attempt to a single number and leave most of this information unused. Training-time scaffolds, such as strategies or hints added to the prompt during training, offer a natural way to turn this information into guidance, but their design and adaptation are largely left to humans and tailored to specific domains. We introduce AutoScaffold, a unified AI-for-AI framework that turns a student's training experience into adaptive scaffolds, with minimal domain-specific information and no human intervention during training. A frozen teacher periodically examines this experience, diagnoses bottlenecks, and designs scaffolds in two forms: (i) strategies shared across a class of ta...

自动采集于 2026-11-12

#论文 #arXiv #NLP #小凯