[论文] Reinforcement Learning Enables Scaling Generalist Robot Policy Improve...

论文概要

研究领域: ML 作者: Michal Korniak, Marco Bagatella, Ralf Römer, Michal Nauman, Yarden As, Pranav Vignesh, Manan Agarwal, Leander Diaz-Bone, Piotr Zalewski, Mikołaj Czarnecki, Lenart Treven, Shashwat Saxena, Sreyas Venkataraman, Sarvesh Patil, Zhiyuan Zhou, Daniel Marta, Oier Mees, Nur Muhammad Mahi Shafiullah, Carmelo Sferrazza, Bhavya Sukhija, Angela P. Schoellig, Andreas Krause, Max Simchowitz, Andrew Wagenmaker 发布时间: 2026-11-04 arXiv: 2611.06905

中文摘要

通用机器人策略的引入使单个大型模型能从大规模预训练语料中学习执行多种任务。虽然这些模型通常展现强大的零样本泛化能力，但部署中往往需要适应阶段才能达到足够性能。通过强化学习（RL）进行迭代自我改进已显示实现这种适应的潜力，但此类方法主要在单任务设置中探索——目标仅是改进一个狭窄部署任务——与基础策略的多任务能力形成鲜明对比。本工作追问：通用机器人策略能否在部署中实现多任务、通用性改进——同时针对众多不同任务——以及什么算法配方能实现。为此，我们首先开发结构化评估测试通用改进能力，系统评估价值估计、策略提取和资源分配的算法与工程设计选择。我们表明：通用策略不仅可在不同任务上在线改进，正确配方还能实现泛化，提升在线训练未覆盖任务的表现。我们刻画了泛化发生的条件，表明训练任务分布和价值型 RL 的使用都至关重要。据我们所知，这是机器人基础模型在线通用改进的首批证据之一。

原文摘要

The introduction of generalist robot policies has enabled a single, large model to learn to perform a variety of tasks from a vast pre-training corpus. While these models often exhibit powerful zero-shot generalization capabilities, they typically require an adaptation phase to achieve sufficient performance in deployment. Iterated self-improvement via reinforcement learning (RL) has demonstrated promise in enabling this adaptation, yet such approaches have been largely explored in single-task settings - where the goal is to simply improve on a narrow deployment task - which lies in stark contrast to the multi-task capabilities of the base policy. In this work we ask whether generalist robot policies are capable of multi-task, generalist improvement in deployment - targeting many diverse t...

#论文 #arXiv #ML #小凯