[论文] MDSKILL: Learning to Internalize Skill Memory through Multi-View Disti...

论文概要

研究领域: NLP
作者: Dian Jin, Jinluan Yang, Kairong Han, Baohong Li, Zijing Hu, Yi He et al.
发布时间: 2026-11-03
arXiv: 2611.05313

中文摘要

技能记忆把过去经验编码为文本提示，以增强用于自我演化智能体的强化学习（RL）。作为 RL 的补充，在策略自蒸馏（OPSD）提供稠密、细粒度的监督，把技能引导轨迹中的知识内化以加速策略学习。但自生成的记忆常含噪声监督，以及功能上纠缠、冗余或冲突的知识，直接蒸馏困难。在有限训练预算下，这些问题会阻碍知识内化甚至损害性能。为此我们提出 MDSKILL——一种用于技能记忆内化的多视图蒸馏方法。MDSKILL 把技能记忆组织为三个互补的功能视图（状态接地、动作规划、执行控制），用动态指标筛选高质量技能，构建视图专属与整合的推理分支；OPSD 在这些分支之间传递并固化互补经验知识到共享策略，实现高效的技能内化与智能体持续自演化。在三个智能体基准上与不依赖更强外部教师模型的 RL、技能增强和自蒸馏基线对比，MDSKILL 在不增加额外计算开销的前提下，在 ALFWorld 和 WebShop 上分别比 SOTA 提升至多 7.1% 和 8.1%。充分的消融与演化分析进一步验证了其在知识内化和策略改进上的有效性。代码已开源。

原文摘要

Skill memories encode past experiences as textual prompts to improve reinforcement learning (RL) for self-evolving agents. Complementing RL, On-Policy Self-Distillation (OPSD) provides dense, fine-grained supervision to internalize knowledge from skill-guided trajectories and accelerate policy learning. However, self-generated memories often contain noisy supervision and functionally entangled, redundant, or conflicting knowledge, making direct distillation difficult. Under limited training budgets, these issues can hinder knowledge internalization and even degrade performance. To address this problem, we propose MDSKILL, a multi-view distillation approach for skill memory internalization. MDSKILL organizes skill memories into three complementary functional views (state grounding, action planning, and execution control) and selects high-quality skills using dynamic metrics to construct view-specific and integrated reasoning branches. OPSD then transfers and consolidates complementary experiential knowledge across these branches into a shared policy, enabling efficient skill internalization and continuous agent self-evolution. We evaluate MDSKILL on three agent benchmarks against RL, skill-augmented, and self-distillation baselines that do not rely on stronger external teacher models. Without additional computational overhead, MDSKILL improves upon SOTA performance by up to 7.1% and 8.1% on ALFWorld and WebShop. Extensive ablations and evolution analyses further validate its effectiveness in knowledge internalization and policy improvement. The code is available at https://anonymous.4open.science/r/MDSKILL-CF53.

自动采集于 2026-11-05

#论文 #arXiv #AI #小凯