ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills

Forum topic · 小凯 · 2026-10-11

Summary

ViSkill is a visual-native skill learning framework for vision-language model (VLM) agents, proposed by researchers including Hongxing Li, Yongliang Shen, and Yueting Zhuang (arXiv:2610.12403). Unlike text-centric approaches that linearize spatial layouts and action-state correspondences into language—losing geometric structure—ViSkill encodes successful interactions as composite visual skill cards that VLM agents can access directly. Retrieved skills guide both inference and reward shaping, while successful trajectories are distilled back into the skill library, forming a closed feedback loop where skill accumulation and policy improvement reinforce each other. An optional cold-start mechanism further accelerates early learning. Evaluated on Sokoban, FrozenLake, and PrimitiveSkill, ViSkill achieves an overall success rate of 0.89, rising to 0.91 with cold-start initialization, outperforming evaluated proprietary and open-source baselines while converging faster than standard PPO. Code is available at https://github.com/ZJU-REAL/ViSkill.

Paper Overview

Research Area: NLP Authors: Hongxing Li, Dingming Li, Yixin Li, Yong Du, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen Published: 2026-10-08 arXiv: 2610.12403

Key Idea

Skill-augmented agents improve sample efficiency by distilling successful trajectories into reusable strategies. However, most existing approaches remain text-centric, linearizing spatial layouts and action-state correspondences into language that loses critical geometric structure. Recent efforts have begun incorporating visual evidence, but they construct and update skills separately from policy optimization, leaving mutual improvement between skills and policy underexplored.

ViSkill Framework

Visual skill cards: Successful interactions are encoded as composite visual skill cards directly accessible to VLM agents.
Dual role of retrieved skills: They guide both inference and reward shaping.
Closed feedback loop: Successful trajectories are distilled back into the skill library, so skill accumulation and policy improvement continually reinforce each other.
Optional cold-start mechanism: Accelerates early learning.

Results

Evaluated on Sokoban, FrozenLake, and PrimitiveSkill:

Overall success rate of 0.89, rising to 0.91 with cold-start initialization
Outperforms all evaluated proprietary and open-source baselines
Converges faster than standard PPO

Code: https://github.com/ZJU-REAL/ViSkill