Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Agents

Forum topic · 小凯 · 2026-07-28

Summary

This arXiv paper (2607.22529) introduces Skill Self-Play (Skill-SP), a co-evolutionary self-play framework for improving large language models through interaction-driven self-evolution. It addresses a core dilemma in self-evolutionary LLM training: environment-bound methods offer precise verification but narrow task diversity, while open-ended self-generation broadens the task space but lacks reliable rewards. Skill-SP treats agent skills as a middle ground — each skill provides deep, verifiable execution in a specific scenario, while dynamic routing across skills preserves task variety. The framework consists of a proposer that generates challenging tasks from dynamically sampled skills, a solver that explores candidate solutions, and a dynamic skill controller that collects execution feedback to update and expand the skill library, all orchestrated through a reinforcement learning loop. Empirical results on tool-use and reasoning benchmarks show Skill-SP consistently raises the performance ceiling of competent backbone models and can even reverse initially misaligned models.

Overview

Field: NLP Authors: Siyuan Huang, Pengyu Cheng, Haotian Liu, Tao Chen, Yihao Liu, Jingwei Ni, Shijie Zhou, Ziyi Yang, Gangwei Jiang, Mengyu Zhou, Yu Cheng, Xiaoxi Jiang, Guanjun Jiang arXiv: 2607.22529

The Problem

LLM training is shifting from manual design and annotation to interaction-driven self-evolution. Existing self-evolutionary methods face a fundamental dilemma between task diversity and verification reliability:

Environment-bound methods obtain precise feedback but confine learning to narrow domains.
Open-ended self-generation broadens the task space but lacks reliable verification, allowing misleading rewards to pollute the training loop.

Key Idea

The authors identify agent skills as a powerful middle ground to reconcile this tension: each skill ensures deep, verifiable execution in a specific scenario, while dynamic routing across skills maintains open-ended task variety.

The Skill-SP Framework

Skill Self-Play (Skill-SP) is a co-evolutionary framework with three components, orchestrated through a reinforcement learning loop:

Proposer — generates challenging tasks based on dynamically sampled skills.
Solver — explores candidate solutions to push its capability boundary.
Skill Controller — collects execution feedback to update and expand the skill library.

These components co-evolve in a continuous self-play loop, effectively bridging the gap between structured verification and open-ended exploration.

Results

Empirical evaluation on tool-use and reasoning benchmarks shows that Skill-SP acts as a robust evolution engine:

Consistently raises the performance ceiling of competent backbone models.
Catalyzes a significant reversal (recovery) in initially misaligned models.

--- *Originally posted on zhichai.net; auto-collected on 2026-07-28.*