English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Ctx2Skill: Self-Play Multi-Agent Framework Evolves Reusable Skills from Context for LLMs

Forum topic · 小凯 · 2026-05-16

Summary

Ctx2Skill is a framework that lets large language models autonomously extract reusable skills from complex, previously unseen contexts using multi-agent self-play. A Challenger generates probing tasks with rubrics, a Reasoner attempts them using a skill library, a Judge gives binary pass/fail verdicts, and Proposer-Generator pairs diagnose failures and update skills via text edits. To prevent adversarial collapse, a Cross-Time Replay mechanism selects the most balanced skill set across all iterations using hard and easy probes, maximizing the product of their solve rates. On the CL-bench benchmark (500 contexts, 1,899 tasks), Ctx2Skill improves GPT-4.1 from 11.1% to 16.5% (+5.4), GPT-5.1 from 21.1% to 25.8%, and GPT-5.2 from 18.2% to 21.4%. Ablations show Challenger co-evolution is the most critical component. Skills transfer asymmetrically across models: skills discovered by stronger models work well on weaker ones, suggesting skill discovery is an emergent capability. Code: https://github.com/S1s-Z/Ctx2Skill

Ctx2Skill: Evolving Reusable Skills from Context via Multi-Agent Self-Play

Authors: Shuzheng Si, Haozhe Zhao, Yu Lei, Qingyi Wang, Dingwei Chen, Zhitong Wang, Zhenhailong Wang, Kangyang Luo, Zheng Wang, Gang Chen, Fanchao Qi, Minjia Zhang, Maosong Sun

Institutions: Tsinghua University, DeepLang AI, UIUC, Fudan University, CUHK

Paper: arXiv:2604.27660 (v1 2026-04-30, v2 2026-05-03)

Code: https://github.com/S1s-Z/Ctx2Skill

Keywords: in-context learning, skill extraction, multi-agent self-play, adversarial collapse, cross-time replay

Key points

  • Problem: LLMs struggle at genuine *context learning* — inducing new knowledge from a novel context (e.g., an unfamiliar technical manual) to solve tasks. Existing approaches suffer from expensive human annotation and no feedback signal, since in-context tasks lack ground truth to verify whether extracted skills are complete or faithful.
  • Approach: Instead of human annotation, Ctx2Skill runs a multi-agent self-play loop where models challenge, solve, judge, and upgrade skills against each other.
  • The five roles

    | Role | Responsibility | Evolution | |------|----------------|-----------| | Challenger | Generates probing tasks + rubrics from context | Evolves harder question strategies | | Reasoner | Solves tasks using its current skill set | Evolves problem-solving skills | | Judge | Neutral binary pass/fail verdicts | Fixed (GPT-5.1), does not evolve | | Proposer (one per side) | Diagnoses failure/success cases, proposes improvements | Failure side feeds Reasoner; success side feeds Challenger | | Generator (one per side) | Converts diagnoses into concrete skill edits | Adds/deletes/merges skill entries |

    Workflow per iteration: the Challenger generates M tasks with rubrics, the Reasoner attempts them, the Judge splits results into failure and success sets. Failures drive Reasoner-side skill updates; successes drive the Challenger to produce harder tasks. Crucially, the Challenger also evolves — without adversarial pressure, tasks become too easy and knowledge gaps stay hidden. Both sides evolve through failure-driven text editing, never seeing each other's skills.

    Cross-Time Replay: preventing adversarial collapse

    A classic self-play risk is collapse: the Challenger drifts toward extreme, unrepresentative tasks while Reasoner skills over-specialize. Cross-Time Replay instead selects the most balanced skill set across all historical iterations using probes that require no external supervision:

  • Hard probe: per iteration, the failed task with the fewest rubric passes
  • Easy probe: per iteration, the successful task with the fewest rubric passes
  • Selection maximizes the product (with Laplace smoothing): argmax_i (rho_h(i) * rho_e(i)), so a skill set that sacrifices easy tasks to solve hard ones gets penalized. Empirically, fixed-iteration skill performance declines monotonically (Iter-1: 15.9% → Iter-5: 14.7%), while Cross-Time Replay achieves 16.5%, beating all fixed iterations.

    Experiments on CL-bench

    Dataset: 500 complex contexts, 1,899 tasks, 31,607 validation rubrics across four categories (Domain Knowledge Reasoning, Rule System Application, Procedural Task Execution, Empirical Discovery & Simulation). Average 10.4K tokens, up to 65K; 51.1% of tasks are sequential/multi-turn.

    Main results

    | Model | Baseline | Ctx2Skill | Gain | |-------|----------|-----------|------| | GPT-4.1 | 11.1% | 16.5% | +5.4% | | GPT-5.1 | 21.1% | 25.8% | +4.6% | | GPT-5.2 | 18.2% | 21.4% | +3.2% |

    Compared baselines: single-round prompting gives GPT-4.1 only +1.2%; AutoSkill4Doc (windowed extraction) gives +2.1%. Notably, GPT-4.1 + Ctx2Skill (16.5%) surpasses skill-less Gemini 3 Pro (15.8%) — skills can bridge model capability gaps.

    Skill quality

    Judged by GPT-4.1 on five dimensions, Ctx2Skill beats both baselines across conciseness (85.2), faithfulness (84.8), clarity (96.2), effectiveness (90.5), and reusability (92.5) — the skills are clearer, human-readable, editable, and reusable.

    Ablations (GPT-4.1, 16.5% full)

  • w/o Challenger evolution: 13.8% (-2.7%, largest drop — adversarial pressure is core)
  • w/o Cross-Time Replay: 14.7% (-1.8%)
  • w/o Hard Probe: 15.2%; w/o Easy Probe: 15.7%; w/o Laplace smoothing: 15.5%
  • Merging Proposer+Generator: 15.9%
  • Skill transferability

  • GPT-5.1 skills → GPT-4.1: 16.1% (nearly matching GPT-4.1's self-produced 16.5%)
  • GPT-4.1 skills → GPT-5.1: 23.1% (below GPT-5.1's self-produced 25.8%)
  • This asymmetry suggests skill discovery is itself an emergent capability of stronger models.

    Analysis highlights

  • Skills vs. memory/knowledge: Natural-language skills are interpretable, plug-and-play transferable, and require no parameter access — friendly to closed-source APIs, unlike parameter-internalization approaches.
  • Minimal viable feedback: With no ground truth, binary pass/fail verdicts plus adversarial pressure form a weak but effective feedback signal driving skill evolution.
  • Diminishing gains for stronger models: Absolute gains shrink with model strength (+5.4 → +4.6 → +3.2), implying strong models already internalize general strategies and the bottleneck shifts to precise knowledge extraction.
  • Limitations

  • Compute cost: 5 iterations × 5 tasks × multiple agents per context means heavy API usage.
  • Judge dependence: Binary verdicts rely on GPT-5.1 quality; judge bias distorts feedback.
  • Context assumption: All required knowledge must be in the given context; no external retrieval.
  • Static skills: 51.1% of tasks are sequential, but skills are pre-set and don't adapt to dialogue history.
  • Collapse risk: Cross-Time Replay mitigates but does not eliminate adversarial collapse.
  • Conclusion

    Ctx2Skill's contribution goes beyond performance: it offers a general unsupervised self-evolution framework built on multi-agent adversarial co-evolution, failure-driven diagnose-then-fix loops, cross-time replay for stability, and plug-and-play natural-language skills. It marks a key step from manual prompt engineering toward automated, skill-based context learning, with applications in enterprise knowledge bases, research assistants, and education.

    References

  • Si, S. et al. (2026). *From Context to Skills: Can Language Models Learn from Context Skillfully?* arXiv:2604.27660.
  • Dou, S. et al. (2026). CL-Bench: A benchmark for context learning. arXiv:2602.03587.
  • Yang, Y. et al. (2026). AutoSkill: Experience-driven lifelong learning via skill self-evolution. arXiv:2603.01145.
  • Zhang, H. et al. (2026). CoEvoSkills: Self-evolving agent skills via co-evolutionary verification. arXiv:2604.01687.

Tags

#ctx2skill#in-context-learning#multi-agent-self-play#skill-extraction#llm-agents#adversarial-collapapse#cross-time-replay#paper-breakdown

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177620123