Overview
Paper: *From Context to Skills: Can Language Models Learn from Context Skillfully?* (Ctx2Skill) Authors: Shuzheng Si, Haozhe Zhao, Yu Lei, Qingyi Wang, Dingwei Chen, Zhitong Wang, Zhenhailong Wang, Kangyang Luo, Zheng Wang, Gang Chen, Fanchao Qi, Minjia Zhang, Maosong Sun Institutions: Tsinghua University, DeepLang AI, UIUC, Fudan University, CUHK Published: arXiv:2604.27660 (v1 2026-04-30, v2 2026-05-03) Code: https://github.com/S1s-Z/Ctx2Skill
---
The Problem: Context Learning, Not Just Long-Context Reading
Modern LLMs struggle when given a novel, complex context outside their pre-training distribution (e.g., an unfamiliar technical manual, experiment data, domain paper). The challenge is context learning: inducing new knowledge from the context and applying it to solve tasks, rather than simple retrieval or reading comprehension.
Two existing approaches hit hard limits:
- Manual annotation is prohibitively expensive for long, dense documents.
- Automated construction lacks feedback: unlike code or math, context-learning tasks have no ground truth to evaluate whether extracted skills are complete or faithful.
- Hard probe Q^h: per iteration, pick the failing tasks that passed the fewest rubrics (hardest failures).
- Easy probe Q^e: per iteration, pick the successful tasks that passed the fewest rubrics (easiest successes).
- 500 complex contexts
- 1,899 tasks
- 31,607 verification rubrics
- Four categories: Domain Knowledge Reasoning, Rule System Application, Procedural Task Execution, Empirical Discovery & Simulation
- Average context length 10.4K tokens, max 65K tokens
- 51.1% of tasks are multi-turn sequential (later steps depend on earlier solutions)
- Prompting (single-shot skill generation): GPT-4.1 only +1.2%; some categories regress.
- AutoSkill4Doc (windowed extraction): GPT-4.1 +2.1%, far below Ctx2Skill's +5.4%.
- Challenger too strong → Reasoner keeps failing → Challenger-side evolution (too many successes) → Challenger rebalances.
- Challenger too weak → Reasoner keeps succeeding → Reasoner-side evolution stalls → Challenger-side evolution (too many successes) forces harder tasks.
- Si, S. et al. (2026). *From Context to Skills: Can Language Models Learn from Context Skillfully?* arXiv:2604.27660.
- Dou, S. et al. (2026). *CL-Bench: A benchmark for context learning.* arXiv:2602.03587.
- Yang, Y. et al. (2026). *AutoSkill: Experience-driven lifelong learning via skill self-evolution.* arXiv:2603.01145.
- Zhang, H. et al. (2026). *CoEvoSkills: Self-evolving agent skills via co-evolutionary verification.* arXiv:2604.01687.
> "Given only the context, there is no external feedback signal to tell whether a generated skill is useful or whether critical knowledge has been omitted."
---
Core Idea: Multi-Agent Self-Play Skill Evolution
Instead of relying on human labels, Ctx2Skill lets the model play against itself: one agent generates tasks, another solves them, a judge scores the results, and losing side reflects and upgrades its skills.
Five Roles
| Role | Responsibility | Evolution | |------|----------------|-----------| | Challenger | Generates probe tasks and rubrics from the context | Evolves task-generation strategy | | Reasoner | Solves tasks with current skill set | Evolves solving skills | | Judge | Binary pass/fail verdict | Fixed (GPT-5.1); no evolution | | Proposer (one per side) | Diagnoses failures/successes, proposes high-level improvements | Updated each round | | Generator (one per side) | Translates proposals into concrete skill edits (add/delete/merge) | Updated each round |
Workflow per Iteration
1. Challenger (skill set S^C_{i-1}) produces M tasks with rubrics. 2. Reasoner (skill set S^R_{i-1}) attempts to solve them. 3. Judge splits outcomes into failure set F_i and success set P_i. 4. Failure set → Reasoner Proposer diagnoses missing knowledge → Reasoner Generator updates S^R_i. 5. Success set → Challenger Proposer diagnoses how to make tasks harder → Challenger Generator updates S^C_i. 6. Iterate under strict adversarial isolation (sides cannot peek at each other's skills).
Key design: The Challenger also evolves. If it stayed static, tasks would become too easy and the Reasoner would stop exposing knowledge gaps.
---
Cross-Time Replay: Preventing Adversarial Collapse
Risk
Pure self-play tends toward adversarial collapse: the Challenger invents increasingly pathological tasks; the Reasoner's skills overfit to those edge cases; redundancy accumulates; generalization collapses.
Solution
Cross-Time Replay does not simply take the final-iteration skill set. It searches all historical iterations and picks the most balanced one.
Probe sets (built without external supervision):
Selection criterion (product form with Laplace smoothing):
S^R_* = argmax_i ( ρ^h(i) · ρ^e(i) )
where ρ^h(i) is the solve rate of iteration-i skills on the hard probe, and ρ^e(i) on the easy probe. The product punishes any skill set that sacrifices easy tasks to brute-force hard ones, or vice versa.
Empirical validation: Fixed-iteration skill performance drops monotonically (Iter-1: 15.9% → Iter-5: 14.7%). Cross-Time Replay reaches 16.5%, surpassing every fixed iteration.
---
Experiments
Benchmark: CL-Bench
Main Results
| Model | Baseline | + Ctx2Skill | Δ | |-------|----------|-------------|---| | GPT-4.1 | 11.1% | 16.5% | +5.4% | | GPT-5.1 | 21.1% | 25.8% | +4.6% | | GPT-5.2 | 18.2% | 21.4% | +3.2% |
Compared with alternatives:
Headline finding: GPT-4.1 + Ctx2Skill (16.5%) outperforms skill-less Gemini 3 Pro (15.8%). Skills can bridge raw capability gaps between models.
Skill Quality (GPT-4.1 as judge)
| Dimension | Prompting | AutoSkill4Doc | Ctx2Skill | |-----------|-----------|---------------|-----------| | Conciseness | 81.2 | 81.3 | 85.2 | | Faithfulness | 79.7 | 81.4 | 84.8 | | Clarity | 80.0 | 92.4 | 96.2 | | Effectiveness | 83.3 | 88.7 | 90.5 | | Reusability | 84.7 | 87.2 | 92.5 |
Self-play skills are not just more effective; they are also more structured, human-readable, editable, and reusable.
Ablations (GPT-4.1 overall)
| Variant | Score | Δ vs full | |---------|-------|-----------| | Full Ctx2Skill | 16.5% | — | | – w/o Challenger evolution | 13.8% | −2.7% (largest drop) | | – w/o Cross-Time Replay | 14.7% | −1.8% | | – w/o Hard Probe | 15.2% | −1.3% | | – w/o Easy Probe | 15.7% | −0.8% | | – w/o Laplace Smoothing | 15.5% | −1.0% | | – Proposer+Generator merged | 15.9% | −0.6% |
Skill Transferability
| Skill source | Reasoner model | Result | |--------------|----------------|--------| | GPT-5.1 skills → GPT-4.1 | GPT-4.1 | 16.1% (≈ GPT-4.1's own 16.5%) | | GPT-4.1 skills → GPT-5.1 | GPT-5.1 | 23.1% (< GPT-5.1's own 25.8%) |
Asymmetry: Strong-model skills transfer well to weaker models; weak-model skills help strong models only marginally. This hints that skill discovery itself may be an emergent capability.
---
Analytical Insights
Why natural-language skills, not memory or parameters?
1. Interpretability: skills are human-readable Markdown, auditable and editable. 2. Transferability: plug-and-play across models. 3. No parameter access required: friendly to closed-source APIs.
This is more practical than parameter-internalization approaches like SKILL0/SkillRL in commercial API settings.
Meta-stability of self-play
The system forms a self-correcting loop:
Cross-Time Replay keeps either side from drifting to extremes.
Minimal viable feedback
Without ground truth, binary pass/fail under adversarial pressure is a weak but sufficient learning signal. The system does not need to know the correct answer; it only needs to know whether the answer cleared all rubrics, and changes in pass-rate drive skill improvement.
Why smaller relative gains for stronger models
GPT-4.1 sees +48.6% relative lift, GPT-5.1 +21.8%, GPT-5.2 +17.6%. Stronger models already internalize general problem-solving strategies, so the bottleneck shifts from "missing strategies" to "precise knowledge extraction." Even so, +4.6% on a 21% baseline is significant.
Comparison with LongNAP
| Dimension | LongNAP (2603.05923) | Ctx2Skill (2604.27660) | |----------|----------------------|------------------------| | Data source | Passive mobile screenshots + clicks | Active, given context | | Learning goal | Predict next user action | Extract reusable skills for tasks | | Feedback | Delayed (compare with later user action) | Binary Judge (pass/fail) | | Training | Policy Gradient + LLM-as-judge | Self-play + text editing | | Core innovation | Retrieving user history | Adversarial evolution + cross-time replay |
Both point to the same direction: let AI self-evolve through environmental interaction rather than human annotation.
---
Limitations and Outlook
| Limitation | Notes | |------------|-------| | Compute cost | 5 rounds × 5 tasks × multiple agents per context → heavy API usage; larger N/M not explored due to budget | | Judge dependency | Binary verdicts inherit GPT-5.1 biases | | Context assumption | All required knowledge assumed inside the given context; no external retrieval | | Multi-turn dependency | 51.1% sequential tasks need prior solutions, but skills are static and not dynamically adapted to dialog history | | Collapse risk | Cross-Time Replay mitigates but does not eliminate adversarial collapse |
Application prospects: enterprise knowledge bases (auto-extract operating procedures), research assistants (induce methodology from papers and data), education (extract problem-solving strategies from textbooks).
---
Takeaways
Ctx2Skill contributes more than a benchmark number. It offers a general unsupervised self-evolution framework:
1. Multi-agent adversarial co-evolution creates sustained optimization pressure. 2. Failure-driven diagnose-then-fix (Proposer-Generator) beats single-shot generation. 3. Cross-Time Replay picks the best historical skill set and prevents collapse. 4. Plug-and-play natural-language skills transfer across models and work with closed-source APIs.
> "We hope Ctx2Skill provides a practical and scalable paradigm for equipping language models with the ability to learn skillfully from complex, previously unseen contexts."
Context learning moves from hand-engineering to automated evolution.
---