Key points
- Problem: Context learning—extracting rules, workflows, and causal knowledge from long, dense, specialized documents and reasoning with them—is extremely hard. On CL-bench (500 complex documents, 1,899 task questions, 31,607 verification criteria), even GPT-5.1 scores only 21.1%, Claude Opus 4.5 21.0%, and Kimi K2.5 19.2%.
- Two bottlenecks: (1) human annotation of skills from long documents is cognitively demanding and economically infeasible; (2) there is no external feedback signal (unlike code or math) to verify whether extracted skills are correct or complete.
- Challenger — generates targeted questions aimed at the Reasoner's unmastered knowledge points, each with grading rubrics. It maintains its own skill library on how to probe blind spots and evolves when its questions are answered too easily.
- Reasoner — holds the deliverable: a Markdown skill file (initially empty) containing rules, workflows, and caveats distilled from the document. It answers questions using the current skill set.
- Judge — a strictly binary referee: pass/fail against all rubrics, with no explanation. This deliberate silence is what creates work for the next two agents.
- Proposer — batch-diagnoses sets of failed (or successful) cases, asking what they have in common and what the skill file lacks, then issues a "prescription": which skills to add, merge, or remove.
- Generator — executes the prescription, actually editing the skill file.
- GPT-4.1 + Ctx2Skill (16.5%) surpasses un-augmented Gemini 3 Pro (15.8%)—well-designed skills can close model capability gaps.
- LLM-judged skill quality (GPT-4.1 as judge) beats prompting and AutoSkill4Doc baselines on all five dimensions: conciseness 85.2, faithfulness 84.8, clarity 96.2, effectiveness 90.5, reusability 92.5.
- Cross-model transfer: skills generated by GPT-5.1 bring GPT-4.1 to 16.1% (near its self-generated 16.5%); GPT-4.1 skills lift GPT-5.1 to 23.1%. Generate once with a strong model, empower many weaker ones.
- Si et al. (2026). "From Context to Skills: Can Language Models Learn from Context Skillfully?" arXiv:2604.27660
- Si et al. (2024). "CL-Bench: A Benchmark for Context Learning"
- Code: https://github.com/S1s-Z/Ctx2Skill
The Ctx2Skill framework: adversarial self-play as education
Core intuition: the best way to learn is to teach. Five agents form a closed loop:
The Proposer/Generator decoupling is a key design: merging them into one agent costs 0.6–0.7% performance in ablations.
Adversarial collapse and Cross-Time Replay
Self-play has a classic trap: if the Challenger finds a question type the Reasoner always misses, it hammers that type, and the Reasoner's skill file becomes bloated with hyper-specific tricks at the expense of general knowledge. This drift is invisible inside the loop, since the Judge only scores current-round questions.
Cross-Time Replay counters this with a time dimension:
1. Collect probes: each round, automatically save the hardest failure and easiest success. 2. Historical replay: after self-play ends, test every iteration's skill set against all probes. 3. Select the best via multiplicative scoring: ρ_hard × ρ_easy, penalizing over-specialized skill sets that ace hard questions but drop easy ones.
Ablations: removing Cross-Time Replay drops accuracy from 16.5% → 14.7% (GPT-4.1) and 25.8% → 23.0% (GPT-5.1). Notably, GPT-4.1's Iter-1 skills score 15.9% vs. 14.7% for the final Iter-5 skills—later iterations suffer more collapse, so early skills are often more general.
Results
| Model | Baseline | +Ctx2Skill | Gain | |---|---|---|---| | GPT-4.1 | 11.1% | 16.5% | +48.6% relative | | GPT-5.1 | 21.1% | 25.8% | +22.3% relative | | GPT-5.2 | 18.2% | 21.4% | +17.6% relative |
Why it matters: skills as an interface
Ctx2Skill decouples knowledge from model parameters into inference-time skill augmentation: plain-language Markdown skill files injected into prompts. Unlike fine-tuning, the knowledge is interpretable, editable without retraining, transferable across models, and cheap (pure inference). Unlike RAG, it distills reusable rule-like knowledge rather than retrieving raw text snippets.
Limitations
1. Cost: five-agent, multi-round self-play means substantial API usage. 2. Judge dependency: binary judging is simple, but Judge errors propagate through the loop. 3. Iteration count: only 5 rounds tested; longer schedules are unexplored.
Future directions include shared public skill libraries (like a Hugging Face for skills), combining skill extraction with RAG, and hierarchical skill structures.
Takeaway
Ctx2Skill solves the missing-feedback problem with adversarial self-play: no labels, no ground truth, no external validation—yet two AIs challenging each other surface what actually matters in a document. Cross-Time Replay hedges the collapse risk. As a twist on Feynman's "What I cannot create, I do not understand": *what I cannot challenge, I have not mastered.*
References