Ctx2Skill: Turning Long Documents into Reusable Skills via Multi-Agent Self-Play

TL;DR: Give an AI a 100-page technical document and it can answer specific questions — but rephrase the question and it fails. Ctx2Skill lets three AI agents play an adversarial "ask–answer–grade" game against themselves, automatically distilling the document's rules, procedures, and decision criteria into a reusable skill book. No human annotation, no external feedback — only the document itself.

Paper: *From Context to Skills: Can Language Models Learn from Context Skillfully?* (arXiv:2604.27660), by a team from Tsinghua University, DeepLang AI, UIUC, Fudan University, and CUHK. Code: https://github.com/S1s-Z/Ctx2Skill

The Problem: "Read and Forget"

Current LLM in-context learning has a ceiling:

• RAG and ICL limits: Retrieval is not understanding; demonstrations are not internalization.
• Long-document complexity: Nested rules, branching procedures, and interleaved judgment criteria can't form a complete mental model in a single pass.
• Prohibitive manual annotation cost: Having humans read a 100-page manual and write reusable skill docs is infeasible.
• No feedback for automatic extraction: How do you know machine-generated skills are complete and faithful?

Architecture: Three Agents in Adversarial Self-Play

Each iteration:

1. Challenger — reads the document plus its own "questioning strategy book," generates probing tasks with rubrics. Key constraint: correct answers must *induce* rules from the document, not *parrot* surface text. Its strategy book evolves: question types easily solved last round get harder variants next round. 2. Reasoner — reads the document, the tasks, and its skill book, then attempts solutions. The skill book distills long, dense content into concise, structured rules and procedures. 3. Judge — strict all-or-nothing binary scoring: a task may have 3–5 rubrics, and failing any single one means Fail. Prevents the "mostly correct = wrong" ambiguity in engineering contexts.

Two additional roles handle updates:

• Proposer (diagnostician) — identifies patterns *across* failure cases, not case-by-case. The Reasoner-side Proposer diagnoses missing/misunderstood knowledge; the Challenger-side Proposer diagnoses weak probing strategies.
• Generator — actually rewrites the skill books: adds new entries, merges redundant ones, keeps irrelevant ones. Proposer and Generator are decoupled — analysis proposes *what and why*, execution decides *how*, mirroring human team workflows.

Preventing Adversarial Collapse: Cross-Time Replay

Self-play loops risk adversarial collapse: the Challenger increasingly targets the Reasoner's residual weaknesses, questions drift from representative document knowledge, and failure-driven skill updates over-specialize. Worse, the in-loop Judge only evaluates new tasks, so it can't detect whether early-learned knowledge was damaged by later edits.

Cross-Time Replay instead selects the most generalizable skill book across *all* historical versions:

• Probe sets (built incrementally, no external supervision): a Hard Probe Set (lowest rubric pass rates) and an Easy Probe Set (fewest rubrics) accumulated each round.
• Selection score: a multiplicative Laplace-smoothed product ρ^h(i) · ρ^e(i) over the two probe sets. Multiplication is essential — an additive score would let an extremely hard-biased skill set win (e.g., ρ^h=0.9, ρ^e=0.1 sums to 1.0 but multiplies to 0.09).

Results on CL-bench

Benchmark stats: 500 contexts, 1,899 tasks, 31,607 rubrics, average input 10.4K tokens (max 65K), 51.1% sequential tasks.

| Backbone | No skills | + Ctx2Skill | Gain | |---|---|---|---| | GPT-4.1 | 11.1% | 16.5% | +5.4% | | GPT-5.1 | 21.1% | 25.8% | +4.6% | | GPT-5.2 | 18.2% | 21.4% | +3.2% |

• GPT-4.1 + Ctx2Skill (16.5%) beats the un-augmented stronger Gemini 3 Pro (15.8%).
• Beats prompting-based skill extraction (12.3% on GPT-4.1) and AutoSkill4Doc (13.2%). Prompting even *hurt* rule-system tasks by 2.5% on GPT-4.1.
• Skill quality (conciseness, faithfulness, clarity, effectiveness, reusability) averages 89.8 vs 81.8 (Prompting) and 86.2 (AutoSkill4Doc), with the largest gains in faithfulness and clarity.

Ablations

| Removed component | Effect | |---|---| | Challenger skill evolution | −2.7% / −3.3% (largest) — sustained adversarial pressure is essential | | Cross-Time Replay | −1.8% / −2.8% — prevents collapse | | Proposer/Generator decoupling | −0.6% / −0.7% | | Multiplicative → additive selection | −0.6% |

Fixed-iteration skill books monotonically degrade after iteration 1 (15.9% → 14.7% by iter-5 on GPT-4.1), confirming collapse; Cross-Time Replay (16.5%) beats all fixed checkpoints.

Asymmetric Skill Transfer

• GPT-4.1 using GPT-5.1's skills: 16.1% (near self-generated 16.5%) — strong-model skills transfer down well.
• GPT-5.1 using GPT-4.1's skills: 23.1% (vs 25.8% self-generated) — weak-model skills add little for strong models.

Why It Works

• From retrieval to modeling: RAG makes the model a courier; Ctx2Skill makes it a modeler. The skill book is not a summary but the rule system *behind* the document.
• Adversarial pressure as automatic annotation: if the Reasoner fails, a skill is missing; if it survives harder questions, the skill is solid. Adversarial pressure reveals knowledge boundaries without human raters.
• Failure-driven learning: both sides must evolve — removing either the Reasoner's or the Challenger's update loop degrades results.

Limitations and Open Questions

1. Only 5 rounds × 5 tasks (API budget ~$30K total). Is that enough rubric coverage? Would larger per-round task counts improve later iterations? 2. LLM-as-a-Judge dependence: Judge errors propagate — False Passes leave knowledge gaps unfixed; False Fails inject noise into skill updates. Error-rate impact is unquantified. 3. Multi-agent cost may not pay off for documents queried only once or twice; best suited to repeatedly queried high-value corpora (company knowledge bases, clinical guidelines). 4. Human verification gap: a skill book distilled from 65K tokens may contain dozens of rules — can reviewers really verify completeness? Ctx2Skill solves automatic extraction, not human validation.

Conclusion

Ctx2Skill's core insight: the best way to understand a document is not to read it, but to *use* it — asking questions, answering, and correcting — letting adversarial play force out the rule system behind the text. It replaces human annotation labor with AI adversariality, turning LLMs from search engines into model builders.