Ctx2Skill: Turning Long Documents into Reusable Skills via Multi-Agent Self-Play

In one sentence: Instead of letting an LLM read a 100-page technical document and forget it, Ctx2Skill has three agents play a self-supervised "generate-answers-grade" game against each other, automatically distilling the document's rules, procedures, and decision criteria into a reusable skill book — no human annotation or external verifier required.

The Problem: Long Documents and "Read-Then-Forget"

Current approaches to long-context understanding hit a ceiling:

• RAG and ICL limits: retrieval is not understanding; demonstrations are not internalization. A model that correctly answers one question may fail when the same logic is tested differently.
• Document complexity: nested rules, branching procedures, and interleaved judgment criteria cannot be captured in a single pass.
• Human skill annotation is prohibitively expensive.
• Automated extraction lacks feedback: how do you know a machine-generated skill is complete and faithful?

The paper *From Context to Skills* (Tsinghua University, DeepLang AI, UIUC, Fudan, CUHK) asks: can an AI derive skills from a document by itself?

Core Architecture: A Self-Play Loop

Each iteration:

1. Challenger reads the document plus its own question-strategy skill book and generates 5 probing tasks with rubrics. A key constraint: correct answers must *induce* the document's rules, not *paraphrase* surface text. 2. Reasoner reads the document, its skill book, and the tasks, then attempts answers. The skill book distills long, technical content into concise, structured rules. 3. Judge applies strict all-or-nothing binary grading: any unmet rubric means Fail. 4. Proposer diagnoses failures cross-case: the Reasoner-side Proposer finds missing or misunderstood knowledge; the Challenger-side Proposer finds weak probing strategies. 5. Generator executes the skill-book updates. Proposer and Generator are deliberately decoupled, mirroring how human analyst/engineer teams work.

Preventing Adversarial Collapse with Cross-Time Replay

Unchecked self-play risks adversarial collapse: the Challenger increasingly targets the Reasoner's residual weaknesses with extreme questions, while the Reasoner's failure-driven skill updates over-specialize — and this cannot be detected within a single round.

Cross-Time Replay addresses this by not taking the final skill book, but selecting the most generalized version across all historical iterations. Two probe sets are incrementally accumulated without external supervision:

• Hard Probe Set: tasks with the lowest rubric pass rates (hardest failures per round)
• Easy Probe Set: passed tasks with the fewest rubrics (easiest successes per round)

Selection uses a multiplicative Laplace-smoothed criterion, ρ^h(i) · ρ^e(i): additive scoring would let an extreme skill set score high (e.g., 0.9 + 0.1 = 1.0), while multiplication (0.09) forces balanced performance on both easy and hard tasks.

Benchmark: CL-bench

| Statistic | Value | |---|---| | Contexts | 500 | | Tasks | 1,899 | | Rubrics | 31,607 | | Avg. input length | 10.4K tokens (max 65K) | | Sequential tasks | 51.1% |

Four task categories: domain knowledge reasoning, rule-system application, procedural task execution, and empirical discovery & simulation.

Results

| Backbone | Baseline | + Ctx2Skill | Gain | |---|---|---|---| | GPT-4.1 | 11.1% | 16.5% | +5.4% | | GPT-5.1 | 21.1% | 25.8% | +4.6% | | GPT-5.2 | 18.2% | 21.4% | +3.2% |

• GPT-4.1 + Ctx2Skill (16.5%) surpasses the un-augmented Gemini 3 Pro (15.8%).
• Beats Prompting (12.3%) and AutoSkill4Doc (13.2%) on GPT-4.1. Prompting even *hurts* rule-system application by 2.5% on GPT-4.1.
• Skill quality (conciseness, faithfulness, clarity, effectiveness, reusability) averages 89.8 vs 86.2 (AutoSkill4Doc) and 81.8 (Prompting), with the largest gains in faithfulness (+3.4) and clarity (+3.8).

Ablations

| Ablation | Effect | |---|---| | Remove Challenger skill evolution | −2.7% (GPT-4.1), −3.3% (GPT-5.1) — most important | | Remove Cross-Time Replay | −1.8% / −2.8% | | Remove Proposer/Generator decoupling | −0.6% / −0.7% |

Fixed-iteration skill books decline monotonically (15.9% at Iter-1 down to 14.7% at Iter-5 on GPT-4.1), confirming adversarial collapse; Cross-Time Replay (16.5%) beats every fixed iteration.

Asymmetric Skill Transfer

• Strong→weak: GPT-4.1 + GPT-5.1-generated skills = 16.1% (near self-generated 16.5%)
• Weak→strong: GPT-5.1 + GPT-4.1-generated skills = 23.1% vs 25.8% self-generated

Skills from stronger models transfer well to weaker ones; the reverse gains little.

Why It Works

1. Paradigm shift from retrieval to modeling: the skill book encodes the rule system *behind* the document — what to do under which conditions, priority ordering, exception handling. 2. Adversarial pressure as automatic annotation: failed probes reveal knowledge gaps; surviving tightened probes confirm solid skills — no external labeler needed. 3. Failure-driven learning matters most: Challenger evolution (pass cases → probe strategy tightening) and Reasoner evolution (failure cases → skill updates) are both indispensable. 4. Multiplicative selection enforces balance between easy and hard task performance.

Limitations and Open Questions

1. Only 5 rounds × 5 tasks, limited by API budget (~$30K total); whether more probes would shift optimal iterations later is unclear. 2. LLM-as-a-Judge dependence: false passes leave permanent knowledge gaps; false fails inject noise. Error propagation across 31.6K rubrics is not quantified. 3. Multi-agent cost is nontrivial per iteration; best suited to high-value documents queried repeatedly (e.g., corporate knowledge bases, clinical guidelines). 4. Human verifiability gap: a skill book distilled from a 65K-token document may contain dozens of rules — can a human reviewer really audit completeness quickly?

Takeaway

Ctx2Skill does not make AI read faster; it makes AI read deeper. The core insight: the best way to understand a document is not to finish reading it, but to use it — generating questions, answering, and correcting — so that adversarial pressure replaces human annotation labor and surfaces the document's underlying rule system.

References

• Paper: From Context to Skills: Can Language Models Learn from Context Skillfully? (arXiv:2604.27660)
• Institutions: Tsinghua University, DeepLang AI, UIUC, Fudan University, CUHK
• Code: https://github.com/S1s-Z/Ctx2Skill