SPICE: Self-Play In Corpus Environments Improves LLM Reasoning

Forum topic · QianXun · 2025-11-23

Summary

SPICE (Self-Play In Corpus Environments) is a reinforcement learning framework in which a single language model plays two roles: a Challenger that mines documents from a large corpus to generate diverse reasoning tasks, and a Reasoner that solves them without access to the source documents. By grounding questions and gold answers in real web documents, SPICE addresses two key problems of prior self-play methods: hallucination amplification from unverifiable synthetic data, and information symmetry between question generator and solver. The Challenger receives a variance-based curriculum reward peaking at a 50% pass rate, while the Reasoner gets binary correctness rewards. Experiments across four base models show consistent gains, e.g., Qwen3-4B-Base improving from 35.8% to 44.9% and OctoThinker-8B from 20.5% to 32.4%, with average improvements of +8.9% on math reasoning and +9.8% on general reasoning benchmarks (MMLU-Pro, GPQA-Diamond, SuperGPQA, BBEH). SPICE requires only a large unstructured corpus and no human supervision, offering an open-ended path to self-improving reasoning (arXiv:2510.24684).

SPICE: Self-Play In Corpus Environments Improves Reasoning

SPICE is a reinforcement learning framework where a single model plays two self-play roles: a Challenger that mines documents from a large corpus to generate diverse reasoning tasks, and a Reasoner that solves them.

Key points

Two problems addressed compared to non-grounded self-play:
*Hallucination amplification*: training on the model's own unverifiable synthetic data compounds factual errors in generated questions and answers.
*Information symmetry*: when generator and solver share the same knowledge, no genuine challenge exists.
How it works: The Challenger samples documents from the corpus and generates tasks (multiple-choice or free-form questions with integer/expression/string answers). The Reasoner answers without document access, prompted to reason step by step and box the final answer. Training is fully self-supervised and needs only a large unstructured corpus.
Reward design:
Challenger: Gaussian-shaped variance-based curriculum reward, maximized when pass-rate variance equals 0.25 (i.e., 50% pass rate = optimal difficulty); invalid tasks receive a penalty.
Reasoner: binary correctness reward verified against the document-grounded gold answer.
Why document grounding matters:
*Information asymmetry* — the Challenger anchors questions in documents unseen by the Reasoner.
*Document diversity* ensures continual novelty beyond internal model knowledge.
*Hallucination prevention* by anchoring answers in real content.
*Universal verification* via task formats that need no executors or rule-based verifiers, breaking the bottleneck confining prior work to math and code.

Results

| Base model | Baseline | SPICE | Gain | |---|---|---|---| | Qwen3-4B-Base | 35.8% | 44.9% | +9.1% | | Qwen3-8B-Base | 43.0% | 48.7% | +5.7% | | OctoThinker-3B-Hybrid-Base | 14.7% | 25.2% | +10.5% | | OctoThinker-8B-Hybrid-Base | 20.5% | 32.4% | +11.9% |

Gains span math reasoning (avg +8.9%) and general reasoning (+9.8% on MMLU-Pro, GPQA-Diamond, SuperGPQA, BBEH).
Adversarial dynamics create an automatic curriculum: a fixed Reasoner's pass rate dropped from 55% to 35% (harder questions), while a fixed Challenger's pass rate rose from 55% to 85% (stronger solving) — evidence both roles co-evolve.

Takeaway

SPICE marks a shift from closed-loop self-play, which often stagnates due to hallucination drift, to open-ended improvement by interacting with the large, verifiable knowledge embedded in web-scale document corpora — enabling self-improvement without explicit human supervision.

Reference: Liu, B., Jin, C., Kim, S., et al. (2025). SPICE: Self-Play In Corpus Environments Improves Reasoning. arXiv:2510.24684v1.