Logic-RL: Rule-Based Reinforcement Learning Unlocks Reasoning in Large Language Models

Logic-RL demonstrates that rule-based reinforcement learning (RL) can elicit deep reasoning abilities in LLMs, producing cross-domain gains on math competition benchmarks from training on purely synthetic logic puzzles.

Core Principles and Technical Innovations

Rule-Based RL Framework

Unlike conventional RL that depends on large-scale human-annotated data or complex learned reward models, Logic-RL uses a clear, explicit, and verifiable set of rules to define "good" behavior. The key advantage is a precise and stable reward signal that helps avoid reward hacking. Rules act directly on the model's output, evaluating whether it follows the prescribed reasoning structure and whether the final answer is correct. The RL objective thus shifts from matching a possibly noisy "golden answer" to learning a reasoning process that yields correct, well-structured answers.

Three pillars support the framework:

• System prompt design: Carefully crafted prompts require the reasoning process to be placed between specific tags, preventing the model from taking shortcuts.
• Format reward function: A strict format reward enforces output conventions; any deviation is penalized, ensuring the model must display its full reasoning process.
• Stable training: A REINFORCE++-based method keeps learning stable and convergent.

The overall pipeline: synthetic logic puzzles (Knights & Knaves) → system prompt with format requirements → <think> reasoning → format reward → compliance check → answer reward (verified correctness) → policy update via REINFORCE++ → capability gains and emergent advanced reasoning behaviors (reflection, verification, summarization).

Key insight: As training progresses, the model spontaneously develops higher-level behaviors such as reflection and verification. These are not pre-programmed; they emerge as strategies for solving problems more effectively, indicating genuine transferable reasoning skills.

Training Data and Tasks

Synthetic Logic Puzzles: An Ideal Training Ground

Logic-RL uses classic "Knights & Knaves" logic puzzles as core training data: an island where "knights" always tell the truth and "knaves" always lie. These puzzles are chosen because:

• Controllable complexity: difficulty scales with the number of characters and dialogue complexity.
• Verifiable answers: solutions are unique and can be automatically, precisely verified by a program.
• Cost-effective: synthetic data is cheap and unlimited.
• Clean environment: pure logic, no external knowledge required.

Cross-Domain Generalization: From Logic Puzzles to Math Competitions

• Training: 5,000 synthetic Knights & Knaves puzzles, complexity 2-8 characters, pure logical reasoning.
• Testing: AIME 2021-2024 and AMC 2022-2023 problems.

Remarkable performance gains: A 7B-parameter model trained on just 5K logic puzzles improves accuracy by 125% on AIME 2021-2024 and 38% on AMC 2022-2023 relative to its baseline.

Such large improvements on tasks very different from the training distribution indicate the model learns transferable reasoning strategies — decomposing complex problems, constructing and validating hypotheses, and systematic search. This cross-domain success strongly suggests Logic-RL cultivates a general underlying "thinking ability" rather than task-specific tricks.

Technical Details and Implementation

Reward Function Design

Total reward = w_format × R_format + w_answer × R_answer

• Format reward (R_format): checks <think> and <answer> tag usage, verifies reasoning completeness, penalizes non-conforming output.
• Answer reward (R_answer): exact match against the standard answer, binary correctness judgment, stable and unbiased feedback.

Training Algorithm and Optimization

• REINFORCE++: policy-gradient base algorithm; introduces a baseline to reduce gradient-estimation variance and improve stability.
• PPO/GRPO: clips policy update magnitudes to prevent drastic policy changes.
• KL-divergence penalty: constrains divergence from the reference policy to prevent instability or policy collapse.

Training dynamics: the model spontaneously lengthens its reasoning — from roughly 500 tokens early in training to nearly 2,000 tokens at the end. This response-length growth closely correlates with performance gains.

Open-Source Resources

• Official repository with full implementation code, datasets, and training scripts.
• A lightweight reproduction project, Logic-RL-Lite, for quick experimentation.

Key implementation elements include hyperparameter configuration, a curriculum/mixed-difficulty strategy (puzzles spanning 2-8 characters), a complete AIME/AMC evaluation pipeline, and reproducibility support.

Conclusion

Logic-RL shows that simple, verifiable, rule-based rewards on synthetic logic puzzles can unlock substantial, transferable reasoning capabilities in LLMs — with emergent reflection and verification behaviors and dramatic cross-domain gains on competition math benchmarks.