Logic-RL: Rule-Based Reinforcement Learning Unlocks Reasoning in Large Language Models

Logic-RL is a framework that uses rule-based reinforcement learning (RL) to guide large language models (LLMs) toward advanced, generalizable reasoning rather than memorizing training data. Inspired by the success of DeepSeek-R1, it enables a relatively small model to make breakthroughs on complex logic and math tasks using only about 5,000 training examples.

Key points

• Rule-based RL core: Instead of large-scale human annotation or learned reward models, Logic-RL uses clear, verifiable rules to define "good" behavior. This yields precise, stable reward signals and avoids reward hacking.
• Training data: ~5,000 programmatically generated Knights & Knaves (K&K) logic puzzles. Difficulty is controllable (number of characters, nesting depth), and answers are unique and automatically verifiable—ideal for stable RL rewards.
• Structured system prompt: The model must place its reasoning inside <think>...</think> tags and the final answer inside <answer>...</answer> tags, mechanically preventing shortcut answers and forcing "thinking out loud."
• Stringent format reward: A composite reward combines an answer reward (exact match against the ground truth) with a format reward (correct use of tags and a non-trivial reasoning chain). Outputs that skip reasoning are penalized even if the answer is correct.
• Training recipe: Primarily based on REINFORCE++, with PPO/GRPO-style optimization and likely KL-divergence penalties (L(θ) = E[log π_θ(a|s) * A(s,a)] - β * KL(π_θ || π_ref)) to prevent policy collapse and ensure stable convergence.
• Emergent reasoning behavior: Response length grows spontaneously from ~500 tokens to nearly 2,000 tokens during training, and tokens associated with reflection and verification increase in frequency alongside performance—without any explicit length reward.
• Cross-domain generalization: The K&K-trained 7B model, applied directly to "super out-of-distribution" math competitions, improves accuracy by +125% on AIME (2021-2024) and +38% on AMC (2022-2023) relative to baselines:

| Benchmark | Performance Improvement | | :--- | :--- | | AIME (2021-2024) | +125% | | AMC (2022-2023) | +38% |

• Data efficiency: Success on just 5K synthetic examples challenges the notion that capability gains require massive data and parameter scaling, showing that well-designed RL on core cognitive skills can unlock transferable reasoning at low cost.
• Comparison with DeepSeek-R1: Logic-RL borrows the rule-based reward and GRPO-style RL ideas from DeepSeek-R1 but distinguishes itself with a mandatory structured output format and stringent format rewards that enforce complete reasoning chains.
• Comparison with traditional RLHF: Rule-based rewards are stable, unbiased, and cheap, unlike model-based reward models prone to inaccuracy, bias, and reward hacking. Training relies on cheap, reproducible synthetic data instead of expensive large-scale human annotation.
• Open source: The team has released code and data, including an official "Logic-RL" repository and a lightweight reproduction project ("Logic-RL-Lite") on GitHub, supporting reproducibility and community extension.

Why it matters

Logic-RL demonstrates that models can learn *how to think* rather than merely memorize answers. By combining controllable synthetic tasks, verifiable rewards, and strict process constraints, a 7B model achieves competitive reasoning performance, opening a path to high-capability, resource-efficient reasoning systems.