Supervised Reinforcement Learning (SRL): A Framework Enabling Small LLMs to Learn Complex Reasoning
Background and Motivation
Large language models (LLMs) struggle with complex tasks requiring multi-step reasoning, especially small open-source models (e.g., the 7B scale). Two major challenges exist in current training paradigms:
1. Reinforcement Learning with Verifiable Rewards (RLVR) fails when the model cannot sample correct solutions, because the reward signal is too sparse. 2. Supervised Fine-Tuning (SFT) relies on strict token-level imitation of long demonstrations, which easily overfits and generalizes poorly.
To address this, the Google Cloud AI Research team proposed Supervised Reinforcement Learning (SRL), which reframes problem solving as generating a sequence of logical "actions," guided by dense, smooth reward signals that teach the model expert reasoning patterns.
Core Method
SRL decomposes expert solutions into step-by-step action sequences and trains the model to generate an internal reasoning monologue before executing each action. Given a correct expert trajectory \(y\), SRL splits it into an action sequence \(y = \{y_{\text{step}_n}\}_{n=1}^N\), where each action represents a meaningful decision step (e.g., an algebraic operation in math derivations or a command in code repair). For each step \(k\), the input prompt is \(x_{\text{step}_k} = [x, y_{\text{step}_1}, \ldots, y_{\text{step}_{k-1}}]\), and the model predicts the next action \(y_{\text{step}_k}\), outputting:
1. An internal monologue \(y'_{\text{think}}\) wrapped in special tags; 2. The executed action \(y'_{\text{step}_k}\).
A sequence-similarity-based reward provides fine-grained supervision:
where \(T\) is the total number of elements in both sequences and \(M\) is the number of elements in non-overlapping matched blocks. The reward is computed via Python's difflib.SequenceMatcher, ranges over \([0,1]\), and a malformed output receives a \(-1\) penalty. The policy model \(p_\theta\) is optimized with this reward through RL objectives such as GRPO. Notably, the reward is computed only on actions, not on the monologue content, allowing the model to develop its own reasoning style while keeping external behavior aligned with expert strategies.
Key Technical Details
1. Dynamic sampling strategy: Since the dense reward may still have variance, SRL filters out samples whose reward variance is near zero, retaining a batch of rollouts only if the reward standard deviation exceeds a threshold \(\varepsilon\). Dynamic sampling improves SRL accuracy from 24.7% to 27.6%.
2. Internal monologue mechanism: The model outputs an "inner monologue" before each action to organize its thoughts. The monologue is excluded from reward computation and serves purely as an auxiliary reasoning aid—for example, outlining solution steps in math tasks or diagnosing root causes before code fixes.
3. Step-level training data construction: Using a strong teacher model to generate solution trajectories, SRL constructs \(N-1\) partial trajectories from each complete solution, converting one expert solution into many training examples that teach the model how to proceed correctly from diverse intermediate states.
Experimental Results
Math Reasoning (AMC23, AIME24, AIME25, Minerva Math)
| Method | Average Accuracy | |---|---| | SFT | 16.6% | | RLVR | 24.5% | | SRL | 27.6% | | SRL → RLVR | 28.3% |
- SRL achieves 27.6% average accuracy, exceeding RLVR by 3.1 points and SFT by 11 points.
- The combined SRL → RLVR pipeline (first learning basic action patterns with SRL, then fine-tuning with RLVR) reaches 28.3%, including 20% on AIME24 Greedy—10 points above RLVR.
- Dynamic sampling alone lifts accuracy from 24.7% to 27.6%.
- vs. Process Reward Models (PRMs): PRMs act as "judges" requiring human annotation of every step's correctness; SRL automatically extracts actions from expert trajectories without manual labeling, learning both "what to do" and "how to think."
- vs. Curriculum learning: The SRL → RLVR pipeline forms an effective curriculum—mastering basic action patterns first, then optimizing final outcomes.
- vs. other supervised-RL variants: Methods like self-supervised RL (SSRL) typically rely on external rewards or task-specific designs, whereas SRL's generic similarity-based reward offers broader applicability.
Software Engineering (SWE-Bench)
With Qwen2.5-Coder-7B-Instruct, SRL achieves a 14.8% resolution rate in the oracle setting—74% higher than the SWE-Gym-7B baseline (~8.5%)—and maintains roughly double the performance in end-to-end evaluation.
Analysis
SRL's gains come not from longer outputs but from improved reasoning quality (better planning and verification). SRL models exhibit three flexible reasoning patterns:
1. Forward planning: outlining all subsequent steps before execution; 2. Dynamic adjustment: inserting multiple reasoning blocks and iteratively revising plans; 3. Reflective verification: pausing to validate before emitting the final answer.
Comparison with Existing Methods
Significance and Outlook
SRL's dense and smooth reward mechanism resolves the sparse-reward problem that plagues conventional RL on hard problems, offering a new way to train small models for complex reasoning. Experiments show it significantly outperforms baselines on math reasoning and software engineering, and combines with RLVR into a strong curriculum strategy. Its generality and robustness lay a foundation for building more capable, versatile AI agents, with potential applications in scientific reasoning, code generation, and agent planning.