Supervised Reinforcement Learning (SRL): A Framework That Teaches Small Models to Reason
*Source: zhichai.net forum post (Chinese). Full translation below.*
Background and Motivation
Large language models (LLMs) struggle with complex tasks requiring multi-step reasoning, especially small open-source models at the ~7B scale. Two existing training paradigms face challenges:
1. Reinforcement Learning with Verifiable Rewards (RLVR) fails when the model cannot sample correct solutions, because the reward signal is too sparse. 2. Supervised Fine-Tuning (SFT) performs strict token-by-token imitation, easily overfitting long demonstrations and lacking generalization.
To address this, Google Cloud AI Research proposed the Supervised Reinforcement Learning (SRL) framework, which reformulates problem solving as generating a sequence of logical "actions," guided by dense, smooth reward signals that teach the model expert reasoning patterns.
Core Method
SRL's core idea is to decompose expert solutions into step-by-step action sequences and train the model to produce an internal reasoning monologue before executing each action. Given a correct expert trajectory \(y\), SRL decomposes it into an action sequence \(y = \{y_{\text{step}_n}\}_{n=1}^N\), where each action represents a meaningful decision step (e.g., an algebraic operation in math derivations or a command in code repair). For each step \(k\), the input prompt is \(x_{\text{step}_k} = [x, y_{\text{step}_1}, \ldots, y_{\text{step}_{k-1}}]\), and the model must predict the next action \(y_{\text{step}_k}\). Each output consists of:
1. An internal monologue \(y'_{\text{think}}\), wrapped in special tags; 2. An executed action \(y'_{\text{step}_k}\).
To provide fine-grained supervision, SRL uses a sequence-similarity-based reward:
where \(T\) is the total number of elements in the two sequences and \(M\) is the total number of elements in non-overlapping matching blocks. The reward is computed with Python's difflib.SequenceMatcher, ranges over \([0,1]\), and a \(-1\) penalty is assigned for malformed output. The policy model \(p_\theta\) is optimized with this reward via RL objectives such as GRPO. Notably, the reward is computed only on actions and does not constrain the monologue, giving the model flexibility to develop its own reasoning style while keeping external behavior aligned with expert strategies.
Key Technical Details
1. Dynamic sampling: Because the dense rewards can still have variance, SRL filters out rollouts whose reward variance is near zero, keeping only batches where the standard deviation exceeds a threshold \(\varepsilon\) and sampling until the batch is filled. Dynamic sampling raised SRL accuracy from 24.7% to 27.6%.
2. Internal monologue mechanism: Before generating an action, the model outputs an "inner monologue" to organize its thinking. It is excluded from reward computation and serves only as an auxiliary reasoning aid, encouraging flexible planning and verification (e.g., outlining solution steps in math, or analyzing root causes before fixing code).
3. Stepwise training data construction: A strong teacher model generates solution trajectories; each full solution yields \(N-1\) partial trajectories, converting a single expert solution into many training instances that teach the model how to advance correctly from arbitrary intermediate states.
Experimental Results
On math reasoning benchmarks (AMC23, AIME24, AIME25, Minerva Math), SRL significantly outperforms baselines:
| Method | Average accuracy | |---|---| | SFT | 16.6% | | RLVR | 24.5% | | SRL | 27.6% | | SRL→RLVR (pipeline) | 28.3% |
- SRL→RLVR combined strategy: first learn basic action sequences with SRL, then fine-tune with RLVR, reaching 28.3% average accuracy and 20% on AIME24 Greedy — 10 points above RLVR alone.
- Dynamic sampling contribution: accuracy improved from 24.7% to 27.6%.
- vs. Process Reward Models (PRM): PRMs act as "judges" requiring human annotation of per-step correctness; SRL extracts actions automatically from expert trajectories, requiring no manual labels — learning both "how to do" and "how to think."
- vs. Curriculum learning: The SRL→RLVR pipeline forms a strong curriculum: master basic action patterns with SRL, then optimize final outcomes with RLVR.
- vs. other supervised RL methods: Approaches like self-supervised RL (SSRL) typically depend on external rewards or task-specific designs, whereas SRL's generic similarity reward gives broader applicability.
On software engineering (SWE-Bench), an SRL model based on Qwen2.5-Coder-7B-Instruct achieved a 14.8% resolution rate in the oracle setting, about 74% higher than the SWE-Gym-7B baseline (~8.5%), and roughly double the baseline in end-to-end evaluation.
Further analysis shows the gains come not from longer outputs but from improved reasoning quality. SRL models exhibit three flexible reasoning patterns: (1) upfront planning — outlining all subsequent steps first; (2) dynamic adjustment — inserting multiple reasoning blocks and iteratively revising plans; (3) reflective verification — pausing to verify before giving the final answer.
Comparison with Existing Methods
Implications and Outlook
SRL offers a new way to train small models for complex reasoning. Its dense, smooth reward mechanism resolves the sparse-reward problem conventional RL faces on hard problems. Proven effective on both math reasoning and software engineering, and combinable with RLVR as a curriculum strategy, the framework's generality and robustness lay groundwork for building stronger, more versatile AI agents with potential applications in scientific reasoning, code generation, and agent planning.