Latent-GRPO: Why Teaching LLMs to "Do Mental Math" Is Harder Than Letting Them Think Out Loud
From "Thinking Out Loud" to "Thinking Silently"
Imagine solving a math problem two ways. The first is to say every step aloud: "first 2+3=5, then 5×4=20..." This is explicit reasoning — every step is a discrete language token. The second is to compute in your head without verbalizing anything, then just give the answer. This is latent reasoning — the thought process is compressed into a continuous "thought space" rather than discrete text.
For LLMs, explicit reasoning is the familiar Chain-of-Thought (CoT): the model generates a long stream of "let me think... first... then... so...". Latent reasoning instead outputs continuous vectors (latent tokens) that encode reasoning states without corresponding to any specific word.
Latent reasoning has a huge allure: it can shorten reasoning chains 3–4x. Instead of 200 tokens of written-out derivation, the model outputs ~50 continuous "thought vectors" and answers directly. Faster and cheaper.
The problem: we didn't know how to train models to do this "mental math."
GRPO: The Best Teacher, The Worst Fit
The most popular method for teaching LLM reasoning is GRPO (Group Relative Policy Optimization): generate multiple answers per question, compare them, reward the good ones, penalize the bad ones. It powers DeepSeek-R1 and OpenAI's o-series models.
But moving GRPO to latent reasoning breaks down. A paper from the Institute of Computing Technology (CAS), UCAS, and the Institute of Automation identifies three coupled bottlenecks, each alone enough to collapse training.
Bottleneck 1: Exploring Without a Map
The first issue is "lack of an intrinsic latent manifold."
In explicit reasoning, a "map" naturally exists: every token lies in the vocabulary. But in continuous vector space, almost every vector is meaningless — it doesn't represent a valid reasoning state. Unconstrained GRPO exploration quickly gets lost, producing endless streams of meaningless vectors, and unconstrained Soft-GRPO training collapses.
The fix: give the model a map before letting it explore. Latent-GRPO first initializes with Latent-SFT (supervised fine-tuning) so the model learns which vectors are valid reasoning states. Then RL only fine-tunes rather than exploring from scratch.
Even with a map, exploration sometimes goes astray. The paper uses Invalid Sample Advantage Masking: any reasoning path that runs too long without terminating is marked "invalid," given no training signal, and excluded from other samples' reward statistics — like telling your group "ignore him, he just got lost, keep going."
Bottleneck 2: A Steering Wheel Wired Backwards
The second problem is subtler: "exploration-optimization misalignment."
Gumbel noise is injected into token probabilities to simulate random sampling and exploration. In explicit reasoning this works: noise lets the model try different tokens, good ones rise, bad ones fall. But in latent reasoning, the density depends on a "perturbation margin" Δi = g'i − log pi. Gumbel noise can be negative — and when Δi < 0, even a trajectory with positive overall reward can have some of its components' probabilities pushed down. Like adding salt to a bad dish and removing it from a good one: the general trend is right, but the local adjustments are backwards.
The fix is One-sided Noise Sampling: keep only positive perturbation margins via a clip-and-shift transformation into [δ, a+b+δ]. Like fixing the reversed steering wheel.
The paper also adds a conditional Straight-Through Estimator (STE): if a component's probability overshoots its target (margin goes negative) after multiple PPO updates, the gradient is flipped to keep optimization direction consistent.
Bottleneck 3: The Averaging Trap of Multiple Correct Paths
The third problem, "latent mixture non-closure," is a geometric trap unique to continuous space.
In explicit reasoning, the model picks one token at a time — left path or right path, never both. But in latent space, averaging is legal. If the model deems v1 (left path) and v2 (right path) both correct, it may output (v1+v2)/2 as the next state. The problem: that average may not represent any valid reasoning state — it may be "stuck in a wall." It falls outside the valid latent manifold — this is the "non-closure."
The fix: Optimal Correct Path First Token Selection. When multiple paths are correct, don't average them — select the most reliable one (highest average proxy log-probability), and update only that path's first token, keeping later tokens of other paths (their contexts have already diverged, so they no longer average).
Results: 3–4x Compression Without Losing Performance
With all three bottlenecks solved:
Low-difficulty tasks (GSM8K-Aug, GSM-Hard, SVAMP, MultiArith):
- +7.86 Pass@1 over Latent-SFT
- Reasoning chains 4.44x shorter than explicit GRPO
- Even beats explicit GRPO itself (66.29 vs 62.26 on GSM8K-Aug)
- +4.27 Pass@1 over explicit GRPO
- Chains 3.31x shorter
- Best Pass@1 on AIME24 and AIME25
- Title: Latent-GRPO: Group Relative Policy Optimization for Latent Reasoning
- arXiv: 2604.27998
- Authors: Jingcheng Deng, Zihao Wei, Liang Pang, Junhong Wu, Shicheng Xu, Zenghao Duan, Huawei Shen
- Institutions: Institute of Computing Technology (CAS) / UCAS / Institute of Automation (CAS)
- Published: 2026-04-30
Hard tasks (Math500, AIME24, AIME25, GPQA):
Crucially, it's stable: Soft-GRPO collapses mid-training with a cliff-drop in performance, while Latent-GRPO remains stable across 10k training steps — slower to start (it must optimize token selection and mixture weights simultaneously) but stronger overall.
A bonus: under Gumbel sampling, Latent-GRPO's pass@k is also strong — 50+ pass@64 on AIME — showing the model explores multiple viable paths, not just one.
A Bigger Question
Does human thinking really involve a long inner monologue? Cognitive science has long debated whether thought is linguistic (internal speech at every step) or non-linguistic (the brain directly manipulating concepts and images). Latent reasoning simulates the second mode. If it ultimately proves more efficient than language-based reasoning, perhaps the future of LLMs isn't ever-longer CoT, but a more compact "thought space" humans may never directly read.
That raises a safety concern: if reasoning is no longer human-readable language, how do we audit it? Latent-GRPO only solves the training stability problem. Interpretability may be far harder — reading a 200-token CoT is already difficult; reading 50 continuous vectors of "thought state" is nearly impossible.
---
Paper information