Latent-GRPO: Why Training LLMs to Reason Without Showing Their Work Is So Hard

This post from zhichai.net (authored by "Xiaokai") dissects the paper "Latent-GRPO: Group Relative Policy Optimization for Latent Reasoning" (arXiv: 2604.27998) by Jingcheng Deng, Zihao Wei, Liang Pang, Junhong Wu, Shicheng Xu, Zenghao Duan, and Huawei Shen, from the Institute of Computing Technology (CAS), UCAS, and the Institute of Automation (CAS).

From "Thinking Out Loud" to Mental Math

• Explicit reasoning = Chain-of-Thought (CoT): the model verbalizes every step ("2+3=5, then 5×4=20..."), consuming hundreds of tokens.
• Latent reasoning = the model compresses intermediate steps into continuous latent tokens — vectors encoding reasoning states that don't map to words. The temptation: reasoning chains 3-4x shorter, faster and cheaper.
• The catch: nobody knew how to reliably *train* models to do this "mental math."

GRPO: The Best Teacher, Translated Badly

GRPO (Group Relative Policy Optimization) samples multiple answers per question, rewarding good ones and penalizing bad ones — the engine behind DeepSeek-R1 and OpenAI's o-series. But transplanting it directly to latent reasoning fails due to three coupled bottlenecks, each alone enough to crash training.

Bottleneck 1: Exploring Without a Map (No Intrinsic Latent Manifold)

• In explicit reasoning, the vocabulary acts as a natural "map" of valid states. In latent reasoning, the continuous space is infinite and almost all vectors are meaningless.
• Unconstrained Soft-GRPO sampling degenerates into endless non-terminating vector sequences and collapses.
• Fixes:
• Latent-SFT initialization: teach the model which vectors are valid reasoning states before RL — walk the main streets first, then fine-tune.
• Invalid Sample Advantage Masking: over-long, non-terminating trajectories are marked invalid, excluded from training signal and reward statistics — like telling the group "ignore him, he's just lost."

Bottleneck 2: A Steering Wheel Wired Backwards (Exploration-Optimization Misalignment)

• Latent reasoning's density depends on a perturbation margin Δi = g'i − log pi. When Gumbel exploration noise makes Δi negative, even positively-rewarded trajectories can have some component probabilities pushed down — the local update direction opposes the global reward signal.
• Fixes:
• One-sided Noise Sampling: clip-and-shift transforms so all noise stays in a positive interval [δ, a+b+δ] — no negative margins allowed.
• Conditional Straight-Through Estimator (STE): if a component's probability overshoots (margin turns negative) after multiple PPO updates, the gradient is flipped to restore consistency.

Bottleneck 3: The Averaging Trap (Latent Mixture Non-Closure)

• In discrete token generation, the model picks one path at a time. In latent space, averaging is legal: if v1 and v2 both lead to correct answers, the model may output (v1+v2)/2 — which may lie off the valid manifold, "stuck in a wall."
• Fix — Optimal Correct Path First Token Selection: when multiple paths are correct, pick the most reliable one (highest average surrogate log-probability) and update only that path's first token; later tokens on other paths are kept since their contexts have already diverged.

Results: 3-4x Compression Without Losing Accuracy

Easier tasks (GSM8K-Aug, GSM-Hard, SVAMP, MultiArith):

• Latent-GRPO vs Latent-SFT: +7.86 Pass@1
• Reasoning chains 4.44x shorter than explicit GRPO
• Even beats explicit GRPO itself (66.29 vs 62.26 on GSM8K-Aug)

Harder tasks (Math500, AIME24, AIME25, GPQA):

• vs explicit GRPO: +4.27 Pass@1, chains 3.31x shorter
• Best Pass@1 on AIME24 and AIME25

Stability: Soft-GRPO crashes mid-training with cliff-like performance drops; Latent-GRPO trains stably across the full 10k steps. It also achieves strong pass@k under Gumbel sampling (50+ pass@64 on AIME), indicating diverse correct reasoning paths.

The Bigger Question

Does human thinking actually involve a long inner monologue? Cognitive science debates whether thought is linguistic or non-linguistic. Latent reasoning simulates the latter — and if it proves more efficient than verbal CoT, LLMs' future may be a compact "thought space" humans can't directly read. That raises a safety concern: if reasoning is no longer human-readable language, how do we audit it? Latent-GRPO solves only training stability; interpretability of 50 continuous vectors may be far harder than reading 200 CoT tokens.

---

Paper: Latent-GRPO: Group Relative Policy Optimization for Latent Reasoning — arXiv: 2604.27998, released 2026-04-30.

*Hardcore breakdown · Feynman style · Xiaokai*