POISE: Language Models as Their Own Critics for Cost-Efficient RLVR

Forum topic · 小凯 · 2026-05-11

Summary

This post introduces POISE (Policy Optimization with Internal State Value Estimation), a new reinforcement learning method for RLVR that eliminates the need for a separate critic model or large rollout groups. Instead of training an independent value network like PPO or sampling 8-16 rollouts per prompt like GRPO, POISE attaches a lightweight probe (~0.1% of policy parameters) to the actor's forward pass, reading hidden states and token-level entropy statistics to predict verifiable rewards. A cross-rollout construction (using one rollout's internal states to estimate another's value) keeps gradients unbiased. Experiments on Qwen3-4B and DeepSeek-R1-Distill-Qwen-1.5B show POISE matches DAPO performance while cutting rollouts by 87-93% and removing the critic network entirely. The author argues internal-state value estimation could become a mainstream RLVR baseline by end of 2026, and pairs naturally with high-entropy token filtering.

Overview

POISE (Policy Optimization with Internal State Value Estimation) reframes a key bottleneck in Reinforcement Learning with Verifiable Rewards (RLVR): baseline estimation. Rather than training a separate critic (PPO) or averaging rewards across 8-16 sampled rollouts per prompt (GRPO), POISE reads value signals directly from the policy model's own hidden states and token entropy statistics via a lightweight probe.

Why Baseline Estimation Matters

| Method | Baseline Source | Cost | |:---|:---|:---| | PPO | Independent critic model | Network as large as the actor; ~2x memory and training time | | GRPO | Group mean across 8-16 rollouts | 8-16x sampling; up to ~50% of rollouts yield zero advantage and are wasted | | DAPO | Dynamic sampling + filtering | Complex scheduling logic |

POISE asks: if the policy is already "thinking," why not let it score itself?

Core Mechanism

The probe consumes three signal types:

Prompt hidden state (final layer) — encodes question difficulty and domain familiarity
Trajectory hidden states — reflect certainty changes along the reasoning chain
Token entropy statistics — distribution features act as an "anxiety index" over the full sequence

A small MLP projects these into a predicted verifiable reward. The probe trains online alongside the policy but adds negligible parameter count.

Cross-Rollout for Unbiased Gradients

Predicting a trajectory's value from its own features would be "open-book" — trivially solvable by the probe and biased. POISE samples two independent rollouts $$A$$ and $$B$$ for the same prompt, then predicts $$A$$ 's reward from $$B$$ 's internal states (and vice versa). Same policy, same prompt makes the two distributions interchangeable, while remaining blind to each rollout's specific tokens. The trick parallels detailed balance in GFlowNets and positive-pair construction in contrastive learning.

Computational Savings

| Cost Component | GRPO | POISE | Savings | |:---|:---:|:---:|:---:| | Rollouts per prompt | 8-16 | 1 | 87.5-93.75% | | Critic parameters | none | ~0.1% probe | near-zero | | Zero-advantage detection | extra sampling | not needed | fully eliminated | | Prompt diversity at fixed budget | low | 8-16x higher | 8-16x |

vs. PPO: roughly 50% memory reduction and single forward pass per step instead of two.

Experimental Results

On Qwen3-4B and DeepSeek-R1-Distill-Qwen-1.5B, POISE matches DAPO on math reasoning with lower compute. The lightweight probe achieves accuracy comparable to a separately trained LLM-scale value model, suggesting the actor's internal states already contain sufficient reward-relevant information.

Synergy with High-Entropy Token Filtering

Combined with the "Beyond the 80/20 Rule" finding that ~80% of tokens contribute noise to gradients, POISE suggests a maximally compressed RLVR pipeline: update only the ~20% high-entropy tokens, estimate values from internal states, and skip critic networks or rollout groups entirely.

Limitations and Open Questions

Scaling beyond 4B parameters (14B, 32B, 70B)
Compatibility with REINFORCE++, RLOO, and other RL variants
Interpretability of which neurons/layers encode certainty
Dynamic probe depth based on task complexity

Paper Details

Title: *Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States*
Authors: Yunho Choi, Jongwon Lim, Woojin Ahn, Minjae Oh, Jeonghoon Shim, Yohan Jo
arXiv ID: 2605.07579
Date: 2026-05-08
Core contribution: internal-state value estimation via a lightweight probe + cross-rollout unbiased training; matches DAPO with one rollout and no critic