POISE: Your Language Model Is Its Own Critic — Value Estimation from the Actor's Internal States
> In May 2026, Choi et al. proposed POISE (Policy Optimization with Internal State Value Estimation), an RLVR method that leverages a language model's own internal representations for value estimation. By extracting hidden states and token entropy statistics during the policy's forward pass, POISE builds a lightweight value probe at negligible cost — eliminating PPO's need for a separate critic model and GRPO's need for multi-rollout group means. Experiments on Qwen3-4B and DeepSeek-R1-Distill-Qwen-1.5B show POISE matches DAPO performance at significantly lower compute cost, with a value estimator whose accuracy approaches that of an independent LLM-scale value model.
Key Points
- The baseline dilemma in RLVR: Rewards arrive only at sequence end, making credit assignment hard. Existing solutions are costly:
- PPO: trains a critic network roughly the size of the policy
- GRPO: requires 8–16 rollouts per prompt for group-mean baselines
- DAPO: needs complex dynamic filtering/scheduling logic
- Core insight: The policy model's internal representations already implicitly encode predictions of the final reward. Hidden states encode "confidence" in the current reasoning path; token entropy reflects uncertainty patterns — signals computed but unused in the forward pass.
- Three-source value probe: The probe combines (1) the prompt's final-layer hidden state, (2) aggregated trajectory hidden states, and (3) token entropy statistics. At ~0.1% of the policy model's parameters, its overhead is negligible. It is trained jointly and online with the policy via MSE loss on reward prediction, and its output serves as the policy-gradient baseline.
- Cross-rollout for unbiased estimation: Predicting a trajectory's value from its own internal features risks overfitting to its specific token sequence. POISE symmetrically swaps features: features from trajectory B estimate V(τ_A), and vice versa. Since both trajectories come from the same policy and prompt, features are unbiased while being blind to the target's specific tokens.
- Efficiency gains:
- vs. GRPO: 1 rollout per prompt instead of 8–16 (6.25%–12.5% of sampling FLOPs); 8–16× more prompt diversity at fixed budget; zero-advantage prompts (all-correct/all-wrong groups) are avoided entirely by design
- vs. PPO: critic parameters drop to ~0.1% of policy size; total memory ~1.001× instead of 2×; one forward pass per step instead of two
- Experimental validation: On Qwen3-4B and DeepSeek-R1-Distill-Qwen-1.5B, POISE matches DAPO on math reasoning at lower cost. The tiny probe's value-estimation accuracy approaches that of an independent LLM-scale value model, and it generalizes across multiple verifiable tasks beyond math.
- Complementary to "Beyond the 80/20 Rule": That work identified high-entropy tokens as key RLVR optimization targets and pruned gradients to 20% of tokens; POISE uses entropy statistics as probe input for baseline compression. Combined: fewer tokens × fewer rollouts × no critic.
- Self-critique lineage: Following STaR/Self-Refine and Self-Rewarding, POISE's distinction is that evaluation happens online, during generation — providing real-time baseline signals rather than post-hoc self-assessment.
- Scale: Validated only at 1.5B–4B. Do internal states retain sufficient value information at 14B+?
- Representation degradation: As training progresses, internal representations may "harden" (cf. low-entropy token invariance), possibly affecting probe generalization.
- Integration: Potential combinations with REINFORCE++ (value baseline), DAPO (dynamic sampling), and RLOO (leave-one-out variance reduction).
- Interpretability: Which hidden dimensions encode value? Do "value-sensitive neurons" exist, analogous to fork tokens?
Relation to Prior Work
Limitations and Open Questions
Conclusion
POISE offers a minimalist, efficient solution to baseline estimation in RLVR by exploiting signals the policy already computes. Its central claim — the model already "knows" the quality of its answer during generation — challenges the traditional assumption that RL requires an independent value network. Amid rising LLM training costs, POISE exemplifies a shift toward smarter use of existing computation rather than more parameters or samples.
Paper Details
| Item | Content | |:---|:---| | Title | Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States | | Authors | Yunho Choi, Jongwon Lim, Woojin Ahn, Minjae Oh, Jeonghoon Shim, Yohan Jo | | arXiv ID | 2605.07579 | | Date | 2026-05-08 | | Key contributions | Internal-state value estimation; lightweight probe; unbiased cross-rollout construction; single-rollout matching DAPO; elimination of zero-advantage sampling waste | | Key results | Matches DAPO on Qwen3-4B / DeepSeek-R1-Distill-Qwen-1.5B; probe performance near LLM-scale value models; generalizes across verifiable tasks |