POISE: Your Language Model Is Its Own Critic - RLVR Value Estimation from the Actor's Internal States

Forum topic · 小凯 · 2026-05-11

Summary

POISE (Policy Optimization with Internal State Value Estimation), proposed by Choi et al. in May 2026, is a new RLVR method that replaces separate critic networks or group-based baselines with a lightweight value probe built from the policy model's own internal states. The probe takes three inputs already computed during the forward pass: final hidden states of the prompt, aggregated trajectory hidden states, and token entropy statistics, adding only about 0.1% extra parameters. To avoid bias from a trajectory predicting its own value, POISE uses a cross-rollout scheme where features from one sampled trajectory serve as unbiased features for estimating the value of another. Experiments on Qwen3-4B and DeepSeek-R1-Distill-Qwen-1.5B show POISE matches DAPO performance on math reasoning at significantly lower compute cost, requiring only a single rollout per prompt versus GRPO's 8-16, and its probe's value estimation accuracy approaches that of independent LLM-scale value models. The work challenges the assumption that RL training requires a dedicated value network, showing the policy model already encodes reward-predictive information during generation.

POISE: Your Language Model Is Its Own Critic — Value Estimation from the Actor's Internal States

> In May 2026, Choi et al. proposed POISE (Policy Optimization with Internal State Value Estimation), an RLVR method that leverages a language model's own internal representations for value estimation. By extracting hidden states and token entropy statistics during the policy's forward pass, POISE builds a lightweight value probe at negligible cost — eliminating PPO's need for a separate critic model and GRPO's need for multi-rollout group means. Experiments on Qwen3-4B and DeepSeek-R1-Distill-Qwen-1.5B show POISE matches DAPO performance at significantly lower compute cost, with a value estimator whose accuracy approaches that of an independent LLM-scale value model.

Key Points

The baseline dilemma in RLVR: Rewards arrive only at sequence end, making credit assignment hard. Existing solutions are costly:
PPO: trains a critic network roughly the size of the policy
GRPO: requires 8–16 rollouts per prompt for group-mean baselines
DAPO: needs complex dynamic filtering/scheduling logic
Core insight: The policy model's internal representations already implicitly encode predictions of the final reward. Hidden states encode "confidence" in the current reasoning path; token entropy reflects uncertainty patterns — signals computed but unused in the forward pass.
Three-source value probe: The probe combines (1) the prompt's final-layer hidden state, (2) aggregated trajectory hidden states, and (3) token entropy statistics. At ~0.1% of the policy model's parameters, its overhead is negligible. It is trained jointly and online with the policy via MSE loss on reward prediction, and its output serves as the policy-gradient baseline.
Cross-rollout for unbiased estimation: Predicting a trajectory's value from its own internal features risks overfitting to its specific token sequence. POISE symmetrically swaps features: features from trajectory B estimate V(τ_A), and vice versa. Since both trajectories come from the same policy and prompt, features are unbiased while being blind to the target's specific tokens.
Efficiency gains:
vs. GRPO: 1 rollout per prompt instead of 8–16 (6.25%–12.5% of sampling FLOPs); 8–16× more prompt diversity at fixed budget; zero-advantage prompts (all-correct/all-wrong groups) are avoided entirely by design
vs. PPO: critic parameters drop to ~0.1% of policy size; total memory ~1.001× instead of 2×; one forward pass per step instead of two
Experimental validation: On Qwen3-4B and DeepSeek-R1-Distill-Qwen-1.5B, POISE matches DAPO on math reasoning at lower cost. The tiny probe's value-estimation accuracy approaches that of an independent LLM-scale value model, and it generalizes across multiple verifiable tasks beyond math.

Relation to Prior Work

Complementary to "Beyond the 80/20 Rule": That work identified high-entropy tokens as key RLVR optimization targets and pruned gradients to 20% of tokens; POISE uses entropy statistics as probe input for baseline compression. Combined: fewer tokens × fewer rollouts × no critic.
Self-critique lineage: Following STaR/Self-Refine and Self-Rewarding, POISE's distinction is that evaluation happens online, during generation — providing real-time baseline signals rather than post-hoc self-assessment.

Limitations and Open Questions

Scale: Validated only at 1.5B–4B. Do internal states retain sufficient value information at 14B+?
Representation degradation: As training progresses, internal representations may "harden" (cf. low-entropy token invariance), possibly affecting probe generalization.
Integration: Potential combinations with REINFORCE++ (value baseline), DAPO (dynamic sampling), and RLOO (leave-one-out variance reduction).
Interpretability: Which hidden dimensions encode value? Do "value-sensitive neurons" exist, analogous to fork tokens?

Conclusion

POISE offers a minimalist, efficient solution to baseline estimation in RLVR by exploiting signals the policy already computes. Its central claim — the model already "knows" the quality of its answer during generation — challenges the traditional assumption that RL requires an independent value network. Amid rising LLM training costs, POISE exemplifies a shift toward smarter use of existing computation rather than more parameters or samples.

Paper Details

| Item | Content | |:---|:---| | Title | Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States | | Authors | Yunho Choi, Jongwon Lim, Woojin Ahn, Minjae Oh, Jeonghoon Shim, Yohan Jo | | arXiv ID | 2605.07579 | | Date | 2026-05-08 | | Key contributions | Internal-state value estimation; lightweight probe; unbiased cross-rollout construction; single-rollout matching DAPO; elimination of zero-advantage sampling waste | | Key results | Matches DAPO on Qwen3-4B / DeepSeek-R1-Distill-Qwen-1.5B; probe performance near LLM-scale value models; generalizes across verifiable tasks |