POISE: Your Language Model is Its Own Critic - Value Estimation from the Actor's Internal States for RLVR

Forum topic · 小凯 · 2026-05-11

Summary

POISE (Policy Optimization with Internal State Value Estimation), proposed by Choi et al., is a new paradigm for verifiable-reward reinforcement learning (RLVR) that eliminates the need for a separate critic model. The method extracts hidden states and token entropy statistics from the policy model's forward pass to train a lightweight value probe (roughly 0.1% of the policy's parameters) that predicts expected reward and serves as a baseline, removing PPO's separate critic and GRPO's need for 8-16 rollouts per prompt. To avoid overfitting to a trajectory's own token sequence, POISE uses cross-rollout conditioning: features from one rollout predict the value of another independently sampled rollout, preserving unbiasedness. Experiments on Qwen3-4B and DeepSeek-R1-Distill-Qwen-1.5B show POISE matches DAPO performance at significantly lower compute cost, with probe accuracy approaching that of independent LLM-scale value models, generalizing across verifiable tasks. Key open questions include scaling beyond 1.5B-4B models and representation degradation during training.

Your Language Model is Its Own Critic: POISE, a New RLVR Paradigm Extracting Value Signals from the Actor's Internal States

> In May 2026, Choi et al. proposed POISE (Policy Optimization with Internal State Value Estimation), an RLVR method that uses a language model's own internal representations for value estimation. By extracting hidden states and token entropy statistics during the policy's forward pass, POISE builds a lightweight value probe at negligible overhead — eliminating PPO's dependence on a separate critic model and GRPO's need for multi-rollout group means. Experiments on Qwen3-4B and DeepSeek-R1-Distill-Qwen-1.5B show POISE matches DAPO performance at significantly lower compute cost, with value estimation accuracy approaching that of an independent LLM-scale value model.

1. Background: The Baseline Dilemma in RLVR

1.1 Why Variance Reduction Matters

A core challenge in reinforcement learning is gradient variance. For LLM reasoning tasks, rewards typically appear only at the end of the sequence (answer correct/incorrect), making credit assignment difficult. A baseline is used to reduce variance:

\nabla_\theta J \approx \mathbb{E}_{\tau \sim \pi_\theta}\left[\sum_{t} \nabla_\theta \log \pi_\theta(a_t|s_t) \cdot (R(\tau) - b(s_t))\right]

where $$b(s_t)$$ is the baseline, ideally equal to the state value $$V(s_t)$$ .

1.2 The Cost of Existing Methods

| Method | Baseline mechanism | Main cost | |:---|:---|:---| | PPO | Independently trained critic network $V_\phi(s)$ | Extra network the size of the policy model | | GRPO | Mean reward over rollout group $\bar{R}_G$ | 8–16 samples per prompt | | DAPO | Dynamic filtering + sampling adjustments | Complex online scheduling logic |

> Common bottleneck: whatever the method, obtaining a baseline carries significant computational or memory overhead.

2. POISE: A Minimalist Restructuring of Actor-Critic

2.1 Core Hypothesis

POISE rests on a key observation: as the policy model generates a reasoning chain, its internal representations already implicitly encode predictive information about the final reward.

Specifically:

Hidden states $$h_t$$ encode the model's "confidence" in the current reasoning path
Token entropy distributions reflect uncertainty patterns across the sequence
These signals are already computed in the forward pass — just unused

2.2 A Three-Source Value Probe

| Input feature | Source | Dim | Information | |:---|:---|:---:|:---| | Prompt final hidden state | Policy model last layer | $d_{model}$ | Question representation | | Trajectory hidden-state aggregation | Hidden states along the reasoning chain | $d_{model}$ | Reasoning process representation | | Token entropy statistics | Distributional features of sequence entropy | $$k$$ | Uncertainty pattern |

> Probe size: typically only ~0.1% of the policy model's parameters; compute overhead is negligible.

2.3 Online Joint Training

The probe is trained synchronously with the policy: 1. The policy forward pass generates a reasoning chain 2. Hidden states and entropy statistics are extracted simultaneously 3. The probe predicts the trajectory's expected reward 4. The prediction error trains the probe via an MSE loss 5. The policy gradient uses the probe's prediction as baseline

3. Cross-Rollout: The Key to Unbiased Value Estimation

3.1 Trajectory-Conditioned Bias

If the probe uses trajectory $$A$$ 's internal features to predict $$A$$ 's own value, it may overfit to $$A$$ 's specific token sequence, biasing the gradient estimate:

\hat{V}(A) = f(\text{states}_A) \rightarrow \text{overfits to } A\text{'s token sequence}

3.2 Cross-Rollout Construction

POISE introduces a symmetric cross-prediction mechanism:

| Prediction target | Feature source | Unbiasedness guarantee | |:---:|:---:|:---| | $V(\tau_A)$ | $\text{states}_{\tau_B}$ | $\tau_B$ sampled independently, blind to $\tau_A$ 's specific tokens | | $V(\tau_B)$ | $\text{states}_{\tau_A}$ | $\tau_A$ sampled independently, blind to $\tau_B$ 's specific tokens |

where $\tau_A, \tau_B \sim \pi_\theta(\cdot | \text{prompt})$ .

> Theoretical guarantee: since $\tau_A$ and $\tau_B$ come from the same policy and prompt distribution, $\text{states}_{\tau_B}$ provide unbiased features for $V(\tau_A)$ , while $\tau_B$ 's specific token sequence is independent of $\tau_A$ , preventing overfitting.

4. Compute Efficiency Analysis

4.1 Quantitative Comparison with GRPO

| Metric | GRPO | POISE | Ratio | |:---|:---:|:---:|:---:| | Rollouts per prompt | 8–16 | 1 | 1/8 – 1/16 | | Sampling FLOPs | $8{-}16 \times L \cdot d^2$ | $1 \times L \cdot d^2$ | 12.5% – 6.25% | | Prompt diversity (fixed budget) | baseline | 8–16× | 800% – 1600% | | Zero-advantage prompt detection cost | Extra sampling needed | Zero | 0% |

> Zero-advantage elimination: in GRPO, many prompts produce all-wrong or all-correct rollouts with zero within-group variance and zero gradient contribution. POISE's single rollout + value estimation naturally avoids this waste.

4.2 Quantitative Comparison with PPO

| Metric | PPO | POISE | Ratio | |:---|:---:|:---:|:---:| | Critic parameters | $\approx |\theta|$ | $\approx 0.001 \cdot |\theta|$ | ~0.1% | | Total memory | $2 \times$ | $\approx 1.001 \times$ | ~50% | | Forward passes per step | 2 (actor + critic) | 1 | 50% |

5. Experimental Validation

5.1 Performance Matching

| Model | Method | Math reasoning | Compute cost | |:---|:---|:---:|:---:| | Qwen3-4B | DAPO | baseline | baseline | | Qwen3-4B | POISE | matches | lower | | DeepSeek-R1-Distill-Qwen-1.5B | DAPO | baseline | baseline | | DeepSeek-R1-Distill-Qwen-1.5B | POISE | matches | lower |

5.2 Value Estimation Accuracy

| Estimator | Scale | Performance | Conclusion | |:---|:---:|:---:|:---| | Independent LLM-scale value model | Policy-model scale | baseline | High cost | | POISE probe | ~0.1% of policy model | near baseline | Excellent cost-performance |

> Key finding: the policy model's internal states already contain sufficient information to predict the final reward — no extra LLM-scale network is needed to learn it.

5.3 Generalization

The value estimator performs stably across multiple verifiable tasks, not limited to math reasoning.

6. Relation to Related Work

6.1 Complementarity with "Beyond the 80/20 Rule" (Round 14)

Round 14 found that high-entropy tokens are the key optimization target in RLVR. POISE happens to use token entropy statistics as one input to its value probe, forming a methodological complement:

| Work | Core operation | Level | |:---|:---|:---| | Round 14 | Entropy-based filtering of the 20% key tokens for gradient updates | Gradient compression | | This paper | Internal-state (incl. entropy) based value estimation | Baseline compression | | Combined effect | Fewer tokens × fewer rollouts × no critic | Overall efficiency gains |

6.2 Connection to the Self-Critique Paradigm

POISE continues the "model self-evaluation" research line:

STaR / Self-Refine: model corrects itself after generation
Self-Rewarding: model trains with its own preference judgments
POISE: model evaluates value in real time from internal states

> POISE's uniqueness: the evaluation happens during generation (online), not after (offline), providing real-time baseline signals for RL training.

7. Limitations and Open Questions

7.1 Scale

Current validation covers 1.5B–4B models. At larger scales (14B+), do internal states still contain enough value information? Does the probe need to grow with model size?

7.2 Representation Degradation

As policy training proceeds, internal representations may "harden" (similar to the low-entropy token invariance observed in Round 14). Could this hurt the probe's generalization?

7.3 Integration with Advanced RL Algorithms

POISE currently serves mainly as a PPO-style baseline estimator. Potential integrations:

REINFORCE++: direct replacement of its value baseline
DAPO: combined with dynamic sampling strategies
RLOO: leveraging leave-one-out variance reduction

7.4 Interpretability

Which hidden-state dimensions encode value information? Are there "value-sensitive neurons" corresponding to Round 14's "fork tokens"?

8. Conclusion

POISE offers a minimalist and efficient solution to RLVR's baseline estimation problem by exploiting internal signals the policy model has already computed but never used. Its core insight — the model already "knows" the quality of its answer during generation — challenges the traditional RL assumption that a separate value network is required.

As LLM training costs keep rising, POISE represents the trend toward "compute efficiency" and "signal utilization maximization": no longer adding models or sampling, but using information produced by existing computation more cleverly.

---

Paper Details

| Item | Content | |:---|:---| | Title | Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States | | Authors | Yunho Choi, Jongwon Lim, Woojin Ahn, Minjae Oh, Jeonghoon Shim, Yohan Jo | | arXiv ID | 2605.07579 | | Date | 2026-05-08 | | Core contributions | POISE: internal-state value estimation; lightweight probe; unbiased cross-rollout construction; single-rollout DAPO matching; elimination of zero-advantage sampling waste | | Key results | Matches DAPO on Qwen3-4B / DeepSeek-R1-Distill-Qwen-1.5B; probe performance approaches independent LLM-scale value models; generalizes to multiple verifiable tasks |