Your Language Model is Its Own Critic: POISE Extracts Value Signals from Actor Internal States for RLVR

Forum topic · 小凯 · 2026-05-11

Summary

POISE (Policy Optimization with Internal State Value Estimation), proposed by Choi et al., is a new RLVR method that replaces the traditional critic network by estimating value directly from the policy model's own internal representations. During the policy's forward pass, POISE extracts final-layer hidden states and token entropy statistics, feeding them into a lightweight value probe (~0.1% of policy parameters) that predicts expected reward as a baseline. To avoid bias from a probe overfitting to its own trajectory, POISE uses a symmetric cross-rollout scheme: features from one independently sampled rollout predict the value of another, ensuring unbiased estimation. Compared to PPO, POISE removes the policy-sized critic network and halves forward passes; compared to GRPO, it requires only a single rollout per prompt (vs. 8-16), eliminating wasted zero-advantage prompts. Experiments on Qwen3-4B and DeepSeek-R1-Distill-Qwen-1.5B show POISE matches DAPO performance at significantly lower compute cost, with probe accuracy approaching independent LLM-scale value models. The core insight: the model already 'knows' the quality of its reasoning chain through signals computed during generation.

Your Language Model is Its Own Critic: Extracting Value Signals from Actor Internal States for RLVR

> In May 2026, Choi et al. proposed POISE (Policy Optimization with Internal State Value Estimation), an RLVR method that leverages a language model's own internal representations for value estimation. By extracting hidden states and token entropy statistics during the policy's forward pass, POISE builds a lightweight value probe at minimal overhead — eliminating PPO's dependence on a separate critic model and GRPO's need for multi-rollout group averages. Experiments on Qwen3-4B and DeepSeek-R1-Distill-Qwen-1.5B show POISE matches DAPO performance at significantly lower compute cost, with value estimation accuracy approaching independent LLM-scale value models.

1. Background: The Baseline Dilemma in RLVR

1.1 Why variance reduction matters

A core challenge in reinforcement learning is gradient variance. For LLM reasoning tasks, rewards typically appear only at the end of the sequence (answer correct/incorrect), making credit assignment difficult. A baseline estimate reduces variance:

\nabla_\theta J \approx \mathbb{E}_{\tau \sim \pi_\theta}\left[\sum_{t} \nabla_\theta \log \pi_\theta(a_t|s_t) \cdot (R(\tau) - b(s_t))\right]

where $$b(s_t)$$ is the baseline, ideally equal to the state value $$V(s_t)$$ .

1.2 The cost of existing methods

| Method | Baseline mechanism | Main cost | |:---|:---|:---| | PPO | Independently trained critic network $V_\phi(s)$ | An extra network the size of the policy model | | GRPO | Mean reward of rollout group $\bar{R}_G$ | 8-16 samples per prompt | | DAPO | Dynamic filtering + sampling adjustments | Complex online scheduling logic |

> Common bottleneck: whichever method is used, obtaining a baseline incurs significant compute or memory overhead.

2. POISE: A Minimal Reconstruction of Actor-Critic

2.1 Core assumption

POISE rests on a key observation: as the policy model generates a reasoning chain, its internal representations already implicitly encode predictive information about the final reward.

Specifically:

Hidden states $$h_t$$ encode the model's "confidence" in the current reasoning path
Token entropy distribution reflects uncertainty patterns across the sequence
These signals are already computed in the forward pass, just unused

2.2 A three-source value probe

| Input feature | Source | Dim | Information content | |:---|:---|:---:|:---| | Prompt final hidden state | Policy model's last layer | $d_{model}$ | Question representation | | Trajectory hidden-state aggregation | Hidden states along the reasoning chain | $d_{model}$ | Reasoning-process representation | | Token entropy statistics | Distributional features of sequence entropy | $$k$$ | Uncertainty patterns |

> Probe size: typically only ~0.1% of policy parameters; compute overhead is negligible.

2.3 Online joint training

The probe is updated synchronously with the policy: 1. Policy forward pass generates a reasoning chain 2. Hidden states and entropy statistics are extracted simultaneously 3. The probe predicts the trajectory's expected reward 4. Prediction error trains the probe via MSE loss 5. The policy gradient uses the probe's prediction as the baseline

3. Cross-Rollout: The Key to Unbiased Value Estimation

3.1 Trajectory-conditioned bias

If the probe directly uses trajectory $$A$$ 's internal features to predict $$A$$ 's own value, it may overfit to $$A$$ 's specific token sequence, biasing gradient estimation:

\hat{V}(A) = f(\text{states}_A) \rightarrow \text{overfits to } A \text{'s token sequence}

3.2 Cross-rollout construction

POISE introduces a symmetric cross-prediction mechanism:

| Prediction target | Feature source | Unbiasedness guarantee | |:---:|:---:|:---| | $V(\tau_A)$ | $\text{states}_{\tau_B}$ | $\tau_B$ is sampled independently; blind to $\tau_A$ 's specific tokens | | $V(\tau_B)$ | $\text{states}_{\tau_A}$ | $\tau_A$ is sampled independently; blind to $\tau_B$ 's specific tokens |

where $\tau_A, \tau_B \sim \pi_\theta(\cdot | \text{prompt})$ .

> Theoretical guarantee: since $\tau_A$ and $\tau_B$ come from the same policy and prompt distribution, $\text{states}_{\tau_B}$ are unbiased features for $V(\tau_A)$ . Meanwhile, $\tau_B$ 's specific token sequence is independent of $\tau_A$ , preventing overfitting.

4. Compute Efficiency Analysis

4.1 Quantitative comparison with GRPO

| Metric | GRPO | POISE | Ratio | |:---|:---:|:---:|:---:| | Rollouts per prompt | 8-16 | 1 | 1/8 ~ 1/16 | | Sampling FLOPs | $8-16 \times L \cdot d^2$ | $1 \times L \cdot d^2$ | 12.5% ~ 6.25% | | Prompt diversity (fixed budget) | Baseline | 8-16× | 800% ~ 1600% | | Zero-advantage prompt detection cost | Extra sampling needed | Zero | 0% |

> Zero-advantage elimination: in GRPO, many prompts produce all-wrong or all-correct rollouts, giving zero within-group variance and zero gradient contribution. POISE's single rollout + value estimation naturally avoids this waste.

4.2 Quantitative comparison with PPO

| Metric | PPO | POISE | Ratio | |:---|:---:|:---:|:---:| | Critic parameter count | $\approx |\theta|$ | $\approx 0.001 \cdot |\theta|$ | ~0.1% | | Total memory footprint | $2 \times$ | $\approx 1.001 \times$ | ~50% | | Forward passes per step | 2 (actor + critic) | 1 | 50% |

5. Experimental Validation

5.1 Performance matching

| Model | Method | Math reasoning | Compute cost | |:---|:---|:---:|:---:| | Qwen3-4B | DAPO | Baseline | Baseline | | Qwen3-4B | POISE | Matched | Lower | | DeepSeek-R1-Distill-Qwen-1.5B | DAPO | Baseline | Baseline | | DeepSeek-R1-Distill-Qwen-1.5B | POISE | Matched | Lower |

5.2 Value estimation accuracy

Comparison between POISE's lightweight probe and an independent LLM-scale value model:

| Estimator | Size | Performance | Conclusion | |:---|:---:|:---:|:---| | Independent LLM-scale value model | Policy-model scale | Baseline | High cost | | POISE probe | ~0.1% of policy model | Near baseline | Extremely cost-effective |

> Key finding: the policy model's internal states already contain enough information to predict the final reward — no extra LLM-scale network is needed to learn it.

5.3 Generalization

The value estimator performs stably across multiple verifiable tasks, not limited to math reasoning.

6. Relation to Related Work

6.1 Complementarity with Round 14

"Beyond the 80/20 Rule" (Round 14) found that high-entropy tokens are the key optimization targets in RLVR. POISE happens to use token entropy statistics as one input feature of its value probe, forming method-level complementarity:

| Work | Core operation | Level | |:---|:---|:---| | Round 14 | Entropy-based selection of 20% key tokens for gradient updates | Gradient compression | | This paper | Value estimation from internal states (incl. entropy stats) | Baseline compression | | Combined effect | Fewer tokens × fewer rollouts × no critic | Comprehensive efficiency gains |

6.2 Connection to the self-critique paradigm

POISE continues the "model self-evaluation" research line:

STaR / Self-Refine: model self-corrects after generation
Self-Rewarding: model trains with its own preference judgments
POISE: model evaluates value in real time from internal states

> POISE's uniqueness: evaluation happens during generation (online), not after it (offline), providing real-time baseline signals for RL training.

7. Limitations and Open Questions

7.1 Scale extension

Current validation is limited to 1.5B-4B models. At larger scale (14B+), do internal states still contain sufficient value information? Does the probe need to grow with model size?

7.2 Representation degradation

As policy training proceeds, internal representations may gradually "harden" (similar to the low-entropy token invariance observed in Round 14). Could this hurt the probe's generalization?

7.3 Integration with advanced RL algorithms

POISE currently mainly serves as a PPO-style baseline estimator. Integration potential with:

REINFORCE++: directly replacing its value baseline
DAPO: combined with dynamic sampling strategies
RLOO: leveraging leave-one-out variance reduction

7.4 Interpretability

Which dimensions of the hidden states encode value information? Do "value-sensitive neurons" exist, analogous to Round 14's "forking tokens"?

8. Conclusion

POISE offers a minimal, highly efficient solution to RLVR's baseline estimation problem by exploiting internal signals the policy model has already computed but not used. Its core insight — the model already "knows" the quality of the answer during generation — challenges the traditional RL assumption that an independent value network is required.

As LLM training costs keep rising, POISE represents the trend toward "compute efficiency" and "maximal signal utilization": instead of adding models or sampling, use information already produced by existing computation more cleverly.

Paper Details

| Item | Content | |:---|:---| | Title | Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States | | Authors | Yunho Choi, Jongwon Lim, Woojin Ahn, Minjae Oh, Jeonghoon Shim, Yohan Jo | | arXiv ID | 2605.07579 | | Date | 2026-05-08 | | Core contributions | POISE: internal-state value estimation; lightweight probe; unbiased cross-rollout construction; DAPO-matching with a single rollout; eliminating zero-advantage sampling waste | | Key results | Qwen3-4B / DeepSeek-R1-Distill-Qwen-1.5B match DAPO; probe performance near independent LLM-scale value models; generalizes to multiple verifiable tasks |