RREDCoT: Segment-Level Reward Redistribution for Reasoning Models — Why Chain-of-Thought Needs Process-Based Pay

Forum topic · 小凯 · 2026-06-06

Summary

A zhichai.net forum post reviews the paper "RREDCoT: Segment-Level Reward Redistribution for Reasoning Models" by Ielanskyi, Schweighofer, Aichberger, and Sepp Hochreiter (arXiv:2606.06475). The paper tackles the credit assignment problem in training reasoning LLMs such as DeepSeek R1 and OpenAI o1/o3, where GRPO-style reinforcement learning assigns a single outcome-based reward to an entire chain of thought. RREDCoT instead redistributes the final reward to individual segments of the reasoning chain, using the model's own value and probability estimates rather than expensive Monte Carlo resampling. The post explains the method with an extended detective metaphor of 'process wages,' compares RREDCoT to MC sampling, RLOO, and PPO, and reports that experiments on mathematical reasoning show higher accuracy than standard GRPO, with the largest gains on long chains (500+ tokens). It also discusses attribution heatmaps for interpreting model reasoning, implications for reducing reward hacking, and connections to Hochreiter's earlier work on long-sequence learning via LSTM.

RREDCoT: Segment-Level Reward Redistribution for Reasoning Models

This forum post on zhichai.net reviews the paper "RREDCoT: Segment-Level Reward Redistribution for Reasoning Models" (Ielanskyi, Schweighofer, Aichberger & Hochreiter, arXiv:2606.06475), framed around a metaphor: paying a detective only when the case is solved — with no feedback on which investigative steps were right — mirrors how reasoning LLMs are trained today.

The Problem: Credit Assignment in Reasoning Models

Modern reasoning models (DeepSeek R1, OpenAI o1/o3) generate long chains of thought (CoT) before producing a final answer. They are typically trained with GRPO (Group Relative Policy Optimization), which:

1. Samples multiple chains of thought (e.g., 16) per problem. 2. Scores each by the final answer's correctness relative to the group. 3. Applies the same reward or penalty to the entire chain.

This Monte Carlo-style approach has high variance and cannot distinguish which of ~1000 tokens actually determined correctness — the classic credit assignment problem. As the post puts it: a company that bonuses or penalizes the whole team equally regardless of individual contribution.

The Method: Process Wages per Segment

RREDCoT's core idea: redistribute the final-answer reward across segments of the reasoning chain, so each segment (e.g., "set up the equation," "solve," "verify") receives credit proportional to its contribution to the outcome.

Key design choices:

No extra sampling: instead of MC resampling (regenerating many continuations per segment), RREDCoT uses the model's own value estimates and the probability distributions it already computed during generation as implicit value signals.
Approximation over exactness: the paper argues a reasonable approximation of segment-level contribution (in the spirit of Shapley-value attribution) suffices.
Low overhead, low variance, making it practical for long chains (1000+ tokens) where MC sampling is infeasible.

| Method | Extra compute | Variance | Suitability | |---|---|---|---| | MC sampling | High | Medium | Short chains | | RREDCoT | Low (reuses existing probabilities) | Low | Long chains |

Experimental Findings

1. Outperforms standard GRPO on mathematical reasoning (algebra, geometry, number theory) at equal training steps. 2. Larger gains on long chains (500+ tokens), where process-level feedback matters most. 3. Attribution heatmaps visualize which segments contributed positively (high reward) versus misleading detours — useful both for training and for interpreting whether the model is genuinely reasoning or "fake thinking." 4. Compared favorably to RLOO, PPO (which requires a separate value network), and direct MC sampling, without extra training or sampling cost.

Broader Implications

Emerging metacognition: rewarding segments encourages the model to implicitly evaluate its own reasoning process, a primitive form of reflection.
AI safety / reward hacking: by rewarding high-quality reasoning rather than only correct answers, RREDCoT reduces the incentive to skip or fake the reasoning process.
Historical symmetry: co-author Sepp Hochreiter (co-inventor of LSTM) again addresses how to preserve important signals across long sequences — from 1997's long-range dependencies to 2026's long chains of thought.

Limitations and Future Directions

The post notes acknowledged limitations: how to segment chains meaningfully and automatically; reliability of the model's own value estimates, especially in unfamiliar domains; and credit propagation across multi-hop reasoning. Future directions include token-level process supervision, multimodal reasoning chains (images, code, formulas), and self-bootstrapped improvement based on RREDCoT-style attribution analysis.

References

Ielanskyi, M., Schweighofer, K., Aichberger, L., & Hochreiter, S. (2026). *RREDCoT: Segment-Level Reward Redistribution for Reasoning Models*. arXiv:2606.06475.
Shao, Z., et al. (2024). DeepSeekMath. arXiv:2402.03300.
Schulman, J., et al. (2017). Proximal policy optimization algorithms. arXiv:1707.06347.
Shapley, L. S. (1953). A value for n-person games. *Contributions to the Theory of Games*, 2(28), 307-317.
Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. *Neural Computation*, 9(8), 1735-1780.
Sutton, R. S., & Barto, A. G. (2018). *Reinforcement Learning: An Introduction*. MIT Press.