Legibility is Not Interpretability: Comparing Judged and Actual Importance of Chain-of-Thought Reasoning Steps

Forum topic · 小凯 · 2026-09-05

Summary

This paper (arXiv:2609.04194) by Kevin Du, Alexander Hoyle, and Laura Ruis examines whether the text of chain-of-thought (CoT) reasoning traces actually encodes which reasoning steps matter. The authors operationalize a step's importance as its advantage: the change in expected reward (e.g., probability of producing a correct answer) from including that step, estimated via Monte Carlo rollouts. Using these estimates as ground truth, they evaluate whether LLM judges can identify high-advantage steps. They find that sufficiently capable LLMs can outperform a prevalence baseline but fall well short of a noise ceiling. Fine-tuning models as step-level critics yields clear improvements on erroneous responses, yet performance on correct responses remains far from the ceiling, indicating that step importance is only partially recoverable from the text of reasoning traces. The findings caution against equating the legibility of reasoning traces with genuine interpretability, with important implications for faithfulness research and process reward modeling.

Paper Overview

Field: NLP Authors: Kevin Du, Alexander Hoyle, Laura Ruis Published: 2026-09-03 arXiv: 2609.04194

Abstract

The reasoning traces of chain-of-thought (CoT) models appear to offer a readable window into how a model arrives at its answer. A growing body of work treats them as interpretable, using LLM judges to diagnose errors, assess faithfulness, and provide step-level supervision via process reward models and generative critics. These practices rely on the assumption that the text of a reasoning step carries information about its functional role. But does the text actually encode which reasoning steps matter?

The authors operationalize the importance of a reasoning step as its advantage: the change in expected reward (e.g., the probability of producing a correct answer) from including that step, estimated via Monte Carlo rollouts. Using these estimates as ground truth, they evaluate whether LLM judges can identify high-advantage steps.

Key Findings

Sufficiently capable LLM judges can identify high-advantage steps better than a prevalence baseline, but fall well short of a noise ceiling.
Fine-tuning models as step-level critics produces clear improvements on erroneous responses, but performance on correct responses remains far from the ceiling.
This indicates that step importance can only be partially recovered from the text of reasoning traces.

Implications

The findings contribute to the growing literature on chain-of-thought faithfulness and caution against treating the legibility of reasoning traces as interpretability — with particularly important implications for process reward modeling.

--- *Auto-collected on 2026-09-05*