Legibility is Not Interpretability: Why Reasoning Traces Don't Reveal Which Steps Matter

Forum topic · 小凯 · 2026-09-05

Summary

This paper (arXiv:2609.04194) by Kevin Du, Alexander Hoyle, and Laura Ruis examines whether chain-of-thought reasoning traces actually encode which reasoning steps are functionally important. The authors operationalize step importance as its advantage: the change in expected reward from including that step, estimated via Monte Carlo rollouts. Using these estimates as ground truth, they evaluate whether LLM judges can identify high-advantage steps. They find that sufficiently capable LLMs can outperform a prevalence baseline at this task but fall well short of a noise ceiling. Fine-tuning models as step-level critics improves performance on erroneous responses but still leaves correct-response performance far from the ceiling, suggesting step importance is only partially recoverable from the text of reasoning traces. The findings caution against equating the legibility of reasoning traces with interpretability, with significant implications for process reward modeling and LLM-judged chain-of-thought fidelity evaluations.

Paper Overview

Field: NLP
Authors: Kevin Du, Alexander Hoyle, Laura Ruis
Published: 2026-09-03
arXiv: 2609.04194

Background

The reasoning traces of chain-of-thought models appear to offer a readable window into how models arrive at their answers. A growing body of work treats them as interpretable: using LLM judges to diagnose errors, assess faithfulness, and provide step-level supervision via process reward models and generative critics. These practices all rest on the assumption that the text of a reasoning step carries information about its functional role.

But does the text really encode which reasoning steps matter?

Method

The authors operationalize the importance of a reasoning step as its advantage: the change in expected reward from including that step (e.g., the probability of producing a correct answer), estimated via Monte Carlo rollouts. These estimates serve as ground truth for evaluating whether LLM judges can identify high-advantage steps.

Key Findings

Sufficiently capable LLM judges can identify high-advantage steps better than a prevalence baseline, but perform far below a noise ceiling.
Fine-tuning models as step-level critics yields clear improvement on erroneous responses, but performance on correct responses remains far from the ceiling.
This indicates that step importance is only partially recoverable from the text of reasoning traces.

Implications

The findings contribute to the growing literature on chain-of-thought faithfulness and warn against equating the legibility of reasoning traces with interpretability — with important consequences for process reward modeling.

--- *Auto-collected on 2026-09-05*