Legibility is Not Interpretability: Can LLM Judges Identify Important Reasoning Steps?

Forum topic · 小凯 · 2026-09-05

Summary

This paper (arXiv:2609.04194) by Kevin Du, Alexander Hoyle, and Laura Ruis examines whether chain-of-thought reasoning traces actually reveal which reasoning steps matter. The authors operationalize step importance as its advantage—the change in expected reward (e.g., probability of a correct answer) from including the step, estimated via Monte Carlo rollouts. Using these estimates as ground truth, they evaluate whether LLM judges can identify high-advantage steps. Findings: sufficiently capable LLMs beat a prevalence baseline but fall well short of a noise ceiling. Fine-tuning models as step-level critics improves performance on incorrect responses but remains far from the ceiling on correct ones, suggesting step importance is only partially recoverable from the text of reasoning traces. The work contributes to chain-of-thought faithfulness research and warns against equating the legibility of reasoning traces with interpretability, with significant implications for process reward modeling and LLM-judge-based oversight.

Paper Overview

Field: NLP Authors: Kevin Du, Alexander Hoyle, Laura Ruis arXiv: 2609.04194

Motivation

Chain-of-thought (CoT) reasoning traces appear to offer a readable window into how a model arrives at its answer. A growing body of work treats them as interpretable: LLM judges are used to diagnose errors, assess faithfulness, and provide step-level supervision via process reward models and generative critics. These practices assume that the text of a reasoning step carries information about its functional role.

Method

The authors operationalize the importance of a reasoning step as its advantage: the change in expected reward (e.g., the probability of producing a correct answer) from including that step, estimated via Monte Carlo rollouts. These estimates serve as ground truth for judging step importance.

Findings

LLM judges can identify high-advantage steps: sufficiently capable LLMs outperform a prevalence baseline.
However, they fall well short of a noise ceiling, meaning much of the signal remains inaccessible.
Fine-tuning models as step-level critics yields clear improvement on incorrect responses, but performance on correct responses remains far from the ceiling.
Conclusion: step importance is only partially recoverable from the text of reasoning traces.

Implications

The findings caution against equating the *legibility* of reasoning traces with *interpretability*. This has important consequences for the growing CoT faithfulness literature and especially for process reward modeling and LLM-judge-based step-level supervision.

---

*Auto-collected on 2026-09-05*