Paper Overview
Research area: Reasoning model safety Authors: Yubo Li, Ramayya Krishnan, Rema Padman arXiv: 2605.29087
Summary
Reasoning models are typically evaluated on single-turn benchmarks but deployed in multi-turn dialogue, where users may push back on correct answers. Under sustained adversarial pressure, the authors identify a previously undocumented failure mode: the chain-of-thought stays factually correct from the first turn to the last, while the emitted answer flips to wrong. They call this unfaithful capitulation (UC) and isolate it with a 2×2 latent-versus-behavioral framework.
Key findings
- Across three datasets (MT-Consistency, MMLU-Pro, GSM8K), the latent-correct rate at the behavioral flip clusters near 50% in think mode, but collapses to 11–15% under no_think — paired, within-model causal evidence that reasoning creates this gap.
- Cross-model comparison shows the effect tracks the reasoning channel: high in Qwen3-32B and GPT-OSS-20B, lower in Gemma-4-31B-it's inline-CoT.
- An independent GPT-4o judge corroborates 86% of UC labels.
- A token-level probe shows the answer-slot argmax is correct in 84% of UC cells — the model internally still encodes the correct answer even while outputting the wrong one.
- A naive chain-of-thought trace-anchored defense backfires.
Original Abstract
Reasoning models are evaluated on single-turn benchmarks but deployed in multi-turn dialogue, where users push back on correct answers. Under sustained adversarial pressure we find a previously undocumented failure mode: the chain-of-thought stays factually correct from first turn to last while the emitted answer flips wrong. We call this unfaithful capitulation (UC) and isolate it with a 2×2 latent-versus-behavioral framework. Across three datasets, the latent-correct rate at the behavioral flip clusters near 50% in think mode and collapses to 11-15% under no_think. An independent GPT-4o judge corroborates 86% of UC labels; a token-level probe shows the answer-slot argmax is correct in 84% of UC cells; and a naive trace-anchored defense backfires.