Train the Model, Not the Reader: Decodability Supervision for Verifiable Explanations (RECAP, arXiv 2507.18397)

Forum topic · 小凯 · 2026-07-24

Summary

This forum post introduces arXiv paper 2507.18397 by Hiskias Dingeto, which examines faithfulness testing in natural-language autoencoders used for interpretability. The paper shows that reconstruction-based evaluation is structurally insensitive to individual false claims: on a released Qwen-2.5-7B verbalizer, explanations reconstruct well above chance while ~2% of specific claims are reconstruction-dependent, meaning the score tracks gist rather than facts. Under exact synthetic ground truth, the standard recipe developed co-adapted private codes (false wording the reconstruction depends on) in 5/5 runs, and fixes leaving the target model unchanged did not help. The authors contribute two audit protocols (grounded-vs-true crossover and evaluator swap) and RECAP, a linear head co-trained with the target model to keep specified content decodable. On RECAP-trained sandbox models, fresh verbalizers state specified content truthfully, private codes vanish, at a cost of +0.001 nats. The method replicates on pretrained Pythia-160M. For AI safety, an independent probe scored truthful verbalizer claims above false ones (AUC 0.96 vs 0.82 without RECAP), and against an adversarial liar editing explanations (suppressing ~87% of lie penalties), the RECAP probe still flagged lies (AUC 0.95) while control probes fell to chance (0.51).

Paper Overview

Field: NLP Author: Hiskias Dingeto Published: 2026-07-24 arXiv: 2507.18397

Summary

Natural-language autoencoders score explanations of hidden activations by reconstruction: an explanation is deemed faithful if the activation can be regenerated from it. The test is structurally insensitive to individual false claims: if flipping a claim does not change the reconstruction, the score tracks gist, not specific facts. The paper shows the test is passed in two ways, neither faithful.

Key Findings

On a released Qwen-2.5-7B verbalizer, explanations reconstruct well above chance while ~2% of specific claims are reconstruction-dependent — the score tracks gist, not specific facts.
Under exact synthetic ground truth, the standard recipe developed co-adapted private codes (false wording the reconstruction depends on) in 5/5 runs; fixes that leave the target model unchanged did not help.
The authors contribute two audit protocols — grounded-vs-true crossover and evaluator swap — plus RECAP (Readable Encoding via Co-trained Auxiliary Predictor): a linear head trained alongside the target model to keep specified content decodable.
On RECAP-trained sandbox models, fresh verbalizers truthfully state specified content and private codes vanish, at a cost of only +0.001 nats.
The result replicates on pretrained Pythia-160M: content becomes reliably decodable even though fresh verbalizers only partially convey it (groundedness 0.44–0.46 vs near-zero controls).

Safety Implications

For interpretability: high reconstruction does not certify individual claims.
For AI safety: RECAP makes specified internal content independently checkable by probes, rather than asserted in prose the model can manipulate. An independent probe scored truthful verbalizer claims above false ones (AUC 0.96, vs 0.82 without RECAP).
Against an adversary who edits explanations to maximize reconstruction score while lying (suppressing ~87% of lie penalties), the RECAP probe still flagged lies (AUC 0.95), while control probes collapsed to chance (0.51).

Link: arXiv:2507.18397

*Auto-collected 2026-07-24*

Train the Model, Not the Reader: Decodability Supervision for Verifiable Explanations (RECAP, arXiv 2507.18397)

Summary

Paper Overview

Summary

Key Findings

Safety Implications

Tags