Encoding Probe: Reconstructing Language Model Representations as a New Interpretability Paradigm

Forum topic · 小凯 · 2026-05-04

Summary

A forum post introduces the paper 'Beyond Decodability: Reconstructing Language Model Representations with an Encoding Probe' (Gaofei Shen, Martijn Bentum, Tom Lentz, Afra Alishahi, Grzegorz Chrupała, arXiv 2605.00607). Traditional probing trains decoders to predict linguistic features from model representations, but accuracy scores cannot fairly compare features of different difficulty, and correlated features (e.g., noun and number) can confound results. The proposed Encoding Probe reverses the direction: instead of decoding features from representations, it trains an encoder that reconstructs model representations from interpretable features, minimizing reconstruction error. This enables fair comparison of feature importance across acoustic, morphological, syntactic, and semantic feature sets, while reducing confounds from feature correlation. Experiments on text and speech Transformers reveal a hierarchical encoding of features—low-level (phonetic/character), middle (syntactic), and high-level (semantic)—that is consistent across modalities, plus evidence of distributed representations requiring multiple interacting features. The post frames this as bidirectional validation of interpretability claims, echoing Feynman-style reasoning about understanding from both directions.

This post introduces the paper "Beyond Decodability: Reconstructing Language Model Representations with an Encoding Probe" by Gaofei Shen, Martijn Bentum, Tom Lentz, Afra Alishahi, and Grzegorz Chrupała (arXiv: 2605.00607, 2026-04-30).

The Limitation of Traditional Decoding Probes

Standard probing asks: does a model's internal representation encode, say, part-of-speech information? A classifier is trained to predict "noun vs. non-noun" from representations, and high accuracy is taken as evidence of encoding. But this has problems:

Features cannot be fairly compared. A noun probe at 90% accuracy vs. a verb probe at 85% does not mean nouns are more important—features differ in intrinsic difficulty.
Correlated features confound results. A probe for "noun" may cheat by exploiting the correlated "number" feature, making conclusions unreliable.

In short: decoding goes from representation → feature, but we need the reverse direction.

Encoding Probe: Reversing the Direction

The core idea: instead of predicting features from representations (decoding), reconstruct representations from features (encoding).

An encoder maps interpretable features into the model's representation space, minimizing reconstruction error. Benefits:

1. Fair feature comparison — reconstruction error directly reflects each feature's contribution. 2. Robustness to correlation confounds — reconstruction requires features to work together; no single correlated feature can cheat. 3. Richer feature sets — the paper uses features spanning acoustics (phonemes, prosody), morphology (affixes, POS), syntax (dependencies), and semantics (meaning, reference).

Analogy from the post: instead of finding evidence from a suspect (decoding), you reconstruct the crime scene from the evidence (encoding).

Key Findings

Experiments on text and speech Transformers show:

Hierarchical feature encoding: lower layers encode low-level features (phonemes, characters), middle layers encode syntax (POS, dependencies), higher layers encode semantics (meaning, coreference).
Cross-modal commonality: text and speech models share a similar hierarchy, suggesting this is a general property of sequence processing rather than modality-specific.
Feature interaction: no single feature suffices to reconstruct representations; information is distributed across multiple interacting features.

Takeaway

Understanding neural representations should be validated bidirectionally: reading features out of representations (decode) and rebuilding representations from features (encode). The post suggests researchers ask: Does my probe only decode? Can I fairly compare features? Are correlated features contaminating my conclusions? Would bidirectional validation strengthen them?