Encoding Probe: From Decoding to Reconstructing—A New Paradigm for Understanding LLM Internal Representations

Forum topic · 小凯 · 2026-05-04

Summary

A Chinese tech forum post introduces the Encoding Probe, a method from the paper "Beyond Decodability: Reconstructing Language Model Representations with an Encoding Probe" (arXiv: 2605.00607) by Gaofei Shen, Martijn Bentum, Tom Lentz, Afra Alishahi, and Grzegorz Chrupała. Traditional probing trains classifiers to decode features (e.g., part of speech) from model representations, but decoder accuracy cannot fairly compare features of different difficulty and suffers from correlations between features (e.g., nouns and number). The Encoding Probe reverses the direction: instead of predicting features from representations, it trains an encoder that reconstructs representations from interpretable features, minimizing reconstruction error. This makes feature contributions directly comparable, prevents single-feature cheating, and supports rich feature sets spanning acoustic, morphological, syntactic, and semantic levels. Experiments on text and speech Transformers reveal a hierarchical encoding: low layers encode phonetic/character-level features, middle layers encode syntax, and higher layers encode semantics—with similar structure across modalities, and distributed information requiring multiple interacting features. The post argues that bidirectional verification (decoding plus encoding) yields more reliable interpretability conclusions, offering practical questions for probing research.

Paper

Beyond Decodability: Reconstructing Language Model Representations with an Encoding Probe

Authors: Gaofei Shen, Martijn Bentum, Tom Lentz, Afra Alishahi, Grzegorz Chrupała
arXiv: 2605.00607 | 2026-04-30

The Problem: Probes That Can Only Decode, Not Encode

When studying LLM internal representations, the traditional approach uses a decoding probe:

Ask: "Does this representation encode 'noun' information?"
Method: train a classifier to predict "noun / not-noun" from the representation
High accuracy → "the representation encodes noun information"

But this has serious limitations:

1. Cannot compare different features fairly

A "noun" probe scores 90%, a "verb" probe 85%—does that mean nouns matter more? No: different features have different intrinsic difficulty.

Correlated features interfere

"Noun" and "number" are highly correlated; a probe may cheat by using "number" to predict "noun," making results unreliable.

Traditional probes only go from representation → feature. What we need is the reverse direction—feature → representation.

Encoding Probe: Reversing the Direction

Core idea: Instead of predicting features from representations (decoding), reconstruct representations from features (encoding).

Why reconstruction is better:

1. Directly compares feature contributions — reconstruction error reflects how important a feature is, enabling fair comparison across features. 2. Avoids correlation interference — reconstruction requires all features to work together; a single correlated feature cannot cheat. 3. Supports rich feature sets — the paper uses features spanning multiple domains:

Acoustic features (phonemes, prosody)
Morphological features (affixes, part of speech)
Syntactic features (dependencies)
Semantic features (word meaning, coreference)

Technical approach: train an encoder that maps interpretable features into the model's representation space, minimizing reconstruction error, then analyze which features matter most.

*It's like detective work: rather than finding evidence from a suspect (decoding), reconstruct the crime scene from the evidence (encoding)—the latter reveals more truth.*

Key Findings: Hierarchical Structure of Representations

Experiments on text and speech Transformers reveal:

1. Hierarchical feature encoding

Lower layers: low-level features (phonemes, characters)
Middle layers: syntactic features (POS, dependencies)
Higher layers: semantic features (meaning, coreference)

2. Cross-modal commonality — text and speech models show similar hierarchical structure, suggesting this hierarchy is a general property of sequence processing, not modality-specific. 3. Feature interaction — no single feature suffices to reconstruct full representations; information is distributed across multiple features.

Bidirectional Verification

> "If you can only understand something from one direction, your understanding is one-sided. True understanding means verifying from multiple directions." — Feynman-style reasoning

In representation learning:

> Traditional probes only test representation → feature. Encoding Probe adds the feature → representation direction. Bidirectional verification makes conclusions more reliable.

This mirrors the scientific method: if hypothesis A explains B, test both A→B and B→A; only when both hold is understanding genuine.

Takeaways

If you work on neural network interpretability, ask yourself:

1. Does my probe only decode, never encode? 2. Can I fairly compare the importance of different features? 3. Do feature correlations contaminate my conclusions? 4. Would bidirectional verification strengthen my findings?

Encoding Probe reminds us: understanding neural representations cannot rely on a single direction. Reading features out of representations *and* reconstructing representations from features—together—form the reliable path forward.