Encoded but Not Decoded: A Three-Level Gap in LLM Syntactic Ability
When asked to fill the blank in "The key to the cabinets __ on the table," a large language model answers "is" correctly. But is this because it truly represents that the subject is "key" rather than "cabinets," or because it has learned surface-level co-occurrence statistics? Behavioral tests cannot distinguish these cases — and the two failures require different fixes: better training data versus better alignment/deployment strategies.
Zhenyan Lu, He Wang, and Xiaohui Huang's paper "Encoded but Not Decoded: Layer-Localized Evidence for a Three-Level Gap in LLM Syntax" proposes a three-level evaluation framework that separates these failure modes.
The Three-Level Framework
Like assessing a student via exam scores, verbal explanations, and brain scans, the paper evaluates models at three levels, from outer to inner:
- Behavioral deployment: actual downstream task performance (e.g., grammaticality judgment accuracy)
- LM-head readout: feeding intermediate hidden states directly to the LM head, bypassing the full decoding path
- Probe recoverability: training a linear probe on intermediate representations to recover syntactic structure
- A systematic three-level gap exists: probe recoverability > LM-head readout > behavioral deployment across model families (Qwen3, Llama, Gemma).
- The 0.653 gap: On a subject-control task, Qwen3-0.6B Instruct achieves probe accuracy of 0.903 but behavioral accuracy of only 0.250 — a gap of 0.653. 90.3% of syntactic information is recoverable, yet only 25.0% is deployed.
- The gap is layer-localized: On Qwen3-14B Instruct it concentrates in mid-to-upper layers; on Gemma-4 E4B, in earlier layers. Activation patching experiments confirm this — patching specific layers changes outputs while patching others does not — ruling out the simple explanation that the LM head merely cannot read uniformly available information.
- Instruction tuning damages deployment, not encoding: Comparing Qwen3-0.6B Base vs Instruct, probe accuracy stays nearly unchanged while behavioral accuracy drops significantly. Tuning alters the "gate" (how information is used), not the "judgment" (what is encoded).
- 14B as an emergence threshold: At Qwen3-14B scale, probes recover syntax equally well regardless of the verb ("is" vs "was") or number — verb-invariant, more abstract syntactic encoding emerges. At 0.6B and 3B, probe accuracy still depends on the specific verb.
Key Findings
Why It Matters
1. Methodology: The framework distinguishes "doesn't know" from "knows but cannot use" — showing the latter is the norm, not the exception. 2. Alignment strategy: If syntactic failures are deployment failures, alignment should focus on helping models access existing information rather than learning more — consistent with lightweight alignment trends. 3. Scale and abstraction: The 14B threshold suggests scale changes not just capability magnitude but capability organization: small models spread syntax across verb-specific subspaces, while large models integrate it into a unified space.
Limitations
The framework is validated mainly on syntax tasks; extension to semantics and reasoning remains open. Linear probes may underestimate stored information, and "probe recoverable" does not strictly mean "used by the model's actual computation" — recovered information could be a byproduct.
Still, as a diagnostic tool, the three-level framework makes one thing sharp: a wrong answer does not mean the model doesn't know, and a right answer doesn't mean it does.
---
Paper: https://arxiv.org/abs/2609.29848
HTML full text: https://arxiv.org/html/2609.29848v1