Why Deeper LLM Layers Are Not Always Better: Confident Layer Decoding to Reduce the Alignment Tax

Key points

• The alignment tax, restated. RLHF and DPO push the final layer toward safe, verbose, generic outputs. On reasoning tasks that require a single correct answer, this bias hurts accuracy: the model substitutes precise tokens with hedged, helpful-sounding phrases.
• A three-stage forward-pass dynamic. Early layers establish rough semantic associations (Guess). Middle layers refine a confident reasoning trajectory (Refine). The final layer then perturbs that trajectory to satisfy alignment objectives (Perturb), manifesting as an entropy rebound on the last layer's logits.
• Where the paper's evidence comes from. Layer-wise entropy curves show a steady decline through middle layers followed by a sharp rise on the final layer, and the effect is stronger on heavily aligned, larger models than on base models. The authors call this the *Planning-Pragmatics Trade-off*: middle layers plan, the final layer enforces pragmatics.
• Confident Layer Decoding (CLD). Instead of trusting the last layer, CLD scans a small window near the output (default 5) from back to front and picks the first local entropy minimum. The intuition: low entropy means high confidence, and the earliest such valley before the final layer is where refinement is complete but alignment perturbation has not yet kicked in.
• Algorithmic shape. During generate, request output_hidden_states=True, project each layer's hidden state through lm_head, compute Shannon entropy on the last-token logits, and select the first valley in the window. Fall back to the final layer if no valley is found. The added cost is dominated by a handful of lm_head projections and entropy calculations, reported as under 2% latency overhead.
• Theoretical framing. The authors cast CLD as an optimal-stopping problem. Under bounded per-layer projection noise and a dominant alignment perturbation at the final layer, the backward search rule is shown to bound the loss relative to an oracle refinement layer.
• Empirical results. Reported headline gains over greedy decoding include +3.7% on GPQA-Diamond for Qwen2.5-72B, +3.6% on Omni-MATH for Qwen2.5-72B, +2.4% on HLE for Qwen2.5-72B, +2.5% on MATH for Llama-3-70B, and +3.9% on GPQA-Diamond for Qwen3-235B-A22B. CLD also beats contrastive decoding baselines, with the largest margins on the hardest benchmarks.
• Scaling behavior. Gains grow monotonically with model size: roughly +1.2% at 7B, +2.1% at 14B, +3.0% at 32B, +3.7% at 72B, and +3.9% at 235B (MoE). The interpretation is that larger models have more capacity to learn alignment shortcuts, which in turn makes the final-layer perturbation more harmful.
• Architecture generality. Because the dynamic shows up in both dense Transformers and Qwen3-235B-A22B's MoE variant, the authors argue it is a property of Transformer forward passes rather than of any specific architecture.
• Where CLD helps and where it does not. It helps on math, code, and science reasoning, on strongly aligned models, at low sampling temperatures, and at larger scales. It is less useful for open-ended creative writing, safety-critical advice, very small models, and lightly aligned base models, where the final layer either helps or has little perturbation to bypass.
• Engineering integration. The paper does not ship an official implementation, but the algorithm fits cleanly into HuggingFace Transformers (use output_hidden_states=True and override the sampling step), vLLM/TGI (extend the sampler to choose among lm_head projections), or agent frameworks such as LangChain and LlamaIndex through a custom decoding strategy. Recommended starting parameters: window_size=5, Shannon entropy on the last-token logits, fallback to the final layer when no local minimum exists.
• Implications for alignment design. The result challenges the assumption that deeper means better and suggests three architectural responses: layered alignment (light or no alignment in early/middle layers, heavy alignment only at the output when appropriate), multi-head decoding with separate heads for reasoning and open-ended dialogue, and decoding-time safeguards that compensate for alignment-induced entropy rebound instead of retraining.
• A reframing of emergent abilities. The authors speculate that some emergent capabilities are not newly produced by scale but are long-standing middle-layer skills that alignment training has been suppressing; CLD exposes them by routing around the final-layer perturbation, which would explain why gains are largest for the biggest models.

References

• arXiv:2606.21906 — Zhang, Zhoubian, Chen, et al., *Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decoding*.
• Ouyang et al., 2022, *Training language models to follow instructions with human feedback*, NeurIPS.
• Rafailov et al., 2023, *Direct Preference Optimization: Your Language Model is Secretly a Reward Model*, NeurIPS.