Disentangling Representation Evolution in Transformers through Parallel-Perpendicular Decomposition

Forum topic · 小凯 · 2026-09-16

Summary

This paper studies how Transformer representations evolve by decomposing learned additive updates into parallel and perpendicular components relative to existing directions—a functional-geometry view of representation dynamics. Across pretrained models, the authors find substantial parallel components beyond the residual identity path. The decomposition is applied in two spaces: attention/MLP updates relative to the hidden state, and attention value aggregation relative to the current token's value. Targeted edits reveal a strong space-dependent asymmetry: exclude-self value-space parallel manipulation is markedly more robust than residual-space and perpendicular counterparts, preserving the direct self message while scaling only the non-self aggregate. The same decomposition yields a component-level account of compression-induced update errors, with perpendicular errors distinguishing compression methods more sharply than parallel errors. Experiments also show that suppressing aggregate parallel components during pretraining from scratch lowers validation loss and improves downstream average performance, with the value-space variant strongest. Code is available in the project repository.

Paper Overview

Field: NLP Authors: Shwai He, Haichao Zhang, Shen Yan Published: 2026-09-14 arXiv: 2609.15975

Abstract

Transformer representations evolve through learned additive transformations that either preserve their current direction or redirect it. We study this evolution as a functional geometry, decomposing learned updates into parallel and perpendicular components. Across pretrained models, we find substantial parallel components beyond the residual identity path. We then apply the decomposition in two spaces: to attention and MLP updates relative to the hidden state, and to attention value aggregation relative to the current token's value. Targeted edits reveal a strongly space-dependent asymmetry: exclude-self value-space parallel manipulation is markedly more robust than residual-space and perpendicular counterparts, preserving the direct self message while scaling only the non-self aggregate.

The same decomposition also yields a component-level characterization of compression-induced update errors: perpendicular errors distinguish different compression methods more sharply than parallel errors. Extensive experiments further show that suppressing aggregate parallel components during pretraining from scratch lowers validation loss trajectories and improves downstream average performance, with the value-space variant being the most effective. These results connect representation geometry to editing robustness, compression diagnostics, and training-time interventions. Code is available in the project repository.

Key Takeaways

Learned Transformer updates contain substantial parallel components beyond the residual identity path.
Exclude-self value-space parallel edits are notably more robust than residual-space or perpendicular edits.
Perpendicular error components best differentiate compression methods.
Parallel suppression during pretraining improves validation loss and downstream performance.

--- *Auto-collected on 2026-09-16*