Disentangling Representation Evolution in Transformers through Parallel-Perpendicular Decomposition

Forum topic · 小凯 · 2026-09-16

Summary

This arXiv paper (2609.15975) by Shwai He, Haichao Zhang, and Shen Yan studies how Transformer representations evolve as a functional geometry, decomposing learned additive updates into parallel and perpendicular components relative to the current hidden state. Across pretrained models, the authors find substantial parallel components beyond the residual identity path, applied both to attention/MLP updates and to attention value aggregation relative to the current token's value. Targeted edits reveal a strong space-dependent asymmetry: exclude-self value-space parallel manipulation is markedly more robust than residual-space and perpendicular alternatives, preserving the direct self message while scaling only non-self aggregates. The same decomposition provides component-level descriptions of compression-induced update errors, with perpendicular errors distinguishing compression methods more sharply than parallel errors. Pretraining experiments show that suppressing aggregate-parallel components from scratch lowers validation loss and improves downstream averages, with the value-space variant strongest. Code is available in the project repository.

Paper Overview

Field: NLP Authors: Shwai He, Haichao Zhang, Shen Yan Published: 2026-09-14 arXiv: 2609.15975

Abstract

Transformer representations evolve through learned additive transformations that either preserve their current direction or redirect it. This work studies this evolution as a functional geometry, decomposing learned updates into parallel and perpendicular components.

Key Findings

Across pretrained models, there are substantial parallel components beyond the residual identity path.
The decomposition is applied in two spaces: to attention and MLP updates relative to the hidden state, and to attention value aggregation relative to the current token's value.
Targeted edits reveal a strongly space-dependent asymmetry: exclude-self value-space parallel manipulation is markedly more robust than residual-space and perpendicular counterparts, preserving the direct self message while scaling only the non-self aggregate.
The decomposition yields component-level descriptions of compression-induced update errors; perpendicular errors distinguish different compression methods more sharply than parallel errors.
Extensive experiments show that suppressing aggregate-parallel components during pretraining from scratch lowers validation loss trajectories and improves downstream average performance, with the value-space variant being most effective.

Implications

These results connect representation geometry to editing robustness, compression diagnostics, and training-time intervention. Code is available in the project repository.

--- *Auto-collected on 2026-09-16*