LIFT: Pretraining Latent Information Feedback Transformers with Teacher Supervision

Forum topic · 小凯 · 2026-10-01

Summary

LIFT (Latent Information Feedback Transformer) is a new architecture and training method by Dor Tirosh, Ido Amos, and Mor Geva that removes the feed-forward bottleneck in Transformer language models. Standard LMs can only propagate information across generation steps through decoded tokens, forcing recomputation of intermediate results. LIFT turns recurrent-state learning into a teacher-forced prediction problem: each input token is paired with an information-dense state derived from the next-token distribution of an off-the-shelf pretrained LM, and the model—extended with few extra parameters—is trained to predict both the next token and the next state. Because input states are precomputed, pretraining is fully parallel. At inference, the model feeds back its own predicted states, with overhead shrinking as model scale grows. Experiments on models from 135M to 1B parameters show LIFT consistently outperforms standard Transformers under token-matched budgets on language modeling, downstream reasoning, and procedural tasks, and matches or beats them under compute-matched comparisons. A controlled state-tracking study shows a tiny LIFT beats an equal-size Transformer trained on 8x more data, even when supervised by states from a failing Transformer.

Paper Overview

Field: NLP Authors: Dor Tirosh, Ido Amos, Mor Geva Published: 2026-09-29 arXiv: 2609.38149

Abstract (English)

Transformer language models (LMs) are feed-forward: deep-layer representations are never fed back to shallower layers, and the only pathway for information to flow downward across generation steps is the decoded token. This narrow channel forces models to recompute intermediate results and to discard alternative continuations.

This work removes that bottleneck during pretraining by introducing LIFT (Latent Information Feedback Transformer), an architecture and training method that enables LMs to propagate state across generation. The key idea is to turn recurrent-state learning into a teacher-forced prediction problem:

Each input token is paired with an information-dense state derived from the next-token distribution of an off-the-shelf pretrained LM.
The model, extended with a small number of additional parameters, is trained to predict both the next token and the next state.
Since input states are precomputed, pretraining is fully parallelizable across positions.
At inference time, the model's own predicted states are fed back; the extra computational overhead diminishes as model size grows.

Key Findings

Experiments on pretrained models from 135M to 1B parameters show LIFT consistently outperforms standard Transformers and baselines under token-matched budgets on language modeling, downstream reasoning tasks, and procedural tasks.
Under compute-matched comparisons, LIFT matches or exceeds standard Transformers.
A controlled study on a state-tracking task shows that even when trained with states from a Transformer that fails the task, a tiny LIFT outperforms an equal-size Transformer trained on 8x more data.

Conclusion

Language models can learn to exploit deep-to-shallow feedback during pretraining through scalable teacher supervision, opening a path beyond purely feed-forward generation.

--- *Auto-collected on 2026-10-01*