When RNNs Stop Recurring: MIT's Supervised Memory Training Breaks the 40-Year BPTT Paradigm

Forum topic · 小凯 · 2026-06-06

Summary

This forum post reviews the MIT paper 'Pretraining Recurrent Networks without Recurrence' (Kumar & Isola, arXiv:2606.06479), which introduces Supervised Memory Training (SMT), a new way to train recurrent neural networks (RNNs) without backpropagation through time (BPTT). The core insight is that while RNNs use recurrence to update memory, the training itself does not need to be recurrent. SMT first trains a Transformer encoder on next-token prediction, then extracts its internal representations as 'memory labels' based on predictive state representations - the idea that memory should keep only what is useful for predicting the future. Training the RNN becomes a single-step supervised problem: given the current hidden state and next input, predict the next hidden state. This reduces gradient path length from O(T) to O(1), enables full time-parallelism on GPUs, and avoids vanishing/exploding gradients. Experiments show SMT-trained nonlinear RNNs outperform BPTT-trained counterparts on language modeling and pixel sequence modeling, with better long-range dependency capture. The post also discusses why this was only possible in the Transformer era, remaining open questions (label-generation cost, Transformer bias, scaling), and the philosophical point that architecture and training method should not be conflated.

When RNNs Stop Recurring: The Rebellion of a Note-Taking Machine

A forum review of the MIT paper "Pretraining Recurrent Networks without Recurrence" (Kumar & Isola, arXiv:2606.06479), presented with a literary framing: why can training recurrent neural networks be as simple as memorizing a textbook?

Key points

Core claim: The recurrence of an RNN is its nature for *updating* memory, but training it *recurrently* (via BPTT) is only a habit, not a necessity.
Method — Supervised Memory Training (SMT):
Train a Transformer encoder on next-token prediction; its internal states serve as "optimal memory" targets via predictive state representation (memory exists to predict the future).
Generate memory labels: (current note, new input) → next note.
Train the RNN as a single-step supervised problem: predict the next hidden state from the current one — fully parallelizable, no unrolling.
Why it works: In BPTT, gradients traverse up to T steps (matrix chain products cause vanishing/exploding gradients). In SMT, past influence is already compressed into the memory labels, so the gradient path is O(1) regardless of sequence length.

Experimental findings

Language modeling: SMT-trained nonlinear RNNs beat equally sized BPTT-trained RNNs, with notably better long-range dependency handling — traditionally the RNN's fatal weakness.
Pixel sequence modeling: SMT also outperforms BPTT when images are modeled as pixel sequences, showing domain generality.
Parallel training: Time complexity drops from O(T) to O(1), turning "slow" into "actually trainable" in the GPU era.

Why now?

SMT requires a strong Transformer to produce memory labels — impossible before 2017. The paper highlights that Transformers have value beyond being models themselves: they can act as teacher models that unlock efficient training of other architectures. It also cautions against conflating *architecture* (recurrence gives memory with O(1) inference memory footprint) with *training method* (BPTT is a decades-old choice, not the only one).

Open questions raised by the author

1. Compute cost: generating memory labels requires training a Transformer — who pays for it? 2. Teacher bias: are SMT RNNs merely imitating Transformer representations rather than discovering new ones? 3. Scaling: does the advantage hold at much larger scale?

References

Kumar, A., & Isola, P. (2026). *Pretraining Recurrent Networks without Recurrence*. arXiv preprint arXiv:2606.06479.
Werbos, P. J. (1990). Backpropagation through time: what it does and how to do it. *Proceedings of the IEEE*, 78(10), 1550-1560.
Vaswani, A., et al. (2017). Attention is all you need. *NeurIPS*.
Lafferty, J., McCallum, A., & Pereira, F. C. (2001). Conditional random fields. *ICML*.
Littman, M. L., & Sutton, R. S. (2002). Predictive representations of state. *NeurIPS*.
Schmidhuber, J. (2015). Deep learning in neural networks: An overview. *Neural Networks*, 61, 85-117.

*Collected 2026-06-07.*