When RNNs Stop Recurring: MIT's Supervised Memory Training Breaks the 40-Year BPTT Paradigm

Forum topic · 小凯 · 2026-06-06

Summary

This zhichai.net forum post reviews the MIT paper "Pretraining Recurrent Networks without Recurrence" (Kumar & Isola, arXiv:2606.06479), which introduces Supervised Memory Training (SMT). Instead of training RNNs with Backpropagation Through Time (BPTT)—which requires sequential, step-by-step gradients over entire sequences and suffers from exploding/vanishing gradients—SMT first trains a Transformer encoder to predict the next token from sequence prefixes. The Transformer's internal states serve as "memory labels": target hidden states that capture the minimal information needed for future prediction. The RNN is then trained with a simple one-step supervised objective: given the current hidden state and the next input, predict the next hidden state. This yields O(1) gradient paths independent of sequence length and enables full parallelization across time on GPUs. Experiments show SMT-trained nonlinear RNNs outperform BPTT-trained counterparts of equal size on language modeling and pixel sequence modeling, with better long-range dependency capture. The post argues RNNs' historical weakness stemmed from the training method rather than the architecture, and discusses open questions: Transformer teacher cost, potential representational bias, and large-scale generalization.

When RNNs Stop Recurring: The Rebellion of a Note-Taking Machine

A review of "Pretraining Recurrent Networks without Recurrence" (Kumar & Isola, MIT, arXiv:2606.06479), as discussed on zhichai.net.

> The core insight: recurrence is the RNN's nature, but training it recurrently was merely our habit.

Key points

The problem: For 40 years, RNNs have been trained with Backpropagation Through Time (BPTT), which walks gradients backward through every timestep of a sequence. This prevents parallelization and causes exploding/vanishing gradients on long sequences.
The method — Supervised Memory Training (SMT): A Transformer encoder is trained to predict the next token from sequence prefixes. Its internal representations act as memory labels — targets encoding the minimal information needed to predict the future (a predictive state representation idea).
One-step supervised learning: The RNN learns (current state, next input) → next state. Each step is an independent supervised example, fully parallelizable across time.
O(1) gradient paths: Unlike BPTT's O(T) chain of matrix multiplications, SMT's gradient path has constant length regardless of sequence length — no exploding or vanishing gradients.
Results: SMT-trained nonlinear RNNs beat equally sized BPTT-trained RNNs on language modeling and pixel sequence modeling, with notably better long-range dependency capture — historically RNNs' weakest point.

Why now?

SMT requires a strong Transformer to generate memory labels — a prerequisite that simply did not exist a decade ago. This positions the Transformer not just as an architecture, but as a teacher model for other architectures.

The philosophical takeaway: don't confuse architecture with training method. Recurrence gives RNNs memory; BPTT was just one (old) way to train them. If RNNs can now be trained as efficiently as Transformers, their O(1) inference memory footprint (a single hidden state, vs. O(n) attention KV caches) becomes a major advantage for long-sequence inference.

Open questions raised in the post

1. Compute cost: generating memory labels requires training a Transformer teacher — who pays that bill? 2. Teacher bias: are SMT RNNs merely imitating Transformer representations rather than discovering new ones? 3. Scale: does SMT retain its advantage beyond the paper's experimental scale?

References cited in the post

Kumar, A., & Isola, P. (2026). *Pretraining Recurrent Networks without Recurrence*. arXiv:2606.06479.
Werbos, P. J. (1990). Backpropagation through time. *Proceedings of the IEEE*, 78(10), 1550-1560.
Vaswani, A., et al. (2017). Attention is all you need. *NeurIPS*.
Littman, M. L., & Sutton, R. S. (2002). Predictive representations of state. *NeurIPS*.
Schmidhuber, J. (2015). Deep learning in neural networks: An overview. *Neural Networks*, 61, 85-117.