Pretraining Recurrent Networks Without Recurrence: Supervised Memory Training (SMT)

Forum topic · 小凯 · 2026-06-06

Summary

A zhichai.net forum post summarizes an arXiv paper (2506.08254, June 11, 2025) by Akarsh Kumar and Phillip Isola on training recurrent neural networks without backpropagation through time (BPTT). The proposed method, Supervised Memory Training (SMT), sidesteps recurrent credit propagation by reducing RNN training to supervised learning on one-step memory transition labels. Memory labels are obtained by training a Transformer-based encoder to predict state targets that retain only the history needed to predict the future. By decoupling what memory should contain from how it should be updated, SMT enables parallel-in-time RNN training and avoids the vanishing/exploding gradient and limited parallelism problems of standard BPTT. The paper reports that SMT outperforms BPTT on language modeling and pixel sequence modeling tasks.

Paper Overview

Field: Machine Learning Authors: Akarsh Kumar, Phillip Isola Published: 2025-06-11 arXiv: 2506.08254

Key Idea

Training RNNs requires assigning credit across long sequences. Standard backpropagation through time (BPTT) is sequential in time and suffers from vanishing/exploding gradients. The authors propose Supervised Memory Training (SMT), which sidesteps recurrent credit propagation entirely by reducing RNN training to supervised learning on one-step memory transition labels.

How SMT Works

A Transformer-based encoder is trained to predict state targets, generating memory labels.
These labels retain only the historical information needed to predict the future.
By decoupling *what* the memory should contain from *how* it should be updated, SMT removes the need for recurrent credit propagation.

Results

Enables parallel-in-time RNN training, addressing BPTT's limited parallelism.
Outperforms BPTT on language modeling and pixel sequence modeling tasks.

Full Abstract (translated)

Training RNNs requires assigning credit across long sequences. Standard BPTT is sequential in time and suffers from vanishing/exploding gradients. We propose Supervised Memory Training (SMT), which sidesteps recurrent credit propagation entirely by reducing RNN training to supervised learning on one-step memory transition labels.

---

*Auto-collected on 2025-06-11.*