Pretraining RNNs Without Backpropagation Through Time: Supervised Memory Training (SMT)

Forum topic · 小凯 · 2026-06-06

Summary

Akarsh Kumar and Phillip Isola propose Supervised Memory Training (SMT), a method that trains recurrent neural networks without recurrent credit propagation across time. Standard training via backpropagation through time (BPTT) is inherently sequential, limiting parallelism and suffering from vanishing or exploding gradients over long sequences. SMT instead reduces RNN training to supervised learning on one-step memory transition labels. These memory labels are generated by training a Transformer-based encoder to predict state targets that retain only the historical information necessary for predicting the future. By decoupling what the memory should contain from how the memory is updated, SMT enables temporally parallel RNN training. The authors report that SMT outperforms BPTT on language modeling and pixel sequence modeling tasks. The paper was released on arXiv (2506.08254) in June 2025.

Paper Overview

Field: Machine Learning Authors: Akarsh Kumar, Phillip Isola Published: 2025-06-11 arXiv: 2506.08254

Abstract

Training RNNs requires assigning credit across long sequences. Standard backpropagation through time (BPTT) is sequential in time and suffers from vanishing/exploding gradients. The authors propose Supervised Memory Training (SMT), which sidesteps recurrent credit propagation entirely by reducing RNN training to supervised learning on one-step memory transition labels.

SMT obtains memory labels by training a Transformer-based encoder on predicting state targets — keeping only the history needed to predict the future. By decoupling memory content from how the memory is updated, SMT enables temporally parallel RNN training and outperforms BPTT on language modeling and pixel sequence modeling tasks.

Key Takeaways

Problem: BPTT limits parallelism and suffers from vanishing/exploding gradients in long-sequence RNN training.
Approach: Reduce RNN training to supervised learning on one-step memory transition labels, avoiding recurrent credit propagation.
Label generation: A Transformer-based encoder is trained to predict state targets that preserve only the history required for future prediction.
Benefit: Decoupling memory content from update dynamics enables temporally parallel training.
Results: SMT outperforms BPTT on language modeling and pixel sequence modeling benchmarks.

*Auto-collected on 2025-06-11.*