Learning Length-Extrapolatable Recurrent Models via Credit Stabilization through Time (CST)

Forum topic · 小凯 · 2026-09-10

Summary

This arXiv paper (2609.09157) by Hanwen Jiang addresses why recurrent models trained with backpropagation through time (BPTT) often fail beyond their training horizon in long-context modeling. Rather than attributing failure solely to vanishing or exploding gradients, the author studies state credit: the signal through which future losses reach earlier recurrent states before contributing to parameter updates. The proposed method, Credit Stabilization through Time (CST), locally rescales the state-credit signal during backward propagation to stabilize its norm without rotating the corrected component, leaving forward computation unchanged. Because controlled synthetic tasks and real data exhibit different credit dynamics, CST is specialized for each regime. Experiments show improved performance beyond the training horizon, with gains observed at up to 128x the training length, enabling length-extrapolatable recurrent models.

Paper Overview

Research area: cs.LG, cs.CL Author: Hanwen Jiang Published: 2026-09-08 arXiv: 2609.09157

Abstract

Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their training horizon. Classical analyses emphasize gradients that vanish or explode along temporal paths. However, dense per-token losses can still train a shared recurrent rule despite severe decay, showing that decay alone does not determine whether learning fails.

The paper instead studies state credit: the signal through which future losses reach earlier recurrent states before contributing to parameter updates. Intervening directly on state credit, the author proposes Credit Stabilization through Time (CST).

Method

During backward propagation, CST locally rescales the state-credit signal to stabilize its norm.
It does so without rotating the component being corrected, and leaves the forward computation unchanged.
Because controlled synthetic tasks and real data exhibit different credit dynamics, CST is specialized to each regime.

Results

CST improves performance beyond the training horizon in both settings.
Gains are observed at up to 128x the training length.

Original Abstract

> Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their training horizon. Classical analyses emphasize gradients that vanish or explode along temporal paths. However, dense per-token losses can still train a shared recurrent rule despite severe decay, showing that decay alone does not determine whether learning fails. We instead study state credit: the signal through which future losses reach earlier recurrent states before contributing to parameter updates. Accordingly, we intervene directly on state credit and propose Credit Stabilization through Time (CST). During backward propagation, CST locally rescales the state-credit signal to stabilize its norm without rotating the component being corrected, while leaving the forward computation unchanged. Because controlled synthetic tasks and real data exhibit different credit dynamics, we specialize CST to each regime. In both settings, CST improves performance beyond the training horizon, with gains observed at up to 128x the training length.

*Auto-collected on 2026-09-10*