Reusing the Same Parameter Block in SSMs: Depth Recurrence Beats Fresh Parameters

Forum topic · 小凯 · 2026-05-18

Summary

A paper by Farsang, Hasani, Rus, and Grosu (MIT CSAIL and TU Wien) explores depth recurrence in state space models (SSMs). Instead of stacking L layers with independent parameters, the proposed approach uses a single SSM block with k parameters and loops it L times along the depth dimension. Although the looped model's hypothesis space is strictly smaller, experiments across four SSM architectures (LRU, S5, LinOSS, LrcSSM) and six time series classification benchmarks show it consistently matches or outperforms independent-parameter models. The authors attribute the gain to a beneficial inductive bias from parameter sharing that simplifies optimization, rather than expressivity. The paper also introduces input reshaping techniques—concatenating timesteps for low-dimensional inputs and re-chunking flattened feature-time dimensions for high-dimensional inputs—which yield 1-6% accuracy gains and stack with depth recurrence. Open questions include choosing the optimal loop count L and generalizing the approach to non-time-series tasks.

State space models (SSMs) have become strong competitors to Transformers, especially on long-sequence tasks. Their core operation is recurrence along the sequence dimension—each timestep's state depends on the previous one. But in this paper, Farsang, Hasani, Rus, and Grosu (MIT CSAIL + TU Wien) explore a different, often overlooked dimension of recurrence in SSMs: recurrence along the depth axis.

The idea

The standard approach stacks L SSM blocks, each with its own parameters. The new approach uses just a single SSM block (with k parameters) and loops it L times along the depth dimension. In other words, the same parameter block is reused L times.

Theory vs. practice

The theoretical result: the looped model's hypothesis space is strictly smaller than that of the parameter-independent model (the latter contains the former as a special case). Yet the experimental result is the opposite: across 4 SSM architectures (LRU, S5, LinOSS, LrcSSM) and 6 time series classification benchmarks, the looped model consistently matches or exceeds the parameter-independent one. Since the larger model contains the smaller one, this advantage cannot come from expressivity. The conclusion is that parameter sharing forms a beneficial inductive bias along the depth dimension, simplifying optimization.

Input reshaping

The paper also digs into another neglected design axis: input reshaping.

For low-dimensional inputs, concatenate consecutive timesteps to increase per-step information density.
For high-dimensional inputs, flatten the joint feature-time dimension and re-chunk it.

Both methods yield 1–6% accuracy improvements across all models. Depth recurrence and input reshaping are independent improvements, and their benefits stack when combined.

Open questions

How to determine the optimal number of loops L—does it relate to the timescale of the task?
The specific input reshaping strategies (concatenation scheme, chunk size) depend on dataset characteristics. Is there a general heuristic?
Does depth recurrence still help on non-time-series tasks such as natural language processing?

References

1. Farsang, M., Hasani, R., Rus, D., & Grosu, R. (2026). *Looped SSMs: Depth-Recurrence and Input Reshaping for Time Series Classification*. arXiv:2605.16048 [cs.LG]. 2. Gu, A., & Dao, T. (2023). *Mamba: Linear-Time Sequence Modeling with Selective State Spaces*. NeurIPS. 3. Lan, Z., et al. (2020). *ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations*. ICLR.