Looped SSMs: How One Layer Looped 10 Times Beats 10 Stacked Layers
| Property | Details | | :--- | :--- | | Paper | Looped SSMs: Depth-Recurrence and Input Reshaping for Time Series Classification | | Authors | Mónika Farsang, Ramin Hasani, Daniela Rus, Radu Grosu et al. (MIT and collaborators) | | arXiv ID | 2605.16048 (May 2026) | | Field | Model architecture design, State Space Models (SSMs), time series analysis | | Keywords | Depth-Recurrence, inductive bias, weight sharing, input reshaping |
The Stacking Dilemma of SSMs
State space models (SSMs) such as Mamba are considered promising successors to Transformers, thanks to their memory efficiency on long sequences and time series. Traditionally, making SSMs smarter means building taller towers: stacking 10, 50, or even 100 distinct SSM modules. If each layer has 100M parameters, 10 layers means 1B parameters.
But many independent layers with separate weights are hard to coordinate during training, prone to vanishing gradients and optimization instability.
The Looped SSM Trick: Depth-Recurrence
The researchers performed a bold operation: cut a 10-layer model down to a single layer, then run the data through that same layer 10 times in a loop.
The result: a looped model with only \(k\) parameters, looped 10 times, matches or even surpasses the 10k-parameter stacked counterpart.
Why Repetition Makes the Model Stronger
Mathematically, this is counterintuitive. The paper proves that the 10-layer model's function space actually *contains* the 1-layer looped model's—the conventional model has a strictly larger expressiveness upper bound. So why does the more constrained model perform better?
The answer is inductive bias. Searching a vast parameter space is like finding a needle in the ocean: too many parameters easily get lost and learn spurious shortcuts (overfitting). Weight sharing acts as a strict constraint that forces the single layer to capture the most general, core patterns in the data. Simple constraints produce elegant strength.
Free Performance: Input Reshaping
As a bonus, the team found that repackaging time series inputs—e.g., merging fragmented data from the past 3 seconds into richer chunks—boosts accuracy by 1–6% regardless of the model used, at zero computational cost.
Caveats and Open Questions
1. Wall-clock time trade-off: Fewer parameters save memory, but looping 10 times is serial computation. The paper emphasizes memory savings but says relatively little about inference latency in real deployments. 2. Task boundary: Experiments focus on time series classification. Whether forced weight sharing hurts diversity in long-form generation or multimodal video prediction remains untested.
Takeaway
Good algorithms are like Occam's razor, trimming away excess complexity. Looped SSMs shows that on the road to capable models, endlessly piling up GPUs and parameters may not be necessary—repeated constraint can sometimes approximate truth better than unconstrained freedom. Less is more.
Source: arXiv:2605.16048