SIREN-RoPE: When Attention Discovers the Hidden Dimension — Learnable Temporal Rotary Encoding

Forum topic · 小凯 · 2026-04-28

Summary

SIREN-RoPE (arXiv:2604.24717, Cheng, Sun, Lu) upgrades Rotary Position Embedding (RoPE) by making its rotation angles learnable and conditioned on real timestamps. Standard RoPE encodes position as fixed rotations of Query and Key vectors, capturing only ordinal distance. SIREN-RoPE adds a dual-branch SIREN network (sinusoidal activations for periodic patterns, an MLP branch for aperiodic trends) that maps timestamp features into rotation angles, combined with the ordinal RoPE term through a learnable gated scalar. The rotation operation itself is unchanged, so it remains compatible with FlashAttention and existing implementations. On a production-scale social feed recommendation dataset, with identical backbone, features, and training pipelines, SIREN-RoPE consistently improved calibration (NE) and ranking quality (AUC) with only ~0.2% extra parameters. Ablations show the gate converges to 0.044, meaning the model relies mainly on temporal modulation. Visualizations reveal the model spontaneously discovers daily, weekly, and yearly patterns, confirmed by FFT peaks at 1/day and 1/7 per day.

SIREN-RoPE: Learning to Rotate — Temporal and Semantic Rotary Encoding for Sequential Modeling

Paper: Learning to Rotate: Temporal and Semantic Rotary Encoding for Sequential Modeling Authors: Hailing Cheng, Daqi Sun, Xinyu Lu arXiv: 2604.24717v1 Source: Papers.Cool daily recommendation

The forgotten stage: RoPE's limitation

RoPE (Rotary Position Embedding, Su et al., 2024) encodes sequence positions as rotation angles applied to Query and Key vectors. The farther apart two positions are, the larger the rotation difference, the smaller the dot product — so attention weights naturally reflect relative distance. It is elegant and has become standard in LLMs.

But the rotation angles are hard-coded, never learned. RoPE only asks "which token index are you?" — never "when did this happen?" For recommendation systems and event-stream modeling, this is a serious flaw: an interaction seven days ago and one seven minutes ago are equally treated as a distance of N positions, even though their semantic weight for predicting the next click differs enormously.

The complex-number inspiration

The authors' analogy: in the complex world, the real axis records magnitude and the imaginary axis records direction — two orthogonal dimensions with distinct roles. Similarly:

> Token embeddings encode semantics (real part) — what it is; rotation angles encode dynamics (imaginary part) — when it happens, at what rhythm, in what context.

The SIREN-RoPE architecture

The core formula is remarkably simple:

\Theta_j(T_i, p_i) = \underbrace{f_\phi(T_i)_j \cdot \omega^s_j}_{\text{Temporal (SIREN)}} + \underbrace{p_i \cdot \theta_j \cdot \lambda}_{\text{Ordinal (scaled)}}

Where:

$$T_i$$ is real timestamp feature (year, month, day, hour, minute, second, plus cyclical features)
$f_\phi$ is a dual-branch SIREN network
$\omega^s_j$ is a learnable per-dimension frequency scaling
$\theta_j$ is RoPE's original inverse-frequency constant
$\lambda$ is a learnable gated scalar controlling the ordinal contribution

The rotation operation itself is identical to standard RoPE — meaning SIREN-RoPE is seamlessly compatible with existing implementations, including FlashAttention.

Dual-branch SIREN network

SIREN (Sinusoidal Representation Networks, Sitzmann et al., 2020) was designed to combat neural networks' spectral bias. Here it becomes a "time translator":

SIREN branch: sinusoidal activations capture periodic time patterns — 24-hour circadian rhythms, 7-day work/weekend cycles, seasonal changes
DNN branch: a regular MLP captures aperiodic trends — time decay, linear or nonlinear drift

Additionally, each dimension learns its own "time sensitivity": some may respond to hour-level changes, others only to month-level changes.

Experimental results

Experiments were run on a production-scale social network feed dataset — a real recommendation scenario, not toy benchmarks.

Controlled setup

All models shared the exact same backbone (AttnMVP), feature set, and training pipeline. The only variable was how time information enters the model — so result differences can be confidently attributed to the encoding itself.

Consistent gains, almost free

SIREN-RoPE consistently improved calibration (NE) and ranking quality (AUC) on three user-engagement tasks, with only ~0.2% additional parameters.

The most fascinating ablation

With ordinal RoPE only (no time signal), the gate scalar $\lambda$ stays near its initial value of 1.0 — the model never "discovers" time's meaning.
With SIREN-RoPE, $\lambda$ converges to 0.044 — the model learns to rely mainly on temporal modulation, nearly abandoning discrete ordinal positions.

What did the model learn on its own?

The authors extracted the trained SIREN-RoPE module, used fixed unit Query/Key vectors, varied only timestamps, and observed attention scores:

Daily cycle: attention curves from two different days overlap almost exactly — the model autonomously discovered the 24-hour circadian rhythm, with no explicit "hour" feature
Weekly cycle: two weeks of curves overlap perfectly — the 7-day work/weekend cycle, the strongest periodicity
Monthly cycle: absent, because 7 does not divide 30 — a mathematical "dissonance"
Yearly cycle: only a smooth monotonic decay, captured by the DNN branch's long-term trend

FFT spectral analysis confirms two dominant peaks precisely at 1 cycle/day and 0.143 cycles/day (1/7).

Deeper significance

The authors frame this as a manifesto about Transformer fundamentals: for seven years the community has polished the semantic embedding space (depth, width, heads) while barely touching the rotation manifold — the orthogonal "imaginary part" of attention:

> "The rotation space should not be viewed as a settled positional encoding detail, but as an undeveloped axis — whose rich structure may influence attention as much as the imaginary unit influenced algebra."

Future directions

1. Theoretical characterization: what function classes can the rotation manifold express? 2. Semantic-temporal rotation: let $f_\phi$ depend on token type or categorical metadata, not just timestamps 3. Cross-attention variants: different angle functions on Query and Key sides for asymmetric time sensitivity 4. Cross-modal unification: vision, audio, and text sequences sharing a learned rotation space while keeping independent semantic embeddings

References

Cheng H., Sun D., Lu X. (2026). *Learning to Rotate: Temporal and Semantic Rotary Encoding for Sequential Modeling*. arXiv:2604.24717v1.
Su J., Lu Y., Pan S., Murtadha A., Wen B., Liu Y. (2024). RoFormer: Enhanced Transformer with Rotary Position Embedding. *Neurocomputing*.
Sitzmann V., Martel J., Bergman A., Lindell D., Wetzstein G. (2020). Implicit Neural Representations with Periodic Activation Functions. *NeurIPS*.
Vaswani A. et al. (2017). Attention Is All You Need. *NeurIPS*.
Kazemi S.M. et al. (2019). Time2Vec: Learning a Vector Representation of Time. *AAAI*.