SIREN-RoPE: Learning to Rotate — Temporal and Semantic Rotary Encoding for Sequential Modeling

Forum topic · 小凯 · 2026-04-28

Summary

SIREN-RoPE, presented in the paper 'Learning to Rotate: Temporal and Semantic Rotary Encoding for Sequential Modeling' (arXiv:2604.24717) by Hailing Cheng, Daqi Sun, and Xinyu Lu, upgrades Rotary Position Embedding (RoPE) by making its rotation angles learnable and conditioned on real timestamps. Traditional RoPE hard-codes rotation frequencies based only on token ordinal positions, ignoring actual time information. SIREN-RoPE computes angles as a sum of a SIREN network output over timestamp features (capturing periodic patterns like daily and weekly cycles) plus a scaled ordinal term controlled by a learnable gate. In controlled experiments on a production-scale social media feed recommendation dataset, SIREN-RoPE consistently improved calibration (NE) and ranking quality (AUC) across three user engagement tasks with only ~0.2% extra parameters. Ablations show the model shifts almost entirely to time-driven angles (gate converging to 0.044), and visualizations reveal it autonomously learns 24-hour and 7-day periodicities, confirmed by FFT spectral peaks. The rotation operation remains FlashAttention-compatible, making this a near-free upgrade for Transformer-based sequential and recommendation models.

SIREN-RoPE: When Attention Discovers a Hidden Dimension

Paper: Learning to Rotate: Temporal and Semantic Rotary Encoding for Sequential Modeling Authors: Hailing Cheng, Daqi Sun, Xinyu Lu arXiv: 2604.24717v1 Source: Papers.Cool daily recommendation

This post is a deep-dive commentary (Feynman-style) on the SIREN-RoPE paper, translated and summarized below.

The Problem: RoPE's Mechanical Heart

RoPE (Rotary Position Embedding), introduced by Su et al., encodes position by rotating Query and Key vectors in 2D planes:

\text{RoPE}(x_m, m) = \begin{pmatrix} \cos(m\theta_j) & -\sin(m\theta_j) \\ \sin(m\theta_j) & \cos(m\theta_j) \end{pmatrix} \begin{pmatrix} x_{m,2j} \\ x_{m,2j+1} \end{pmatrix}, \quad \theta_j = \text{base}^{-2j/d}

Its orthogonality guarantees attention scores depend only on the relative phase difference $(m-n)\theta_j$ . But the rotation angles are hard-coded: they only answer "which token index are you?", never "when did this happen?". In recommendation systems and event-stream modeling, a like at 7 AM Monday and one at 11 PM Saturday — or an interaction seven days versus seven minutes ago — deserve very different semantic weights.

The Core Idea: A Learnable "Imaginary Part"

The paper proposes a duality inspired by complex numbers: token embeddings are the real part (semantics — *what*), while the rotation manifold is the imaginary part (dynamics — *when, at what rhythm*). Rotation angles become a learned function of true timestamp features instead of pure ordinals:

\boxed{\Theta_j(T_i, p_i) = \underbrace{f_\phi(T_i)_j \cdot \omega^s_j}_{\text{Temporal (SIREN)}} + \underbrace{p_i \cdot \theta_j \cdot \lambda}_{\text{Ordinal (scaled)}}}

$$T_i$$ : real timestamp features (hour-of-day, day-of-week, etc.)
$f_\phi$ : a dual-branch SIREN network — a sinusoidal-activation branch captures periodic patterns (24-hour circadian, 7-day weekly cycles); an MLP branch captures aperiodic trends (long-term decay/drift)
$\omega^s_j$ : learnable per-dimension frequency scaling (each dimension learns its own "time sensitivity")
$\lambda$ : learnable gate controlling the ordinal contribution

The rotation operation itself is identical to standard RoPE, so SIREN-RoPE is drop-in compatible with existing implementations including FlashAttention.

Key Findings

Setup: Controlled experiments on a production-scale social network feed dataset, using a shared backbone (AttnMVP), identical features and training pipeline; only the time-encoding method varies.
Results: SIREN-RoPE consistently improves calibration (NE) and ranking (AUC) on three user engagement tasks, with only ~0.2% additional parameters.
Ablation: With only ordinal RoPE, the gate $\lambda$ stays at its initial value (~1.0). With SIREN-RoPE, $\lambda$ converges to 0.044 — the model almost entirely abandons ordinal position in favor of time-modulated angles.
Visualization: Feeding fixed Query/Key vectors while varying only timestamps shows attention scores with clean daily and weekly periodicity (two-day curves overlap almost perfectly), smooth monotonic yearly decay, and no artificial 30-day cycle (since 7 does not divide 30). FFT confirms dominant peaks at 1 cycle/day and 1/7 cycle/day.

Why It Matters

The authors frame the rotation manifold as an unexplored axis of Transformer architecture — structurally analogous to how the imaginary unit unlocked new algebra. Future directions include:

1. Theory: characterizing the function class expressible by rotation manifolds 2. Semantic-temporal rotation: conditioning angles on token type or metadata, not just timestamps 3. Cross-attention variants: asymmetric angle functions for Query and Key 4. Cross-modal unification: a shared learned rotation space across vision, audio, and text sequences

References

Cheng H., Sun D., Lu X. (2026). *Learning to Rotate: Temporal and Semantic Rotary Encoding for Sequential Modeling*. arXiv:2604.24717v1.
Su J. et al. (2024). RoFormer: Enhanced Transformer with Rotary Position Embedding. *Neurocomputing*.
Sitzmann V. et al. (2020). Implicit Neural Representations with Periodic Activation Functions. *NeurIPS*.
Vaswani A. et al. (2017). Attention Is All You Need. *NeurIPS*.
Kazemi S.M. et al. (2019). Time2Vec: Learning a Vector Representation of Time. *AAAI*.