DAMI: Dual-Axis Memory Initialization — Setting Each Memory's "Shelf Life" at Birth

This post introduces and explains the paper *DAMI: Dual-Axis Memory Initialization for Enhanced Long-Context Retrieval in Gated Linear Attention* (arXiv:2610.41994), by Zhangchi Feng (SpeedAI; Beihang University), Harry Jiang, Ray Zhang, and Richong Zhang (corresponding author, Beihang).

Key points

A curious observation: training barely moves the forgetting parameters

• In gated linear attention models (Mamba-2, RWKV-7, GDN family), the parameters controlling decay speed — A_log (base decay rate) and dt_bias (input-dependent modulation) — stay highly correlated with their random initial values after training: Pearson correlation 0.871–0.989 across five architectures (GDN, KDA, GDN2, Mamba-2, RWKV-7).
• Implication: if training only fine-tunes around initialization, initialization quality becomes decisive — like a native accent set at birth.

How fading memory works

• Linear attention replaces the Transformer's unbounded "blackboard" with a fixed-size state whose old contents are multiplied by a decay factor (<1) each step. Forgetting is inevitable; the question is whether it follows a principled pattern.
• Each head has its own A_log/dt_bias; the effective decay rate depends on their product with input activations.

Axis 1: geometric time scales across heads

• Natural language mutual information decays as a power law in token distance (exponent roughly -0.5 to -0.9, per Lin & Tegmark 2017). By the Beylkin–Monzón theorem, a power law can be represented as a mixture of exponentials with geometrically spaced time constants.
• DAMI assigns 6 heads time scales from 1 to 4096 tokens at a ratio of ~5.28: {4096, 776.05, 147.03, 27.86, 5.28, 1} — short heads handle syntax and recent references, long heads handle distant dependencies.

Axis 2: low–high–low input sensitivity across depth

• Sensitivity = the Softplus derivative at z + dt_bias. DAMI targets ~0.25 in the bottom third of layers, ~0.75 in the middle, ~0.25 in the top (for an 18-layer net).
• Rationale: bottom layers should stably encode local lexical patterns; middle layers should decide what is worth remembering; top layers should preserve long-term information without frequent rewriting.

Gradient-free calibration (LSUV-style)

1. Set dt_bias per head via bisection search so sensitivity hits the layer's target. 2. Set A_log by measuring the empirical distribution of z and solving A_log = -log(τ) - log(E[softplus(z + dt_bias)]) so each head's *effective* decay time scale equals the target τ.
• No new trainable parameters, no change to the update rule or training pipeline. Architecture-specific adaptations: centered perturbations (std 0.03) for GDN/GDN2 channel-level biases, resampled scales for Mamba-2, 160 frozen scales for RWKV-7.

Results

• Setup: five architectures, ~100M parameters, 5B tokens, 4k context; baselines are random init and PoST.
• RULER NIAH gains vs random init: GDN +13.95, KDA +16.71, GDN2 +13.44, Mamba-2 +14.47, RWKV-7 +5.29 points; vs PoST: +8.47 to +10.87. Best in 42 of 55 task-length combinations; gains persist at 8K test length (2× training context).
• Ablations: randomizing either axis while keeping the other performs *worse* than full random (e.g., GDN: 22.80 with only dt_bias configured vs 30.42 fully random vs 44.36 full DAMI) — the two axes must be jointly configured. Coverage of long scales matters (1–4096: 44.36 vs 1–1024: 23.13). Linear (rather than geometric) spacing collapses NIAH-2 from 35.59 to 5.65. Alternative sensitivity profiles score only 26.85–32.85.
• General ability on 30 benchmarks (WikiText PPL, WinoGrande, ARC, OpenBookQA, Social IQa) is essentially unchanged, leading on 17.

Honest limitations

• DAMI improves retrieval, not reasoning. The power-law basis derives from natural language; code, DNA, or other modalities may need different spectra. Validation is at 100M parameters / 4k context; optimal constants at larger scale are open. However, the initialization-determinism observation may grow *stronger* with scale, since individual parameters receive less per-parameter gradient attention.

Why doesn't training move these parameters?

Possible reasons discussed: long gradient chains dilute updates to decay parameters; log-parameterization creates flat regions where gradients barely move; and most training tokens involve short-range dependencies, giving weak signal for decay tuning. DAMI accepts this and optimizes initialization instead of fighting optimization dynamics.

References (as cited in the post)

1. Feng et al., *DAMI: Dual-Axis Memory Initialization for Enhanced Long-Context Retrieval in Gated Linear Attention*. arXiv:2610.41994, 2026. 2. Yang et al., *Gated Delta Networks*, ICLR 2025. 3. Peng et al., *RWKV-7 "Goose"*, arXiv:2503.14456, 2025. 4. Gu & Dao, *Mamba*, COLM 2024. 5. Lin & Tegmark, *Critical Behavior from Deep Learning*, arXiv:1712.06556, 2017. 6. Beylkin & Monzón, *Approximation by Exponential Sums Revisited*, ACHA, 2010. 7. Mishkin & Matas, *All You Need Is a Good Init*, ICLR 2016. 8. Hsieh et al., *RULER*, COLM 2024. 9. Xiao et al., *Pretraining-on-Test*, arXiv:2509.14739, 2025. 10. Bai et al., *Segmentation of DNA Sequences and Power-Law Statistics*, PRL, 1992. 11. Waleffe et al., *An Empirical Study of Mamba-based Language Models*, 2024.

The post's closing thought: this is not a paper about remembering more, but about forgetting more wisely — the best models may be "delivered," not merely trained, with each memory's shelf life set on day one.