Temporal Credit Is Free: Why Did It Take 30 Years to Realize the Jacobian Is Unnecessary?
> Imagine learning to ride a bike. Traditional wisdom says you must precisely compute pedal force, handlebar angle, and center-of-mass shifts—a complicated set of "physics equations." But one day you realize: you only need to sense the current imbalance; your body adjusts automatically. Past experience is already encoded in muscle memory. > > That is the core finding of this paper.
1. The Dilemma of Online Learning
Online learning means a model updates itself while processing a data stream—like a robot learning to walk while walking.
Recurrent neural networks (RNNs) pose a special challenge: the hidden state acts as a memory box. When the model errs, which past input is to blame? This is the temporal credit assignment problem.
The Standard Answer: RTRL
In 1989, Williams and Zipser proposed RTRL (Real-Time Recurrent Learning), maintaining a sensitivity tensor P_t = ∂h_t/∂θ via the update P_t = J_t · P_{t-1} + immediate gradients, where J_t = ∂h_t/∂h_{t-1} is the state Jacobian. The cost of computing J_t · P_{t-1} is O(n⁴) per step—tractable for n=64, prohibitive for n=1024.
30 Years of Approximations
| Method | Complexity | Strategy | |--------|-----------|----------| | UORO (Tallec & Ollivier, 2017) | O(n²) | Rank-1 compression | | KF-RTRL (Benzing et al., 2019) | O(n³) | Kronecker-sum approximation | | Sparse RTRL (Menick et al., 2021) | O(n²) | Graph-structured sparsity | | LRUs (Zucchet et al., 2023) | O(n) | Diagonal recurrence |
All share one core assumption: the J_t · P_{t-1} term carries gradient information unavailable elsewhere. This paper says: that assumption is wrong.
2. Two Overlooked Mistakes
An Accidental Discovery
In prior work on sparse Jacobian propagation, setting k=1 (with a circular-mask implementation bug) propagated no Jacobian at all—only immediate derivatives—yet recovered 86% of full RTRL performance. If zero propagation recovers most of the signal, why do standard eligibility traces fail?
Mistake #1: Severely Miscalibrated Trace Decay
Eligibility traces use e_t = λ · e_{t-1} + immediate gradient with λ typically 0.95. But in a trained vanilla RNN (n=64), the measured neuronal self-propagation factor is ≈ 0.01 per step:
| Time step | λ=0.95 weight | True influence | Ratio | |-----------|---------------|----------------|-------| | t-10 | 0.95¹⁰ ≈ 0.60 | 0.01¹⁰ ≈ 10⁻²⁰ | ~10²¹× | | t-100 | 0.95¹⁰⁰ ≈ 0.006 | 0.01¹⁰⁰ ≈ 10⁻²⁰⁰ | astronomical |
At t=500, the accumulated trace is 6.7x larger than the immediate derivative and points in a different direction (cosine similarity 0.577). Traces deliver stale gradients at the wrong scale—not temporal credit.
Mistake #2: Hidden Gradient-Scale Catastrophe
In a trained vanilla RNN (n=64):
- Recurrent-weight gradient norm ∂L/∂W_hh ≈ 0.001
- Output-weight gradient norm ∂L/∂W_out ≈ 0.1
- Decay from 0.001 to 0.5: recovery rates of 102–200%
- Decay = 0.95 (the standard default): 0% recovery (every seed fails)
- Decay 0.7: high variance (77±56%); decay 0.5: reliable (130±8%)
- Vanilla GRU: the update gate creates a linear-interpolation gradient path bypassing the nonlinear candidate computation (ratio only 5–19x).
- SSM (S4): direct output path via C and D matrices bypasses the recurrent update entirely.
- Block-GRU+MLP: the MLP receives gradients 50,000x larger than the GRU internals; on Lorenz, SGD beats Adam by 50x.
- Cross-session BCI decoding (primate reaching, 7 months of electrode drift): Adam 82%, RMSprop (α=0.99) 106%, Sparse RTRL (k=4) 85%, Full RTRL 50% (2 of 5 seeds partially diverged).
- Streaming ML benchmarks (River Hyperplane): RNN + decay 0.0 + Adam achieves 0.925 accuracy vs Adaptive Random Forest 0.842 (+8.3 pp).
- For single-layer recurrent architectures: set trace decay to zero and use Adam or RMSprop—matches or beats full RTRL at far lower cost, with no Jacobian computation.
- For pretrained models: use LoRA + Adam for online domain adaptation. SGD cannot.
- Paper: Shalev Merin, A. (2026). *Temporal Credit Is Free*. arXiv:2603.28750.
- RTRL: Williams, R. J., & Zipser, D. (1989). A learning algorithm for continually running fully recurrent neural networks.
- Eligibility traces: Murray, J. M. (2019). Local online learning in recurrent networks with random feedback.
- Adam: Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization.
A 100x gap, caused by the tanh derivative (1 − h_t²) compressing recurrent gradients while the output path is a direct linear projection. SGD applies the same learning rate to both, so recurrent dynamics barely adapt.
3. The Fix: Two Simple Corrections
Fix #1: Set Trace Decay to Zero
The hidden state already carries all temporal information via the forward pass—the hidden state itself is the carrier of temporal credit. Experiments show:
Everything below 0.5 is a safe plateau; the standard 0.95 default sits on the wrong side of a cliff.
Fix #2: Adam's β₂ (or RMSprop)
With immediate derivatives only:
| Optimizer | Sine task recovery | Delayed task recovery | |-----------|-------------------|----------------------| | SGD | −34% | −58% | | SGD + Momentum | 12% | −59% | | Adam β₁-only | −34% | −58% | | RMSprop (α=0.99) | 91% | 179% | | Adam β₂-only | 33% | 167% | | Adam (full) | 102% | 147% |
Every configuration with second-moment normalization (β₂) adapts; none without it does. Momentum (β₁) alone adds nothing. β₂ equalizes effective learning rates across parameter groups.
4. Architectural Rule: When Is β₂ Needed?
Testing 10 architectures:
| Architecture | Adam/β₂ | SGD | Gradient ratio | Output bypass | |--------------|---------|-----|----------------|---------------| | Vanilla RNN | 102% | −34% | 100x | none | | LSTM | 99.7% | 59.4% | ~31x | none | | CTRNN | 89.8% | ~0% | 278x | none | | xLSTM (sLSTM) | 91.5% | 60.0% | ~24x | none | | RWKV | adapts | 0% | ∞ | none | | RetNet | 93% | −17% | ~64x | none | | Vanilla GRU | adapts | adapts | 5–19x | update gate | | SSM (S4) | adapts | adapts | <6x | C/D outputs | | Block-GRU+MLP | adapts | 50x better | 50000x | MLP output |
The rule: per-parameter gradient normalization is required when (1) the recurrent state update involves nonlinear/multiplicative transformations that compress gradients, AND (2) no parallel output path of sufficient capacity exists. If either condition is absent, SGD suffices.
5. Remarkable Scalability
Memory
| Method | Complexity | n=64 | n=1024 | |--------|-----------|------|--------| | Immediate derivatives (decay=0) | O(n²) | 32 KB | 12.6 MB | | Sparse RTRL (k=4) | O(n²) | ~1 MB | ~500 MB | | Full RTRL | O(n³) | 1 MB | 12.9 GB |
At n=1024, a 1000x gap.
Real-World Validation
Pretrained Language Models (LoRA adapters, online, no replay)
| Model | Params | Adam β₂ | SGD | |-------|--------|---------|-----| | GPT-2 | 124M | 21.6% | 0.0% | | Mamba-130M | 130M | 51.6% | 0.1% | | TinyLlama | 1.1B | 31.6% | 0.1% | | Mamba-1.4B | 1.4B | 133.5% | 2.4% | | Qwen2 | 7B | 31.3% | 0.0% |
From 124M to 7B, SGD achieves zero adaptation on every model. With rank-16 adapters and 40K tokens, GPT-2 reaches 136.8% recovery.
6. Why Did This Take 30 Years?
1. Nobody measured the actual self-propagation factor. RFLO used λ=0.9, e-prop used membrane time constants, 0.95 became convention—yet computing diag(1−h_t²)·diag(W_hh) is trivial. 2. Adam didn't exist until 2014. For 25 years everyone used SGD, for which the "temporal propagation is necessary" conclusion was actually true. The field never revisited it after per-parameter normalization became standard. 3. The errors reinforced each other. Believing temporal propagation was necessary, nobody investigated why immediate-only + SGD failed; when it failed, they concluded propagation was necessary. Nobody tried immediate-only + Adam.
7. Deeper Insight: Isotropy of the Parameter Jacobian
The parameter Jacobian of trained RNNs is near-isotropic—singular values roughly uniform, condition number 2.6–6.5 at n=64, scaling as n^0.216. This isotropy is inherited from random Gaussian initialization and approximately preserved by training. Because gradient information is spread roughly evenly across all parameter directions, any single direction—including the immediate derivative—captures most of the relevant signal. There is no privileged subspace for time to reveal.
8. Limitations and Open Questions
1. Single-layer networks: all from-scratch experiments use one layer. Cross-layer Jacobians in two-layer RNNs are highly anisotropic (condition number 3773). 2. Temporal horizons: tasks involve dependencies up to 50 steps; the claim concerns gradient computation, not long-horizon credit assignment. 3. Pretrained models: direct online updates of foundation models yield near-zero adaptation regardless of optimizer; findings apply only through parameter-efficient adapters like LoRA.
Open questions: deep recurrent networks' cross-layer anisotropy bottleneck; whether immediate derivatives suffice under non-gradient rules (e.g., predictive coding); practical engineering for streaming adaptation (update frequency, adapter rank, long-stream stability).
9. Practical Recommendations
Checklist: 1. Nonlinear state update? (tanh, sigmoid, multiplicative interactions) 2. Output bypass? (linear path, C/D matrices, MLP output) 3. If (1) yes and (2) no → β₂ normalization is mandatory 4. If (2) yes → SGD may suffice or be better
10. Closing Thoughts: A Free Lunch
The title "Temporal Credit Is Free" is a pun: temporal credit assignment needs no expensive Jacobian propagation—the hidden state provides it for free through the forward pass—and it took 30 years to notice because of two simple mistakes that kept the field searching for complex solutions.
Sometimes scientific progress comes not from discovering new complexity, but from realizing a simple truth we had ignored all along. Like riding a bike: you don't recalculate physics each time. The RNN's hidden state already knows what to do—we just need to listen properly.
References
*Written April 2026, based on a deep reading of arXiv:2603.28750.*