GRU Explained: How Two Gates Beat LSTM's Three — A Deep Dive into Gated Recurrent Units

> Source: Episode 25 of "Plain-Language LLMs" — GRU (Gated Recurrent Unit) > Paper: Cho et al. (2014) "Learning Phrase Representations using RNN Encoder-Decoder" > Comparison experiments: Chung et al. (2014) arXiv:1412.3555 > Variant study: Dey et al. (2017) arXiv:1701.05923

---

1. The Problem with LSTM: It's Too Complex

LSTM solves the vanishing gradient problem of vanilla RNNs using three gates (input, forget, output) and a cell state. But this design comes at a cost:

Too many parameters.

An LSTM cell has 4 weight matrix groups (input gate, forget gate, output gate, candidate state), each requiring matrix multiplications with both the input xₜ and the previous state hₜ₋₁. The parameter count is 4× that of a plain RNN.

In 2014, Cho and Bengio's team asked:

> "Can we keep the core idea of gating while making the architecture simpler?"

The answer is GRU: two gates, one hidden state, and 1/3 fewer parameters than LSTM.

---

2. The Core of GRU: A Story of Two Gates

GRU simplifies LSTM's complex memory management into two gates:

1. Reset Gate

Question: Should old memory still participate?

The reset gate rₜ decides how much of the previous hidden state hₜ₋₁ is "ignored" when computing the current candidate state:

When rₜ approaches 0, GRU ignores old memory and computes the new state based only on the current input — effectively "forgetting past interference."

2. Update Gate

Question: How should old and new memories be mixed?

The update gate zₜ decides how much of the new state comes from the old state versus the candidate state:

where the candidate state is:

The Key Simplification: No Separate Cell State

LSTM maintains two states: the cell state cₜ (long-term memory) and the hidden state hₜ (output). GRU merges both into a single hidden state hₜ. This means:

• Fewer parameters: no separate cell-state weights
• More direct gradient flow: information doesn't need to pass through two gates
• Faster computation: one fewer matrix multiplication per step

---

3. GRU vs LSTM: Is the Performance Really Comparable?

In late 2014, Chung, Gulcehre, Cho, and Bengio ran systematic comparisons.

Experimental Setup

| Task | Datasets | Sequence Length | |------|----------|-----------------| | Polyphonic music modeling | Nottingham, JSB Chorales, MuseData | Hundreds to thousands of steps | | Speech signal modeling | Internal Ubisoft data | Raw waveforms |

Key Findings

"GRU and LSTM are comparable in performance."

Under fixed parameter counts (a controlled, fair comparison), GRU on multiple datasets:

• Converges faster: less CPU time
• Requires fewer parameter updates: higher training efficiency
• Matches generalization: test-set performance on par with or slightly better than LSTM

This leads to a key conclusion:

> For sequence modeling, the presence of gating matters more than the number of gates.

---

4. Why Does GRU Work? The Mathematical Essence of Gating

The Root of Vanishing Gradients

When gradients backpropagate through time in a vanilla RNN, they pass through tanh activations, which compresses them into the (-1,1) range. Multiplying by the weight matrix at each step causes gradients to either decay exponentially (vanish) or grow exponentially (explode):

\[\frac{\partial h_t}{\partial h_{t-1}} = W^T \cdot \text{diag}(1 - \tanh^2(h_{t-1}))\]

The Gating Solution

GRU's update gate zₜ provides an additive path (a residual-like connection):

\[h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t\]

When zₜ ≈ 0, hₜ ≈ hₜ₋₁ — gradients can pass through time nearly losslessly. This is the core mechanism by which GRU solves vanishing gradients.

Key insight:

• LSTM uses the cell state cₜ as a gradient highway
• GRU uses the hidden state hₜ itself as the gradient highway (when zₜ ≈ 0)
• Both are essentially the same: providing a shortcut path that bypasses activation-function compression

---

5. Can It Be Simplified Further? GRU Variant Studies

In 2017, Dey et al. explored the limits of GRU: does performance degrade if gate parameters are further reduced?

Three Variants

| Variant | Gate Computation | Parameter Reduction | |---------|------------------|---------------------| | GRU1 | Only hₜ₋₁ + bias, no xₜ | Reduces 2×nm | | GRU2 | Only hₜ₋₁, no xₜ or bias | Reduces 2×(nm+n) | | GRU3 | Only bias (constant gates) | Reduces 2×(nm+n²) |

Results (MNIST pixel-by-pixel sequences)

• GRU1/GRU2: performance nearly identical to the original GRU, with far fewer parameters
• GRU3: clearly degraded, but still trainable (requires a lower learning rate)

This means: among the gate inputs, the historical state hₜ₋₁ matters more than the current input xₜ.

An Even More Radical Simplification: MGU

Minimal Gated Unit (MGU) — keep only one gate (the update gate) and drop the reset gate entirely:

\[h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tanh(W_h x_t + U_h h_{t-1})\]

Reports show MGU matches GRU/LSTM performance on multiple tasks, further confirming:

> The core of gating is "selectively passing information," not "precisely controlling every step of the computation."

---

6. Design Philosophy: Why "Simpler" Is Right

GRU's success teaches a deeper lesson about AI design:

1. Occam's Razor Works in Neural Networks

LSTM's three gates plus a cell state offer finer-grained control in theory, but experiments show this extra complexity brings no performance payoff.

Why?

Because gradient descent — the optimizer — isn't good at exploiting complex gate structures. Simpler gates are actually easier to optimize to good parameters.

2. Parameter Efficiency = Data Efficiency

GRU has 1/3 fewer parameters, meaning:

• Less prone to overfitting on small datasets
• Faster training (fewer matrix multiplications per step)
• More feasible on mobile/embedded devices

> "Parameters are not free. Every extra parameter increases the model's data requirements."

3. The Value of Gating Lies in Its Existence, Not Its Quantity

The fundamental difference between GRU, LSTM, and vanilla RNNs is not the number of gates but whether gating exists at all.

With gates → long-term dependencies can be learned Without gates → gradients vanish; only short-term patterns are captured

The specific gate design (two vs. three) is secondary.

---

7. GRU's Limitations

The papers and follow-up research also point out GRU's weaknesses:

1. LSTM May Be Better with Large Data

LSTM's extra parameters can pay off on very large datasets. With enough data, LSTM's "over-engineering" can be fully exploited by the optimizer.

2. Some Tasks Need Fine-Grained Gating

On counting tasks (e.g., learning to copy binary sequences), GRU underperforms LSTM because it lacks an output gate. Schmidhuber's team noted GRU "can neither learn to count" — tasks requiring precise gate control.

3. Superseded by Transformers

In NLP, GRU and LSTM have largely been replaced by the Transformer architecture. Self-attention provides more direct "any-to-any connections," removing the need for gates to carry gradients.

But GRU remains active in:

• Time series forecasting (finance, energy, weather)
• Small sequence models (resource-constrained environments)
• Hybrid RNN–Transformer architectures (inspiration for RWKV, Mamba)

---

8. Modern Relevance: What GRU Means in 2026

1. Why Learn GRU Today?

Transformers aren't omnipotent. For:

• Online learning (streaming data; can't wait for full sequences)
• Long-sequence inference (memory limits make O(n²) attention infeasible)
• Small devices (phones, IoT — every parameter saved counts)

the RNN family's efficiency advantage persists, and GRU is one of its most elegant members.

2. Mamba's Inspiration

The Mamba architecture (2023–2024, S4 + selection mechanism) can be seen as carrying forward GRU's spirit:

• Selective state spaces: akin to GRU's gating, but more efficient
• Hardware-aware design: like GRU, focused on computational efficiency
• Linear complexity: solving Transformer's O(n²) bottleneck

GRU's design philosophy — "simplify as much as possible while preserving performance" — directly influenced this new generation of sequence models.

3. Practical Advice for Developers

| Scenario | Recommended Architecture | Reason | |----------|--------------------------|--------| | Small dataset + sequence task | GRU | Fewer parameters, less overfitting | | Large dataset + complex sequences | LSTM | More parameters, more capacity | | Resource-constrained environments | GRU | Higher computational efficiency | | Modern NLP tasks | Transformer | Parallel training, SOTA results | | Ultra-long online sequence tasks | Mamba/RWKV | Linear complexity, RNN's streaming advantage |

---

9. Conclusion: What GRU Teaches Us

GRU is not just an architecture — it's a design principle:

> "Truly great design isn't about maximum complexity, but finding just the right amount of it."

• LSTM proved gating can rescue RNNs
• GRU proved you don't need that many gates to achieve the same results
• MGU proved a single gate can even suffice

The lesson: in deep learning, identifying the truly important mechanism (gating) and stripping away all unnecessary ornamentation often yields better results.

GRU's two gates — reset and update — are not a degraded version of LSTM, but its refinement. It kept the essence of gating, removed the redundancy, and ultimately proved:

Simplicity itself is a form of power.

---

References

• Cho et al. (2014). "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation." arXiv:1406.1078
• Chung et al. (2014). "Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling." arXiv:1412.3555
• Dey et al. (2017). "Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks." arXiv:1701.05923
• Schmidhuber et al., commentary on GRU: Annotated History of Modern AI and Deep Learning