GRU Explained: How Two Gates Beat Three Gates — A Deep Dive into Gated Recurrent Units

> Source: Episode 25 of "Plain-Language LLMs" video series · Gated Recurrent Unit (GRU) > Original paper: Cho et al. (2014), "Learning Phrase Representations using RNN Encoder-Decoder" > Comparison experiments: Chung et al. (2014), arXiv:1412.3555 > Variant study: Dey et al. (2017), arXiv:1701.05923

1. The Problem with LSTM: It's Too Complex

LSTM solves the RNN vanishing-gradient problem with three gates (input, forget, output) plus a cell state. But there's a cost: too many parameters.

An LSTM cell has 4 groups of weight matrices (input gate, forget gate, output gate, candidate state), each requiring matrix multiplication with both the input xₜ and the previous state hₜ₋₁ — 4× the parameters of a plain RNN.

In 2014, Cho and Bengio's team asked:

> "Can we keep the core idea of gating while making the structure simpler?"

The answer is GRU: two gates, a single hidden state, one-third fewer parameters than LSTM.

2. The Core of GRU: A Story of Two Gates

GRU simplifies LSTM's complex memory management into two gates:

1. Reset Gate

Question: Should old memory still participate?

The reset gate rₜ decides how much of the previous hidden state hₜ₋₁ is "ignored" when computing the current candidate state:

When rₜ approaches 0, GRU ignores old memory and computes the new state based only on the current input — effectively "forgetting past interference."

2. Update Gate

Question: How should old and new memory be mixed?

The update gate zₜ decides how much of the new state comes from the old state versus the candidate state:

where the candidate state is:

Key Simplification: No Separate Cell State

LSTM maintains two states: cell state cₜ (long-term memory) and hidden state hₜ (output). GRU merges them into a single hidden state hₜ. This means:

• Fewer parameters: no separate cell-state weights
• More direct gradient flow: information doesn't pass through two gates
• Faster computation: one fewer matrix multiplication per step

3. GRU vs LSTM: Is Performance Really Comparable?

In late 2014, Chung, Gulcehre, Cho, and Bengio ran systematic comparisons.

Experimental Setup

| Task | Datasets | Sequence Length | |------|----------|-----------------| | Polyphonic music modeling | Nottingham, JSB Chorales, MuseData | Hundreds to thousands of steps | | Speech signal modeling | Ubisoft internal data | Raw waveforms |

Key Findings

"GRU and LSTM are comparable in performance."

With parameter counts fixed for a fair comparison, GRU achieved on multiple datasets:

• Faster convergence: less CPU time
• Fewer parameter updates: higher training efficiency
• Comparable generalization: test performance equal to or slightly better than LSTM

This leads to a key conclusion:

> For sequence modeling, the presence of a gating mechanism matters more than the number of gates.

4. Why GRU Works: The Mathematical Essence of Gating

Root of Vanishing Gradients

In a plain RNN, gradients backpropagated through time pass through tanh activations, compressed to the (-1,1) range. Multiplying by weight matrices at each step, gradients either decay exponentially (vanish) or grow exponentially (explode):

\[\frac{\partial h_t}{\partial h_{t-1}} = W^T \cdot \text{diag}(1 - \tanh^2(h_{t-1}))\]

The Gating Solution

GRU's update gate zₜ provides an additive path (a residual-like connection):

\[h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t\]

When zₜ ≈ 0, hₜ ≈ hₜ₋₁ — gradients can pass through time almost losslessly. This is the core mechanism by which GRU combats vanishing gradients.

Key insight: LSTM uses the cell state cₜ as a gradient highway; GRU uses the hidden state hₜ itself (when zₜ ≈ 0). Both are essentially the same: providing gradients a shortcut path unimpeded by activation-function compression.

5. Can It Be Simplified Further? GRU Variant Studies

In 2017, Dey et al. explored GRU's limits: does performance degrade if gate parameters are reduced further?

Three Variants

| Variant | Gate Computation | Parameter Reduction | |---------|------------------|---------------------| | GRU1 | Only hₜ₋₁ + bias, no xₜ | 2×nm fewer | | GRU2 | Only hₜ₋₁, no xₜ or bias | 2×(nm+n) fewer | | GRU3 | Only bias (constant gates) | 2×(nm+n²) fewer |

Results (MNIST pixel-by-pixel sequences)

• GRU1/GRU2: performance nearly identical to the original GRU despite large parameter reductions
• GRU3: clear performance drop, but still trainable (requires a lower learning rate)

Implication: among gate inputs, the historical state hₜ₋₁ is more critical than the current input xₜ.

More Aggressive Simplification: MGU

The Minimal Gated Unit (MGU) keeps only one gate (the update gate) and removes the reset gate entirely:

\[h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tanh(W_h x_t + U_h h_{t-1})\]

Studies report MGU performing comparably to GRU/LSTM on multiple tasks, further confirming:

> The core of gating is "selective information transfer," not "precise control over every step."

6. Design Philosophy: Why "Simpler" Is Right

1. Occam's Razor Works in Neural Networks

LSTM's three-gate-plus-cell-state design offers finer-grained control in theory, but experiments show the extra complexity brings no performance payoff. Gradient descent, it turns out, doesn't exploit complex gate structures well — simpler gates are easier to optimize.

2. Parameter Efficiency = Data Efficiency

With one-third fewer parameters, GRU:

• is less prone to overfitting on small datasets
• trains faster (fewer matrix multiplications per step)
• is more feasible on mobile/embedded devices

> "Parameters are not free. Every extra parameter increases the model's data appetite."

3. The Core Value of Gating Is "Existence," Not "Quantity"

The fundamental difference between GRU, LSTM, and plain RNNs is not gate count but whether gating exists:

• Gated → can learn long-term dependencies
• Ungated → vanishing gradients, short-term patterns only

The specific gate design (two vs. three) is secondary.

7. GRU's Limitations

1. LSTM May Be Better with Large Data

LSTM's extra parameters may pay off on very large datasets, where its "over-engineering" can be fully exploited by the optimizer.

2. Some Tasks Need Fine-Grained Gate Control

On counting tasks (e.g., copying binary sequences), GRU underperforms LSTM due to the missing output gate. The Schmidhuber group noted GRU "can neither learn to count" — tasks requiring precise gate control.

3. Displaced by Transformers

In NLP, GRU and LSTM have largely been replaced by Transformers, whose self-attention provides direct "any-to-any" connections without gates to carry gradients.

But GRU remains active in:

• Time-series forecasting (finance, energy, weather)
• Small sequence models (resource-constrained environments)
• Hybrid RNN-Transformer architectures (an inspiration behind RWKV and Mamba)

8. Modern Relevance: Why GRU Matters in 2026

1. Why Learn GRU Today?

Transformers aren't a panacea. For:

• Online learning (streaming data, can't wait for full sequences)
• Long-sequence inference (memory limits; O(n²) attention infeasible)
• Small devices (phones, IoT — every saved parameter counts)

the RNN family's efficiency advantage persists, and GRU is one of its most elegant members.

2. Mamba's Inspiration

The Mamba architecture (2023–2024, S4 + selection mechanism) can be seen as continuing GRU's spirit:

• Selective state spaces: gating-like, but more efficient
• Hardware-aware design: GRU-style focus on computational efficiency
• Linear complexity: addressing Transformer's O(n²) bottleneck

GRU's design philosophy — "simplify as much as possible while maintaining effectiveness" — directly influenced this new generation of sequence models.

3. Practical Advice for Developers

| Scenario | Recommended Architecture | Rationale | |----------|--------------------------|-----------| | Small data + sequence task | GRU | Fewer parameters, less overfitting | | Large data + complex sequences | LSTM | More parameters, more capacity | | Resource-constrained environments | GRU | Higher computational efficiency | | Modern NLP tasks | Transformer | Parallel training, SOTA results | | Very long online sequences | Mamba/RWKV | Linear complexity, RNN-style streaming advantage |

9. Conclusion: What GRU Teaches Us

GRU is not just an architecture — it's a design principle:

> "Great design isn't about maximizing complexity; it's about finding just the right amount of complexity."

• LSTM proved gating could save the RNN
• GRU proved you don't need that many gates to achieve the same effect
• MGU proved a single gate can suffice

In deep learning, identifying the mechanism that truly matters (gating) and stripping away all unnecessary ornamentation often yields better results.

GRU's two gates — reset and update — are not a degraded LSTM but a refined one. It kept the essence of gating, removed the redundancy, and ultimately demonstrated:

Simplicity itself is a kind of power.

---

References

• Cho et al. (2014). "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation." arXiv:1406.1078
• Chung et al. (2014). "Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling." arXiv:1412.3555
• Dey et al. (2017). "Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks." arXiv:1701.05923
• Schmidhuber et al., Annotated History of Modern AI and Deep Learning