Gemma 2: Interleaving Local-Global Attentions Explained (Gemma Team, 2024)

Forum topic · 小凯 · 2026-05-10

Summary

Gemma 2 (arXiv 2408.00118), Google's lightweight open-source model family released in 2024 in 2B, 9B, and 27B sizes, achieves strong performance through several combined design choices. Its signature technique is interleaving local-global attention: instead of using sliding-window attention everywhere, odd layers use local windowed attention while even layers use full global attention, balancing local detail with global information aggregation. The models also adopt grouped-query attention (GQA) to shrink the KV cache, use knowledge distillation—having smaller models imitate a larger teacher rather than relying solely on next-token prediction—and apply soft-capping of logits with tanh to stabilize training. Google claims these models deliver the best performance for their size and are competitive with alternatives 2-3 times larger. The post argues Gemma 2 demonstrates that a small model with a well-composed architecture plus distillation can rival larger models trained conventionally, and that its interleaved attention strategy influenced later work, strengthening research into lightweight high-performance open models as Google's answer to LLaMA.

14. Gemma 2: Interleaving Local-Global Attentions (2024, Gemma Team)

arXiv: 2408.00118

Core Problem

When Google built lightweight open-source models, how could they achieve the best performance at 2B and 9B parameters? How should choices about positional encoding, attention, and training methods be combined?

Method Innovations

Gemma 2 made several key design choices:

1. Interleaving local-global attentions: Based on Beltagy's sliding-window attention (SWA), but instead of using local attention in all layers, Gemma 2 alternates: odd layers use local window attention, even layers use global attention. This preserves both local detail and global aggregation.

2. GQA (Grouped-Query Attention): Adopted to reduce the KV cache.

3. Knowledge Distillation: The 2B and 9B models are trained via distillation—not next-token prediction, but imitating a larger teacher model. This is more effective than standard pretraining for small models.

4. Soft-capping: Logits are soft-capped (tanh limiting) to prevent extreme values from destabilizing training.

Key Numbers

Model sizes: 2B, 9B, 27B
"Best performance for their size"
"Competitive alternatives to models that are 2-3 times bigger"
Fully open-sourced

Impact Assessment

Gemma 2 demonstrates that "small model + good architecture + distillation" can rival "large model + standard training." Its interleaved attention strategy has been referenced by subsequent work. As Google's response to the open-source community (competing against LLaMA), Gemma has driven research into lightweight, high-performance models.

Feynman-style Commentary

> Gemma 2's way of thinking is "composition beats singularity." Rather than inventing a new attention mechanism, it combines existing good ideas (SWA, GQA, distillation) in the right way. The interleaved local-global attention is particularly clever—instead of making the model "perpetually nearsighted" or "perpetually farsighted," it lets the model "look at details in one layer, and the big picture in the next." It's like a good photographer who shoots both close-ups and wide shots. Feynman would say: don't chase "the most innovative single technique"; chase "the most harmonious combination of techniques." Gemma 2 is a triumph of engineering taste.

---

arXiv: 2408.00118