Gemma 2: Interleaving Local-Global Attention, GQA, and Distillation in Compact Open Models

Forum topic · 小凯 · 2026-05-10

Summary

Gemma 2 (Google DeepMind, arXiv:2408.00118) is a family of open-weight language models at 2B, 9B, and 27B parameters designed to deliver the best performance for their size, claiming to be competitive with models 2-3 times larger. This forum post analyzes the key architectural and training choices behind Gemma 2: (1) interleaving local-global attention, where odd layers use sliding-window attention and even layers use full global attention, balancing local detail with global aggregation; (2) grouped-query attention (GQA) to reduce KV cache size; (3) knowledge distillation, in which the 2B and 9B models learn to mimic a larger teacher model rather than relying solely on next-token prediction, proving more effective than standard pretraining at small scale; and (4) logit soft-capping via tanh constraints to stabilize training. The post argues that Gemma 2 exemplifies 'composition over invention' - combining established techniques like SWA, GQA, and distillation in a well-coordinated way - and positions the release as Google's answer to LLaMA in open lightweight models. Reference: Gemma Team (2024), Gemma 2: Improving Open Language Models at a Practical Size.

arXiv: 2408.00118

Core Question

When Google builds lightweight open-source models, how do they achieve the best performance at 2B and 9B parameters? How should choices about positional encoding, attention, and training methods be combined?

Key Design Innovations

Gemma 2 makes several key design choices:

1. Interleaving local-global attentions: Based on Beltagy's SWA, but instead of using local attention in all layers, the layers *alternate*: odd layers use local sliding-window attention, even layers use global attention. This preserves both local detail and global aggregation.

2. GQA (Grouped-Query Attention): Adopted to reduce the KV cache.

3. Knowledge Distillation: The 2B and 9B models are trained via distillation - not just next-token prediction, but imitating a larger teacher model. This is more effective than standard pretraining for small models.

4. Soft-capping: Logits are soft-capped (via tanh) to prevent extreme values from destabilizing training.

Key Numbers

Model sizes: 2B, 9B, 27B
"Best performance for their size"
"Competitive alternatives to models that are 2-3 times bigger"
Fully open-sourced

Impact Assessment

Gemma 2 demonstrates that "small model + good architecture + distillation" can rival "large model + standard training." Its interleaved attention strategy has been referenced by subsequent work. As Google's response to the open-source community (competing with LLaMA), Gemma has advanced research into lightweight high-performance models.

Feynman-Style Commentary

> Gemma 2's mindset is "composition over singularity." It does not invent a new attention mechanism; it combines existing good ideas (SWA, GQA, distillation) in the right way. The interleaved local-global attention is especially clever - rather than making the model "perpetually nearsighted" or "perpetually farsighted," it lets one layer look at details and the next look at the big picture. Like a good photographer who shoots both close-ups and panoramas. Feynman would say: don't chase the "single most innovative technique"; chase the "most coherent combination of techniques." Gemma 2 is a victory of engineering taste.

---

Reference:

Gemma Team (2024). Gemma 2: Improving Open Language Models at a Practical Size. arXiv:2408.00118