Memory as a Multi-Story Apartment: How Gemma 4 Makes Large Models Both 'Fat' and 'Thin'

*Translated and edited from a Chinese forum post on zhichai.net. The original uses Feynman-style analogies; technical details are based on the Gemma 4 public technical report.*

1. A Strange Math Problem

Consider this puzzle:

> An AI model has 5.1 billion parameters in total, but when answering a question, only 2.3 billion of them actually participate in computation. > > Where did the remaining 2.8 billion go?

The answer: those 2.8 billion parameters are a dictionary — like a thick English-Chinese dictionary on your desk, except this one contains 2.8 billion numbers.

What Gemma 4 does is clever: it turns this dictionary from something you must carry everywhere into something kept on a bookshelf, consulted when needed.

2. One ID Card for the Whole Journey

How does an AI 'know' a word? Like humans converting a word into a web of associations, a Transformer uses a giant table (the embedding table): each word maps to a vector — a numeric 'ID card'.

In a traditional Transformer:

1. Layer 1: look up the word, get its embedding (the ID card) 2. Layers 2 through N: carry the ID card upward, each layer annotating and transforming it 3. Final layer: generate an answer based on the fully processed card

The hidden problem is information decay. After dozens of layers of annotations, the original details on the card get drowned out. When the model processes a pronoun like 'it' referring to a phrase 50 words back ('quantum entanglement... Einstein called it "spooky action at a distance"...'), the card may no longer clearly remember what 'it' points to. Long-range dependencies are hard to capture.

3. Re-Introducing Yourself on Every Floor

Gemma 4's Per-Layer Embeddings (PLE) asks a simple but bold question: why not consult the dictionary at every layer?

Instead of carrying one ID card to the top, the token re-introduces itself at each floor:

• Layer 1: 'I am "quantum", a physics concept.'
• Layer 5: 'I am "quantum", currently in a paragraph about the microscopic world.'
• Layer 20: 'I am "quantum", the subject of this sentence — remember my position.'

Each layer re-queries the embedding table to get the freshest, context-aware representation. You don't carry a heavy dictionary around; every floor has a librarian who fetches the word on demand.

4. 2.8 Billion Parameters on the Bookshelf

Back to the math: why 5.1B parameters but only 2.3B active?

Because Gemma 4's PLE embedding tables are static and position-agnostic:

• They are effectively read-only after training — the numbers are fixed.
• Embeddings do not depend on a word's position in the sentence.

This means the embedding tables can be stored on disk or flash memory rather than in precious VRAM. VRAM is your desk workspace; disk/flash is the bookshelf behind you — bigger, slightly slower to access, but ample.

The result:

• The model is fat: 5.1 billion parameters of knowledge capacity.
• The model is also thin: only 2.3 billion parameters live in VRAM during inference.

5. The Power of Refreshed Understanding

Reading a long novel the traditional way means relying on first impressions of characters from chapter one, and by chapter 50 you may have mixed up their motivations. PLE is like re-reading the character bios every chapter — you always have an updated understanding.

For AI, this means long-range dependencies are no longer a fundamental problem: information is refreshed at every layer instead of decaying between them.

6. Why Wasn't This Done Before?

Deep learning carried an implicit assumption: more parameters means more compute resources and slower speed — like saying the more knowledge a person has, the more luggage they must carry.

Gemma 4 breaks that intuition: you can hold vast knowledge (2.8B parameters) without carrying it (store it on flash). This required rethinking the architecture: per-layer embedding tables, static (read-only) tables so they can live on flash, position-independent embeddings for reuse across layers, and efficient memory management so flash reads don't become a bottleneck.

7. What This Means for You

• Smarter phones and devices: lower VRAM requirements make it easier to deploy large language models on phones, tablets, and embedded devices — even offline.
• Better understanding of 'long talk': PLE helps with long novels, complex legal contracts, and long-range dependencies in multi-turn conversations.
• Capability grows without proportional hardware cost: models can grow larger while runtime footprint grows much more slowly — potentially changing the industry's rules.

8. A Feynman-Style Summary

If you remember only one thing:

Gemma 4 lets the AI re-consult the dictionary at every layer, instead of carrying a single ID card all the way up.

The payoff:

• More accurate understanding (no information decay)
• Better long-range memory
• Lower VRAM usage (the big dictionary stays on the shelf)
• 5.1B parameters of knowledge with 2.3B-parameter efficiency

Simple and easy are not the same thing. Turning a one-sentence idea into reality required the courage to question 'more parameters = slower' and the design skill to make a fat model run thin. That is the charm of engineering: sometimes it doesn't invent new things — it just lets you see old things in a completely new way.

References

• Gemma 4 Technical Report, Google DeepMind, 2025. Commit hash: 2c47ab1
• Vaswani et al. 'Attention Is All You Need.' NeurIPS 2017.
• Technical details are based on the Gemma 4 public technical report; all analogies and explanations are the original author's.