The Multi-Story Memory Apartment: How Gemma 4 Makes Large Models Both 'Fat' and 'Thin'

1. A Strange Math Problem

Consider this question:

> An AI model has 5.1 billion parameters in total, but only 2.3 billion are actually active during inference. Where did the other 2.8 billion go?

The answer: those 2.8 billion parameters are a dictionary — the embedding table. Gemma 4's clever move was turning this dictionary from something the model must carry everywhere into something stored on a bookshelf, consulted only when needed.

2. One ID Card Walks Through the Whole Building

Traditional Transformers work like this:

1. Layer 1: Look up the token in the embedding table to get its "ID card" (a vector) 2. Layers 2 through N: Carry this ID card upward, each layer annotating it 3. Finally: Generate output based on the annotated card

The problem: information decays. Like an ID card scribbled on across dozens of floors, early information gets buried. When processing a pronoun like "it" referring back to "quantum entanglement" many words earlier, the model often can't clearly remember the connection. This is the classic Transformer bottleneck: information fades with depth, and long-range dependencies are hard to capture.

3. Re-Introducing Yourself at Every Floor

Gemma 4's idea — Per-Layer Embeddings (PLE) — is deceptively simple:

Why not look up the dictionary at every layer?

Instead of carrying one ID card to the top, the model re-introduces the token at each floor:

• Layer 1: "I'm 'quantum', a physics concept."
• Layer 5: "I'm still 'quantum', now in a paragraph about the microscopic world."
• Layer 20: "Quantum here — I'm the subject of this sentence, remember my position."

At each layer, the model re-queries an embedding table to get a fresh, contextually updated representation of the token. You don't carry a heavy dictionary around; each floor has a librarian who fetches the word for you.

4. 2.8 Billion Parameters Live on a Bookshelf

Why do 5.1 billion parameters need only 2.3 billion in memory? Because Gemma 4's PLE embedding tables are static and position-agnostic:

• Once training is done, the embeddings are frozen (read-only)
• The embeddings don't depend on a token's position in the sequence

This means the embedding tables can live on disk or flash storage rather than VRAM. VRAM is like your desk — limited workspace; disk is the bookshelf behind you — much bigger, slightly slower, but sufficient.

The result:

• The model is 'fat': 5.1 billion parameters of knowledge capacity
• The model is 'thin': only 2.3 billion parameters in VRAM during inference

5. The Power of Refreshed Understanding

Traditional reading is like remembering characters only from their first chapter appearance and inferring everything after from that first impression. PLE is like re-reading the character bios every chapter — your understanding stays current.

For AI, this means long-range dependencies are no longer a problem. When the model hits "it" referring to "quantum entanglement" 50 words back, PLE lets every layer re-access the freshest, most relevant meaning. Information is refreshed at each layer rather than decaying.

6. Why Wasn't This Thought of Before?

Deep learning carried an implicit assumption: more parameters means more compute resources and slower speed — like saying the more knowledge a person has, the more luggage they must carry.

Gemma 4 breaks this intuition: you can hold vast knowledge (2.8B parameters) without carrying it (keep it in flash memory). This required redesigning the whole architecture:

• Every layer has its own embedding table
• Tables must be static to live in flash
• Embeddings must be position-agnostic to be reusable across layers
• Efficient memory management ensures flash reads don't become a bottleneck

7. What Does This Mean for You?

• Smarter phones: running large models with less VRAM makes on-device, offline AI on phones, tablets, and embedded devices more feasible.
• Better understanding of long text: novels, legal contracts, and multi-turn conversations with long-range dependencies become easier to handle.
• Capability grows without proportional hardware cost: models can get bigger without runtime requirements scaling in lockstep — potentially a game changer for the industry.

8. A Feynman-Style Summary

If you remember one thing, remember this:

Gemma 4 'cheats' cleverly — it re-consults the dictionary at every layer instead of carrying one ID card to the end.

This lets it:

• Understand more accurately (no information decay)
• Remember longer (long-range dependencies solved)
• Use less VRAM (the big dictionary sits on a shelf)
• Be more capable (5.1B parameters of knowledge at 2.3B-parameter efficiency)

The best ideas often look simple enough that you wonder why you didn't think of them. But simple and easy are different things: realizing this one required questioning the common sense that "more parameters means slower," and redesigning the entire architecture.

References

• Gemma 4 Technical Report, Google DeepMind, 2025. Commit hash: 2c47ab1
• Vaswani et al. "Attention Is All You Need." NeurIPS 2017.
• Technical details are based on the public Gemma 4 technical report; all analogies and explanations are the author's original work.