The Elephant in Your Pocket: How Gemma 4 Brings AI from the Cloud to Your Jeans

> Structured English summary of a Chinese-language forum post on zhichai.net. The original is written in a Feynman-inspired, conversational style; key technical claims are preserved below.

Key points

• 2 million downloads in week one: The author frames Gemma 4's launch as a shift in the AI narrative — from "bigger is better" cloud models (GPT-4, Claude Opus) to models that run entirely on personal devices.
• Demos that shouldn't work: Community members reportedly ran Gemma 4 at ~40 tokens/second on an iPhone 17 Pro (about human speaking speed), on a Raspberry Pi 5, and even a TinyStories-derived model on a 1998 iMac G3 with only 32MB of RAM.

How the "elephant fits in the fridge"

1. Parameters are connections, not knowledge

The author argues parameters are "knobs" — potential connections — not stored wisdom, and that they must all fit in memory during inference, which is why large models historically lived in the cloud.

• Per-Layer Embeddings: Of Gemma 4's ~5.1B parameters (in the discussed variant), roughly 2.8B are static, context-independent embedding weights. These can live in flash storage and be paged in on demand, leaving only ~2.3B "active" parameters requiring resident memory. As the author puts it: you don't shove the whole elephant in at once — one leg at a time.

2. MoE: not every expert clocks in

Using a hospital analogy (26 departments, but only the relevant one sees you), the post explains Gemma 4's Mixture-of-Experts design:

• The 26B MoE variant has 26 expert parameter groups; per input, only the ~2 most relevant experts activate (~2.6B active parameters of 26B total).
• Community comparison cited: a code audit took 30–50 minutes with a 31B dense model vs. 2 minutes with the 26B MoE version — roughly an order of magnitude faster.
• The author stresses MoE is not "cutting corners" but using parameters more intelligently; idle experts remain fully capable in their domains.

Speed and economics

• 40 tokens/second exceeds the latency threshold where generation feels continuous and conversational; the Raspberry Pi 5 is slower but usable — like "your grocery car lapping an F1 track."
• FoodTruck Bench (agentic task evaluation): Gemma 4's 31B version ranked third behind GPT-4 and Claude 3 Opus, at ~$0.20 per run, implying an ROI above 1100%. Self-hosting the open-weight model could push marginal costs near zero.

Why it matters (and caveats)

The author argues local AI redistributes power away from a "feudal" cloud model:

• Privacy — data never leaves the device; sensitive text can be processed fully offline.
• Reliability — no dependence on remote datacenters that can crash or change terms.
• Accessibility — billions of people without fast connectivity can still benefit.

Caveats the author raises:

• Local deployment still requires technical knowledge (MLX, quantization, memory mapping) and capable hardware.
• Local models remain "good enough," not "the best" — top cloud models still win on the hardest tasks.
• This is a story of more choices, not local replacing cloud.

Closing thought

The author admits not knowing whether Gemma 4 marks a turning point, or Google's true motives for open-sourcing it — but notes that yesterday's impossibilities (neural nets on phones, billion-parameter models on PCs) keep falling. "The elephant can fit in the fridge. And once we're used to it, we'll ask: what else fits? Maybe a bigger elephant."

---

*Translated and summarized from the original Chinese post.*