An Elephant in Your Pocket: How Gemma 4 Brings AI from the Cloud to Your Jeans

*Translated and edited from a zhichai.net forum post (originally in Chinese).*

The strange phenomenon

In the first week after Gemma 4's release, the model hit 2 million downloads — not views or likes, but people actually installing it on their own devices. People ran it at ~40 tokens/second on an iPhone 17 Pro, on a cheap Raspberry Pi 5, and one enthusiast even got a (tiny) model running on a 1998 iMac G3 with just 32MB of RAM.

This breaks the recent narrative that AI must be "bigger is better": cloud-hosted, trillion-parameter models accessed only through APIs.

Key points

• Parameters are weight, not wisdom. Gemma 4's ~5.1B-parameter configuration splits its "knobs" into two piles: ~2.3B "active" parameters kept in memory, and ~2.8B static, context-independent parameters via Per-Layer Embeddings, which are moved out of expensive memory into flash storage and fetched only when needed. Analogy: you don't need the whole elephant in the fridge at once — just the leg you're currently using.
• MoE: not every expert clocks in. The 26B Mixture-of-Experts variant has 26 expert parameter subsets, but per query it activates only the ~2 most relevant ones, so only about 2.6B parameters are actually used per token. Community comparisons: a code audit took 30–50 minutes on the 31B dense version vs. 2 minutes on the 26B MoE version — an order-of-magnitude speedup. MoE is not "cutting corners"; it's using parameters more intelligently.
• Speed changes experience. At 40 tokens/second on iPhone, generation exceeds the latency-perception threshold, making interaction feel continuous. On Raspberry Pi 5 it's slower but usable — previously such devices simply couldn't run LLMs at all.
• Economics. On the FoodTruck Bench agentic evaluation, Gemma 4's 31B version placed third behind GPT-4 and Claude 3 Opus, at $0.20 per call — a claimed ROI above 1100%. Self-hosting the open-weights model can reduce marginal cost further.
• TinyStories on a 1998 iMac G3. A heavily compressed TinyStories model runs on 32MB of RAM. Beyond geek spectacle, it signals AI becoming embedded in any powered device: fridges, door locks, hearing aids.

What actually changes

The author argues the real shift is in the distribution of power. Cloud AI is "feudal": providers own compute and models, set prices and rules, and can cut access anytime. Gemma 4 is a democratizing counterweight — downloadable, modifiable, fine-tunable. Practical benefits of local AI:

• Privacy: data never leaves the device.
• Reliability: no dependence on cloud uptime.
• Accessibility: works without fast, stable internet.

Caveats

• Local deployment still requires technical knowledge (MLX, quantization, memory mapping).
• It isn't free — capable hardware costs money.
• Local models remain "good enough," not the best; GPT-4 / Claude 3 Opus still lead on complex tasks.

So this is not "local replaces cloud" — it's "more choices." A healthy ecosystem means diversity, not monopoly.

Outlook

The author admits not knowing whether Gemma 4 marks the tipping point from cloud era to local era, or Google's true motives for open-sourcing. But history shows "impossible" things — phones running neural networks, billion-parameter models on PCs, open models nearing closed ones — keep becoming possible. Two million people have already made their choice: they downloaded it and started playing, out of pure curiosity. In the Feynman spirit the author invokes, that's how all good things begin.