Gemma 4: How Per-Layer Embeddings Make Large Models Efficient on Phones and Raspberry Pi

Forum topic · 小凯 · 2026-04-11

Summary

This forum post analyzes Gemma 4's Per-Layer Embeddings architecture, which separates static embedding parameters from the active compute core. In the E2B variant, of 5.1 billion total parameters, 2.8 billion are static vocabulary embeddings that can reside on disk, leaving roughly 2.3 billion parameters resident in VRAM. The post reports real-world performance figures: about 40 tokens/second on an iPhone 17 Pro via the MLX framework, and 41.76 tokens/second on a Raspberry Pi 5 with an M.2 SSD and quantization. On the FoodTruck Bench agentic evaluation, Gemma 4 31B ranked third at roughly $0.20 per run, behind Opus 4.6 and GPT-5.2, with ROI exceeding 1100%. The author argues Gemma 4 signals a design philosophy shift from parameter scaling toward efficiency: decoupling static from dynamic components, storage from compute, and ownership from usage. Examples cited include the 1.3M-parameter SauerkrautLM-Doom beating cloud models in VizDoom control and the 0.6B Falcon Perception outperforming SAM 3 in on-device image segmentation. Source commit: 2c47ab1.

Background

The post opens with an analogy: traditional large models are like carrying an entire library in your backpack—tens of billions of parameters fully loaded into VRAM. Gemma 4 introduces Per-Layer Embeddings as a smarter solution.

A Library You Can Detach

Gemma 4's E2B variant has 5.1 billion parameters total.
Of these, 2.8 billion are static embeddings—tied only to the vocabulary, independent of position.
Since this part is static, Gemma 4 keeps it on disk and looks it up on demand, leaving only about 2.3 billion parameters resident in VRAM.

Analogy from the post: leave the dictionary at the hotel and carry only a portable guide; consult the dictionary when needed.

The Art of Speed

iPhone 17 Pro: ~40 tokens/second via the MLX framework.
Raspberry Pi 5: community tests show 41.76 tokens/second with an M.2 SSD and sensible quantization.

Cost-Effectiveness

On the FoodTruck Bench agentic evaluation:

Gemma 4 31B placed third at roughly $0.20 per run, behind Opus 4.6 and GPT-5.2.
ROI exceeded 1100%.

The takeaway: more parameters ≠ better performance; because of Per-Layer Embeddings, the actual compute burden of the 31B model is far smaller than comparable models.

Lessons for the Industry

The post argues Gemma 4 represents a new design philosophy—efficiency over parameter bloat—supported by examples:

SauerkrautLM-Doom (1.3M parameters) can beat cloud LLMs on VizDoom control tasks.
Falcon Perception (0.6B) runs image segmentation on Mac via MLX, outperforming SAM 3.

Formula proposed: scenario fit + engineering optimization > blind scaling.

Per-Layer Embeddings open a path for large-embedding + small-operator models: decoupling static from dynamic parts, storage from compute, ownership from usage.

Closing Thought

AI democratization may not mean everyone buys a top-tier GPU—it means ordinary phones, Raspberry Pis, even future smartwatches running capable AI. When large models learn to slim down, they get closer to everyone.

> The highest form of technology is when users no longer feel its presence.

---

*Source commit: 2c47ab1*