Google Gemma 4 was released just three months ago (April 2, 2026), and teams are already putting it to novel use. On July 23, the open-source inference framework Cactus launched Cactus Hybrid, built on Gemma 4 E2B: a confidence probe embedded in the model checkpoint gives the model a structured 0–1 confidence score for every generated answer.
What's the score for? It directly determines "can this question be answered locally?" High score → answered entirely on-device (zero network, zero latency, zero cost; voice recordings and image queries never leave the device). Low score → automatically handed off to a larger cloud model (Claude, GPT, Gemini — your choice). In other words, the model knows its own ceiling and offloads what exceeds it to the cloud.
What surprised me most isn't the binary "local + cloud" routing — that's already a standard move for cloud-side products like Cursor Router. The interesting part: the confidence probe was trained with zero audio training data, yet reaches 0.79–0.88 AUROC on four public audio benchmarks, far better than the token-entropy baseline for the same task (mean 0.549). A score of 0.549 means that using "uncertainty in the model's own output" to decide whether to offload a request is basically a coin flip.
Product specifics:
- 80/20 split: 80% of ordinary requests (voice transcription, image understanding, chat) stay on phone, Quest, or Vision Pro with latency < 0.3s, audio and images never leave the device; 20% of complex requests (writing database migrations, debugging distributed systems) go to Claude and similar models
- Cross-platform: Cactus supports macOS / iOS / Android / Linux, with React Native / Flutter / Swift / Kotlin / Python / Rust / C++ all supported
- Hardware: INT4 quantization, ARM optimization, memory kept under 4GB; E2B runs at a stable 59 tok/s on an RTX 4060 (my own 8GB-VRAM machine)
- License: MIT open source, commercial use and modification allowed
1. It hits enterprise AI compliance pain points head-on. Healthcare, legal, and finance scenarios often hard-require that audio and documents never touch the cloud. Cactus Hybrid makes "local" and "capable" no longer an either-or choice. 2. A shift from "model weights" to "model self-assessment capability". Cursor Router does this in the cloud; Cactus does it on-device, and replaces the entropy baseline with a probe — a big engineering difference. 3. Gemma 4's small size (PLE compresses 5.1B to 2.3B effective) finally gets a usable product form. 2.3B effective parameters + 262K vocabulary + 128K context, running on a phone — flagship-level in 2025, and by 2026 Q3 it powers a workable hybrid product.
My prediction for what's next: the confidence probe concept will flow back from Cactus's edge side into cloud router design. Cursor Router currently uses heuristics like "if the cache works, use it; otherwise go to a bigger model" — next it will likely replace those with a Calibrated Confidence Probe, letting the model directly score its own judgments as a routing signal. On 07-23 we discussed how Cursor Router's cache-aware routing cut Fable 5 costs by 60%; if this approach connects edge and cloud, the AI inference cost curve gets pressed down again in 2026 H2.
Link: https://github.com/cactus-compute/cactus-hybrid
Side note: Gemma 4 is Apache 2.0 open source (a first for Google's Gemma series), and Cactus Hybrid adds an MIT license layer on top.