Google Gemma 4 shipped only three months ago (April 2, 2026), and teams are already finding new ways to use it. On July 23, the open-source inference framework Cactus released Cactus Hybrid, based on Gemma 4 E2B: a confidence probe embedded directly in the model checkpoint makes the model emit a structured 0–1 confidence score for every generated answer.
What does this number do? It directly decides whether a request can be answered locally. A high score means the answer is produced entirely on-device (zero network, zero latency, zero cost — voice recordings and image queries never leave the phone). A low score means the request is automatically handed off to a larger cloud model (Claude, GPT, Gemini — your choice). In other words, the model knows its own ceiling and routes anything beyond it to the cloud.
The most surprising part isn't the binary "local + cloud" routing — that's already standard practice in cloud-side products like Cursor Router. The interesting detail here: the confidence probe was trained with zero audio training data, yet reaches 0.79–0.88 AUROC on four public audio benchmarks, far better than the token-entropy baseline for the same task (mean 0.549). That 0.549 means that using the model's own output uncertainty to decide whether to offload a request is basically a coin flip.
Product specifics:
- 80/20 split: 80% of routine requests (voice transcription, image understanding, chat) stay on phones, Quest, or Vision Pro with < 0.3s latency; audio and images never leave the device. The 20% of complex requests (database migrations, distributed-systems debugging) go to Claude and similar models.
- Cross-platform: macOS / iOS / Android / Linux, with React Native / Flutter / Swift / Kotlin / Python / Rust / C++ all supported.
- Hardware adaptation: INT4 quantization, ARM optimization, memory kept under 4GB. E2B runs at a stable 59 tok/s on an RTX 4060 (my 8GB-VRAM machine).
- License: MIT open source — commercial use and modification allowed.
1. It directly targets enterprise AI compliance pain points. In healthcare, legal, and finance, keeping audio and documents off the cloud is a hard requirement. Cactus Hybrid means "local" and "capable" are no longer an either/or choice. 2. The shift from "model weights" to "model self-assessment." Cursor Router, discussed a few weeks ago, does this in the cloud; Cactus does it on-device and replaces entropy baselines with a probe — the engineering details differ significantly. 3. Gemma 4's small size (PLE compresses 5.1B to 2.3B effective) finally has a usable product shape. 2.3B effective + 262K vocabulary + 128K context, running on a phone — flagship-class in 2025, and by 2026 Q3 it can power a working hybrid product.
My prediction for what's next: the confidence probe concept will flow back from Cactus's on-device side into cloud router design. Cursor Router currently uses heuristics like "try it, and if it doesn't work escalate to a bigger model"; it's likely to replace these with calibrated confidence probes — the model scoring its own judgments as a routing signal. On 07-23 we discussed how Cursor Router's cache-aware routing cut Fable 5 costs by 60%; if this idea is unified across on-device and cloud, AI inference cost curves could get compressed again in 2026 H2.
Link: https://github.com/cactus-compute/cactus-hybrid
Side note: Gemma 4 is Apache 2.0 (a first for Google's Gemma series), and Cactus Hybrid adds an MIT license layer on top.