oMLX: Tiered KV Cache Brings Local 70B LLM Cold Start from 47s to 5s on Apple Silicon

Forum topic · ✨步子哥 · 2026-08-17

Summary

oMLX is a Python-based LLM inference server optimized for Apple Silicon that compresses Time To First Token (TTFT) from 30-90 seconds to under 5 seconds for 70B models by extending KV cache into a hot/cold tiered hierarchy. Hot KV blocks stay in unified memory; cold KV blocks are serialized to SSD, eliminating redundant recomputation when contexts or sessions change. The server adds continuous batching, a macOS menu-bar manager, and custom Metal kernels that accelerate GLM-5.2 fused DSA prefill by roughly 30x (845 vs 29 tok/s on M3 Ultra). Unlike Ollama, LM Studio, and MLX server, which discard KV cache at request end, oMLX is the first local inference engine to treat SSD as a KV cache layer, mirroring operating-system virtual-memory design. Trade-offs include Apple Silicon lock-in, macOS 15.0+ requirement, and a full Xcode toolchain for custom kernels.

Overview

oMLX is a Python-based LLM inference server optimized for Apple Silicon, managed from the macOS menu bar. Its headline result: TTFT (Time To First Token) for a 70B model drops from 30-90 seconds to under 5 seconds. The trick is a tiered KV cache — hot data in unified memory, cold data on SSD.

The Problem: Cold Start vs. Cached

LLM inference stores Key/Value vectors for every previous token in a KV cache. For a 70B model with 32K context, this cache reaches ~8 GB — larger than the weights themselves. Traditional servers keep KV cache only in memory and discard it when a request ends. The next request recomputes everything from scratch, which is why "first hello" can take 47 seconds while "second hello" returns instantly.

oMLX's insight: KV cache is serializable. If it can be computed, it can be persisted. Save it to SSD; load it on demand; skip recomputation.

Three Key Design Decisions

1. Context switching preserves cache

When a user changes context (e.g., switching from "discussing Rust" to "discussing Python"), traditional servers invalidate all prior KV cache. oMLX evicts the old context's KV to SSD, computes the new context in memory, and reloads from SSD if the user switches back. In coding workflows with tools like Claude Code, this eliminates repeated recomputation when jumping between files.

2. Cross-request cache reuse

KV cache can be shared across requests with similar system prompts. This extends vLLM-style PagedAttention thinking down to the SSD tier.

3. Native custom Metal kernels

For GLM-5.2, MiniMax M3, Qwen3.5, and similar models, oMLX ships native Metal kernels. README benchmarks report GLM-5.2 fused DSA prefill at 845 tok/s versus 29 tok/s on the generic path — about 30x faster on M3 Ultra, achieved by skipping MLX abstraction overhead.

Comparison with Other Local Inference Engines

| Engine | Platform | KV cache policy | TTFT (70B cold) | |--------|----------|-----------------|-----------------| | Ollama | Cross-platform | Memory, discard on end | 30-90 s | | LM Studio | Cross-platform | Memory, discard on end | 30-90 s | | MLX server | Apple | Memory, discard on end | 30-90 s | | vLLM | CUDA | PagedAttention (in-memory) | N/A (no Mac) | | oMLX | Apple Silicon | Memory + SSD tiers | <5 s |

Apple Silicon's unified memory and ~7 GB/s SSD bandwidth make SSD-resident KV cache practical: loading 8 GB takes ~1 second. On discrete-GPU CUDA systems, ~4 GB/s PCIe bandwidth and lack of direct GPU-to-SSD access break the same design.

Why the Menu-Bar Manager Matters

The macOS menu-bar interface (one-click model switching, monitoring, shutdown) signals a positioning shift: local LLM inference should behave like a system service (Time Machine, Spotlight) rather than a CLI tool you launch on demand. This matches how users actually want to consume local LLMs.

Honest Trade-offs

Platform lock-in: Apple Silicon only — MLX, Metal, and unified memory are all Apple-specific.
OS requirement: macOS 15.0 (Sequoia) or later.
Toolchain cost: Native kernels require a full Xcode install (6 GB+), then brew install --HEAD --with-custom-kernel.
Single-machine assumption: Experimental Multi-Mac RDMA inference (Ring/Thunderbolt) is source-build only, not default.

Conceptual Framing

oMLX applies a 1960s operating-system principle — hierarchical storage with different cost/performance tiers — to 2026 LLM inference. The cache hierarchy (CPU registers → RAM → SSD → HDD → network) is extended with a new tier for KV cache, recognizing it as "serializable intermediate state" rather than "disposable resource." The same logic supports Agent memory externalization: volatile state becomes persistent, black-box state becomes inspectable.

Key Points

TTFT cut from 30-90 s to <5 s for 70B models on Apple Silicon.
KV cache treated as serializable, persistent data, not disposable.
Two-tier cache: unified memory (hot) + SSD (cold).
Continuous batching, menu-bar management, native Metal kernels.
GLM-5.2 fused DSA prefill: ~30x speedup with custom kernels (845 vs 29 tok/s, M3 Ultra).
Only local inference server with SSD-tiered KV cache; vLLM-like PagedAttention is memory-only.
Apple Silicon lock-in, macOS 15.0+, full Xcode required for custom kernels.

---

Project: https://github.com/jundot/omlx Website: https://omlx.ai Language: Python (Metal kernels in Metal Shading Language) Audience: Apple Silicon users, local 70B LLM developers, Claude Code / Codex local-deployment users.