Not the model that's slow—it's the cold start. For a 70B model, the KV cache must be loaded from SSD into unified memory, and a 30–90 second TTFT (Time To First Token) is the norm. During that time you just watch the cursor blink and wonder if it's frozen.
The second time you ask "hello", the answer is instant—because the KV cache is already in memory. But switch topics or change the context, and you start over.
oMLX solves exactly this problem: it treats the KV cache as hot data and uses the SSD as a cold tier. TTFT drops from 30–90 seconds to under 5 seconds.
What It Is
oMLX is an LLM inference server optimized for Apple Silicon, written in Python and managed from the macOS menu bar. Core features:
1. Continuous batching: multiple requests are batched dynamically, unlike vLLM which waits for a full batch. 2. Hierarchical KV cache: the hot tier lives in unified memory, the cold tier on SSD. On context switches, old KV isn't discarded—it's written to SSD. 3. Menu-bar management: no command line needed; click the menu-bar icon to switch models, view monitoring, or stop the service.
But the truly interesting part isn't the features themselves—it's the design philosophy: it overturns the assumption that "the KV cache is a one-shot resource."
The KV Cache Problem
Some background. During LLM inference, attention computation for each token requires the Key and Value vectors of all preceding tokens. The cache of these K/V vectors is called the KV cache. For a 70B model with 32K context, the KV cache can reach 8GB—larger than the model weights themselves.
The traditional approach: the KV cache lives only in memory and is discarded when the request ends. The next time the same user comes back, everything is computed from scratch. That's why "the first 'hello' takes 47 seconds"—those 47 seconds aren't spent computing "hello"; they're spent recomputing the KV cache of the entire preceding conversation.
oMLX's breakthrough insight: the KV cache is serializable. If it can be computed, it can be stored. Save it on SSD, load it next time, skip recomputation.
This is the same idea as OS virtual memory: when memory runs short, use disk—slower than memory, but far faster than recomputing. oMLX applies this 1960s design principle to LLM inference in 2026.
Three Key Designs of Hierarchical Caching
1. No cache loss on context switches
Traditional LLM servers: change the context (say, from "discussing Rust" to "discussing Python") and the previous KV cache is entirely invalidated and must be recomputed. oMLX instead writes the old context's KV to SSD while computing the new context's KV in memory. If you switch back later, it loads from SSD—no recomputation.
This is especially useful for coding. In Claude Code you edit file A and ask "how do I call this function"; switch to file B and ask "how do I fix this bug"; then switch back to file A—a traditional server must recompute file A's KV; oMLX just loads it from SSD.
2. Cross-request cache reuse
More aggressive still: KV caches can be reused across different requests. If you and a colleague use the same model with similar system prompts, the system-prompt portion of the KV cache can be shared. This is similar in spirit to vLLM's PagedAttention, but oMLX extends it to the SSD tier.
3. Native custom kernels
For models like GLM-5.2, MiniMax M3, and Qwen3.5, oMLX provides native Metal kernels. The README has a striking number: GLM-5.2's fused DSA prefill with native kernels is 30x faster than the generic path (845 tok/s vs 29 tok/s, M3 Ultra).
That's not a "10% optimization"—it's 30x. The reason: the generic path goes through MLX's abstraction layer, incurring dispatch overhead on every operation; native kernels are written directly as Metal shaders, skipping all intermediate layers. Same idea as "let Rust be the accountant, let Python be the poet"—push the deterministic, performance-critical parts down to the compiled layer.
How It Differs from Other Local Inference Options
| Solution | Platform | KV cache strategy | TTFT (70B cold start) | |------|------|--------------|-------------------| | Ollama | All platforms | In memory, discarded after request | 30–90s | | LM Studio | All platforms | In memory, discarded after request | 30–90s | | MLX server | Apple | In memory, discarded after request | 30–90s | | vLLM | CUDA | PagedAttention (memory paging) | N/A (no Mac support) | | oMLX | Apple Silicon | Memory + SSD tiering | <5s |
The key difference: oMLX is the only local inference server that brings SSD into the KV cache hierarchy. vLLM's PagedAttention pages within memory; oMLX tiers between memory and SSD—the former solves "memory fragmentation", the latter solves "not enough memory".
Apple Silicon's unified memory architecture makes this design particularly sensible: the M3 Ultra has 192GB of unified memory and ~7GB/s SSD reads, so loading an 8GB KV cache from SSD takes barely a second. This doesn't hold on CUDA + discrete GPU architectures—PCIe bandwidth is only ~4GB/s, and the GPU can't directly access the system SSD.
Conceptual Lineage: The Return of Tiered Storage
In the conceptual lineage of "solving the problem at a different layer", oMLX is the thirteenth member:
- Octopuses use RNA editing for "DNA pretraining + inference-time computation"
- SOPHIA routes different states to different output directions
- Euclid-MCP outsources reasoning to Prolog
- ai-memory moves an agent's memory out into the filesystem
- oMLX extends the KV cache from memory to SSD
- Platform lock-in: Apple Silicon only; Linux/Windows users are out of luck. This is a design choice—MLX is Apple's, Metal is Apple's, unified memory is Apple's. But it limits the audience.
- macOS 15.0+ required: Sequoia and above. Older Macs are excluded.
- Custom kernels need Xcode: Homebrew installs the generic path by default; to get the 30x speedup you must install full Xcode (6GB+) and rebuild with
--HEAD --with-custom-kernel. Not a low barrier. - Single-machine assumption: experimental Multi-Mac inference (Ring/Thunderbolt RDMA) is still at the source-build stage, not a default feature.
Shared principle: don't strengthen the same resource—schedule at a different layer. If memory is insufficient, use SSD; if SSD is insufficient, recompute. The key isn't "which is fastest" but "which granularity fits best"—the KV cache's granularity is "serializable tensors", so serialize to SSD; model weights are "read-only large files", so mmap into memory.
This is in the same tradition as operating system design: CPU cache, memory, SSD, HDD, network storage—a five-tier storage hierarchy, each layer with different access patterns and costs. What oMLX does is bring this mature design pattern into the new domain of LLM inference.
An Overlooked Detail: Menu-Bar Management
oMLX's menu-bar design is easy to dismiss as "UI sugar". But it's actually an important engineering signal: local LLM inference should behave like a system service, not a command-line tool.
The Ollama and LM Studio experience is "open a terminal → type a command → wait". The oMLX experience is "a persistent menu-bar icon → one click → switch models / view monitoring / stop the service". It's on the same tier as macOS's Time Machine and Spotlight—a system-level service, not an app.
This positioning difference shapes user behavior: command-line tools are "start when I need them"; system services "always run in the background, ready when I need them". The latter is the right way to use a local LLM—you wouldn't open a terminal and type find / every time you want to search files, so why open a terminal and type ollama run every time you want to ask an LLM?
Honest Assessment
It's not without problems:
Conclusion
oMLX recalls an observation: many technical breakthroughs aren't about being "faster" but about the "right granularity".
The KV cache isn't a "one-shot resource"—it's a "serializable intermediate state". Once you adopt that view, SSD caching is a natural corollary. Going from 47 seconds to 5 seconds isn't about faster chips; it's about expanding the storage hierarchy from "memory only" to "memory + SSD".
In the agent era, this idea carries an extra lesson: an agent's context window is finite, but its working memory can be externalized (what ai-memory does). An agent's KV cache is volatile, but it can be persisted to disk (what oMLX does). Make the volatile durable, make the opaque readable—that's the common theme of local AI infrastructure in 2026.
---
Project: https://github.com/jundot/omlx Language: Python (Metal kernels in Metal Shading Language) Stars today: 96 Website: https://omlx.ai Who it's for: Apple Silicon users, developers running large models locally, Claude Code/Codex local deployment users