You ask a large model a question, and words appear on your screen one character at a time, like a drop of water seeping from a mountaintop and flowing down the slope to your eyes. The image feels effortless. But the same mountain, at the same moment, may be supplying water to hundreds of thousands of people — each of whom demands continuous flow, no impurities, and no cross-contamination. What makes this possible is not the mountain itself but its water infrastructure: the inference framework.
Last week (October 16), DeepSeek open-sourced this water system from its production environment, naming it DeepInfer. It gained 21,349 stars in one week. It serves exactly one model: DeepSeek-V4.1-Flash. Other frameworks try to fit everything; this one repairs only a single mountain — but replaces every pipe inside it with custom-built ones.
Three Mountains Before It
To understand what DeepInfer does, look first at the three frameworks that came before.
vLLM, open-sourced in 2023, made its name with one trick: paged VRAM. It slices the KV cache into small blocks allocated on demand, the way an operating system manages memory, so VRAM no longer reserves huge empty spaces for "answers of unknown length." Paired with continuous batching — new requests jump in whenever they arrive, without waiting for a batch to finish — this is terraced fields: land divided into small plots so not a single drop is wasted.
SGLang's signature is the prefix tree (RadixAttention): if you ask "introduce Hangzhou" and I follow up with "what about its West Lake?", the second question's prefix is identical to the first, so that compute should not be paid twice — all request prefixes are organized into a tree, with shared branches computed once, stored, and reused by everyone. Add a pipeline overlapping CPU scheduling with GPU computation. This is a water system: an upstream reservoir fills once and irrigates every downstream field.
TensorRT-LLM is NVIDIA's closed-source engine, rooted deep in the chip: FP4 low-precision compute, GB300-class machines with 72 GPUs in one interconnect domain. This is a stone channel: stone-built, paired with a dedicated mountain.
Then comes DeepInfer. The launch video frames it as a fourth mountain — that's fair — but its acknowledgments contain an easily missed line: the framework was originally built primarily on vLLM, with references to SGLang and TensorRT-LLM. In other words, the fourth mountain was rebuilt on the skeleton of the first — the blueprint of general-purpose terraces remains, but every terrace has been re-carved for this mountain's specific water flow.
Serving Only One Mountain
DeepInfer's supported model list has exactly one line: DeepSeek-V4.1-Flash — the October 17 snapshot with a 1M-token context window and MIT-licensed weights. A general framework must accommodate dozens of models and hardware combinations, so every optimization must be a "lowest common denominator"; DeepInfer doesn't have to. Every special corner of the architecture can be swapped for a dedicated pipe. That's what "perfecting one mountain" concretely means: not doing general things faster, but doing what only this mountain needs.
Its homepage benchmarks are measured accordingly: prefill on 4 SM100 GPUs, decode on 64, each within a single scale-up interconnect domain — ordinary 8-GPU machines handle the prelude, 64-to-128-GPU supernodes carry the main performance. No worship of giant machines, just stacks of commodity hardware.
A Hydraulic Engineering in Four Words
The video summarizes DeepInfer's craft in four phrases: compute less, never wait, stay balanced, stay consistent. All four have verifiable homes in the code and docs, and each is more interesting than it sounds.
Compute less. Don't pay a single unit of avoidable compute. V4.1-Flash's attention is two-tiered: a lightweight indexer first roughly filters which historical content is relevant, then main attention computes precisely over the filtered results. The indexer's own KV is stored in MXFP4; the main KV in FP4 (e4m3, scaled per 16-element group); routed expert weights in MXFP4; sliding-window KV in MXFP8 at 528 bytes per token. Sparsity stacked on low precision. Old context beyond the sliding window keeps only a hard 128-token window, and replay during recomputation touches only the most recent 128 tokens — "bounded replay" with a hard boundary. Cross-layer sharing is handled by YOCO (You Only Cache Once): for truncated-prefill versus full-prefill modes, the repo ships a script verifying element-wise equivalence — seconds to verify metadata, logprob alignment on 8 GPUs, even confidence scores compared bit-for-bit.
Never wait. Any wait that can be overlapped is never queued. The scheduler is asynchronous and zero-overhead: batches rearrange at any time, new requests insert at any time. Speculative decoding uses DSpark — not fixed-length draft verification, but a confidence head predicting the acceptance probability of each draft token, spending verification compute where marginal returns are highest. Even better is its interplay with async scheduling: when the next batch is being scheduled, the previous step's confidences aren't ready yet, so the CPU only sets a total budget, and each request's share is recomputed on the fly by a TileLang kernel on the GPU. Waiting isn't eliminated — it's taken apart and stuffed into other gaps.
Stay balanced. Water pressure must be equal everywhere. MoE models have inherently skewed expert loads — some experts crushed, others idle — so EPLB migrates experts to keep things even; long contexts are split into uniform segments by context parallelism. Less discussed is balance on the storage side: sliding-window KV stays in local memory; cross-machine hits go over a 400G VLAN; paged blocks are cached in 3FS distributed storage; KV caches are retained for over 72 hours — one cache hit eliminates an entire prefill. Three storage tiers, each with its own lifetime, each with its own pipe.
Stay consistent. The same question asked today and tomorrow must get the same answer. DeepInfer provides batch-invariant inference (NVIDIA side, for now) and deterministic sampling — no matter who your request shared a batch with or which GPU it landed on, the output is bit-identical. This isn't mere fastidiousness: evaluation PPL baselines are locked per-sample per platform and workload, and multi-node baseline refreshes go through cross-machine file locks, so nothing can quietly drift. Finally, "dual operators": the same framework runs on both NVIDIA SM100 and Huawei Ascend 950 — different sockets, same water pipe.
Water Infrastructure Is a Business
Behind these engineering details are three numbers. First, a single cluster currently serves 155 trillion tokens per day. Second, overall revenue is $38 per GPU-hour — and that's off-peak pricing. Third, KV caches are retained for over 72 hours, turning returning users' repeated prefill into a single cache hit.
"Cheap to read, expensive to write" — that's the cost structure of V4.1-Flash, the 890B model with 8B active parameters in prefill and 16B in decode, as dissected previously. DeepInfer essentially moves this cost structure from the model side to the datacenter side: the model governs each token's read/write volume; the framework governs where each token's water comes from, how many gates it passes through, and how long it's stored. Both sides are accounting for the same bill.
Those springs — FlashMLA (13,175 stars), DeepEP (10,347), DeepGEMM (9,089), EPLB — are old gates open-sourced back in February. This time they were installed together into a complete scheduling system. The parts existed long ago; the blueprint only came out last week.
If you want to inspect the pipes yourself, the repository is github.com/deepseek-ai/DeepInfer, Apache 2.0, 424 files — running the YOCO equivalence verification script takes only a few seconds. "Staying consistent" is a claim you can verify bit for bit.