> Published: 2026-06-29 · Category: Product release/update · Tags: AI inference infrastructure > Source: Xiaohongshu Tech (Redtech) official account + arXiv 2606.06256 > Original post: https://mp.weixin.qq.com/s/qRrZvL0aZzYI82djFSrLug > Paper: https://arxiv.org/abs/2606.06256
What Happened
On June 29, Xiaohongshu's engine architecture AI Infra team open-sourced RedKnot, a long-context inference engine rebuilt around "head-wise KV Cache decomposition." The paper is on arXiv and the code is publicly maintained.
The starting point is simple: today every LLM's KV Cache is a dense tensor with shape [B, H, L, D] (batch size, number of heads, sequence length, per-head dimension). All heads for the same tokens are stored, moved, and computed together. But Xiaohongshu's empirical finding is: the value of the KV Cache is not uniformly distributed across tokens—it is strongly stratified by head.
Some heads genuinely need full context (e.g., global semantic alignment); others mainly look at local windows (e.g., local syntactic patterns). Current inference systems ignore this and process all heads as one bundle.
RedKnot does three things:
1. Head-classified sparsity. Offline classification of each (layer, head) pair—few "global heads" (12-15%, full context) and mostly "local heads" (85-88%, sliding window only). Validated on Mistral-7B, Qwen3-32B, Llama-3.3-70B, Qwen3.5-397B, and DeepSeek-V4-Flash, the local-head share is stable at 83.4-96.8%.
2. Sparse FFN. Dense FFN runs only on the top-k tokens by attention score; all other tokens take the residual identity path. This targets the 2-8K segments common in agent workloads—where FFN, not attention, is the real prefill bottleneck (57-62% of TTFT).
3. SegPagedAttention storage. Replaces the dense layout with paged KV storage segmented by (layer, head), plus a fused variable-length attention kernel that physically stores only the tokens each head actually needs. Every head stays on the FlashAttention fast path with no attention mask constructed.
The three techniques act on orthogonal dimensions—heads, storage, and channels—so gains multiply rather than compete.
Benchmark Results
Measured on 8x H800, across 3 models, 6 QA datasets, 8K-128K context:
- TTFT speedup: DeepSeek-V4-Flash goes from 3.51x at 16K to 5.16x at 128K—speedup grows with context, which token-level baselines cannot do;
- Llama-3.3-70B HotpotQA: exact match (EM) improves from 0.60 (dense baseline) to 0.80; first-token top-1/top-10 agreement with the dense path is 0.93/0.87;
- Single-GPU concurrency: rises from 4 to 31 at 32K context;
- KV transfer bytes: up to 6.3x savings in PD-disaggregated scenarios;
- Prefill compute: reduced by 67-79.5%.
- Prefix caching—compute the KV for a conversation prefix once and reuse it (DeepSeek's MTP, Qwen's PD disaggregation);
- Position-independent KV reuse (PIC)—reuse KV when the same document chunk appears at different positions (CacheBlend, Epic, ProphetKV).
- Experimental data is based on H800; reproduction on consumer GPUs (RTX 4090/5090) and domestic accelerators has not been shown;
- The stability of offline head classification needs long-term observation—if distribution shift invalidates the classification on some tasks, sparsity could hurt accuracy;
- Whether "sparse FFN via residual identity" truly preserves the model's original gradient flow requires more downstream validation;
- Section 4.4 of the paper admits ~5% accuracy degradation on certain long-tail tasks for DeepSeek-V4-Flash—whether production can tolerate that depends on the scenario.
On accuracy, results are generally ≥ 95% of dense F1 and can even exceed dense in long-context regimes—because low-value token noise is suppressed while high-quality attention structure is preserved.
Engineering is based on SGLang. The runtime is architecture-agnostic: it covers standard GQA, Qwen3.5-style hybrid attention + MoE, and DeepSeek-V4-style MLA-compressed attention.
Deep Analysis
Why does the "per-head" view matter so much?
Over the past year, long-context inference optimization has followed two paths:
But RedKnot's Figure 1 exposes an awkward fact: PIC methods are "unreliable" because they select recomputation subsets at token granularity—yet different heads attend to different token subsets. To satisfy all heads, a system must take the union of each head's important tokens, which often covers most of the chunk and cancels the reuse benefit.
In other words: token granularity is the wrong granularity. "Per-head" was the workload's true sparsity structure all along.
Why does FFN optimization matter so much for short-context agent workloads?
RAG, coding agents, and long-session systems typically run prompts of 2-8K tokens. In this range, attention is not the bottleneck—FFN accounts for 57-62% of TTFT. Even eliminating attention recomputation entirely leaves that cost untouched.
FFN is hard to sparsify—cutting FFN for some tokens breaks the residual stream. RedKnot's solution is "run dense FFN only on top-k tokens by attention score; everything else takes the residual identity"—using attention as a router for FFN sparsity.
"Sparsification as denoising" is more counterintuitive than the speedup itself.
A quote worth re-reading: "The longer the context, the more concentrated attention becomes. RedKnot can actually *surpass* dense computation in long-context regimes—because it suppresses low-value token noise while preserving high-quality attention structure."
This means RedKnot is not just "faster dense compute"—it is "better compute." Low-value tokens in long prompts genuinely drag down accuracy. Anthropic and OpenAI engineers have surely considered this internally; Xiaohongshu turned it into an engineerable open-source implementation.
Relation to DeepSeek DSpark (June 28).
Yesterday's DSpark accelerates speculative decoding (small model drafts, large model verifies; 60-85% overall generation speedup). Today's RedKnot sparsifies prefill (splitting dense attention + dense FFN to cut first-token latency). The two are orthogonal: one targets the generation phase, the other the input phase; one from a model architecture angle, the other from a system scheduling angle. The open-source cadence of Chinese LLM inference infrastructure is becoming systematic.
Why It Matters
1. The cost structure of long-context agents may be rewritten. For coding agents, deep research agents, and customer-service agents that run half-hour sessions, TTFT and concurrency are the core costs. If RedKnot's speedups hold in production, unit costs could drop to 1/3-1/5 of current levels. 2. "Per-head" may become the common language of long-context inference. Future releases of vLLM, SGLang, and TensorRT-LLM will likely introduce similar head-classification mechanisms. 3. "Sparsification as denoising" opens a new research dimension—long-prompt accuracy may be about using context more precisely, not adding more of it. 4. First full appearance of Xiaohongshu's AI Infra team. Their earlier open-source work (e.g., TensorFusion) was training-side; RedKnot is their first publicly disclosed inference-engine release, revealing team scale and research direction.