ActKV Dissected: 99% of an Agent's KV Cache Is Scratch Work, and the Budget Follows Hesitation

Start with the numbers. An agent completing one purchase on WebShop — search, pagination, comparison, checkout — averages 33,000 tokens of trajectory under the ReAct framework; ALFWorld is slightly shorter at 31,000. All of it sits in GPU memory as KV cache and participates in attention with every new request. A USTC team (Xuehai Zhou's group, seven authors, arXiv 2609.31395) broke trajectories down by observation-reasoning-action segments: observations and reasoning account for over 99% of KV cache entries, while the action tokens that actually determine task success are under 1%. That asymmetry is the entire premise of ActKV: actions are where the agent truly commits; everything else is scratch paper.

Why a new framework

The authors claim ActKV is the first KV cache management framework purpose-built for agent inference. The qualifier holds: prior work (H2O, SnapKV, StreamingLLM, PyramidKV, Ada-KV) compresses generic dialogue or reasoning; the closest, R-KV (NeurIPS 2025), handles reasoning redundancy, but its vLLM adaptation copies KV into a contiguous buffer before compressing, giving back half the savings. ThinKV modifies block-table metadata and is tied to specific policies. A framework-level treatment (eviction + budget + page management) for agents was a genuine gap.

Component 1: Action-attention-guided eviction

Action boundaries are mechanically detectable because the model emits actions via standardized templates (function-call format). For each KV entry, ActKV tallies attention scores from all action tokens; entries in the top 90% are "action-critical." This signal then feeds LRFU — the classic OS page-replacement algorithm — with its hit signal replaced by "hit by action attention" and a decay coefficient λ = 0.5 balancing recency and frequency. Note the marketing trap: LRFU alone has no agent semantics; the real work is done by the action-attention signal, with LRFU merely smoothing it over time.

Component 2: Budget follows hesitation

The budget is not set by offline tuning but adjusted in real time from the model's own hesitation, measured as the average negative log-probability of the top-20 tokens in each step's output distribution. Token-level signals are aggregated into trajectory-level confidence via sliding-window min-pooling (window 64, stride 8), targeting local confidence collapses — which the paper notes occur when the model restates missing context, patches its reasoning, or utters "wait," "I guess," "let me check." A linear fit over trajectory-level confidence with negative slope indicates the budget is too small (information was compressed away, so the model compensates with more words), triggering a 1.75x budget expansion, capped at 30,000 entries.

Crucially, expansion is one-way. When confidence recovers, ActKV does not shrink the budget: recovery only shows the current budget suffices, not that a smaller one would, and shrinking risks a second round of information loss. It is a ratchet — up only. Any claim of "zero memory waste when the model is confident" misreads "no expansion" as "active shrinking." A documented side effect: compressed trajectories average 9.72% longer than FullKV, which raises peak memory — a feedback loop the confidence monitor must absorb.

Component 3: Systems engineering

Token-level eviction hits two walls under paged memory management. First, paged attention engines (the basis shared by vLLM/SGLang) don't expose the full attention matrix, and recomputing it would forfeit the savings. ActKV's trick: softmax's log-sum-exp is already a byproduct of paged attention, so attention scores can be recovered per-block by subtracting LSE from the current query's logits — no gather, no recompute. Second, eviction leaves holes; naive parallel in-place migration causes read/write conflicts. ActKV first plans conflict-free slot assignments — surviving entries skew toward the tail, so the last X logical blocks become the target region; surviving entries outside are paired one-to-one with holes inside, copied in one parallel pass, and freed blocks are returned to the scheduler. Both kernels are written in Triton; a single compaction is on average 10.41x faster than paged R-KV, with no extra KV buffer.

Results

Setup: four models (Qwen3-30B-A3B-Thinking, Qwen3-235B-A22B-Thinking-FP8, GPT-OSS-20B/120B), three benchmarks, 1,500 tasks each, eight runs per task, on 8x RTX PRO 6000 Blackwell.

• Long-trajectory tasks (ALFWorld + WebShop): 98.53% of FullKV accuracy with 25.98% peak memory (self-consistent averages of 98.04/99.02 and 26.38/25.58 across datasets).
• Strongest baselines (R-KV, SnapKV, StreamingLLM) at the same budget top out at 74.22% — a 24-point gap.
• Throughput: 3.96-4.14x token throughput, 3.62-3.80x task throughput.
• Counterintuitively, compression sometimes *improves* accuracy: up to +1.40 on ALFWorld, +3.13 on WebShop — attention dilution: stale scratch work inflates the softmax denominator, so discarding it sharpens focus on action-relevant content. With just 10% budget, ALFWorld retains 81.34%; on WebShop, 25% budget reaches 101.41% of FullKV.

Caveats

• "Over 99% is redundant" conflates share with redundancy: what can actually be discarded for free is about three-quarters; much of what is retained is precisely the observations/reasoning LRFU selected.
• "Perfect vLLM/SGLang compatibility" oversells: only Transformers 5.3.0 and vLLM 0.19.0 were tested; SGLang is cited only as the origin of PagedAttention, with no SGLang experiments.
• No official code link; the only same-named GitHub repo (Maurya30/actkv-open) is a third-party replica with zero stars, created October 4 — not from USTC.
• No standalone Limitations section, though disclosures are embedded: gains on GPT-OSS are smaller because half its layers use sliding-window attention and already discard KV.

Lineage

The paper cites the same group's January Crystal-KV (arXiv 2601.16986), which compresses chain-of-thought KV under an "answer-first" principle. Crystal-KV to ActKV moves the "allocate memory by answer value" idea from reasoning into agents: same playbook, upgraded semantics — in reasoning the answer sits at the end; in agents the "answer" is the actions scattered throughout.

The common trend across recent KV accounting schemes (KVMem's placement-side pooling vs. ActKV's admission-side budgeting): context value is determined not by the token, but by who actually acts in the task. Assertion-execution separation has landed in memory management: observations and reasoning assert; budget follows execution.

One number you can verify yourself: adaptive mode starts with a 512-entry budget, multiplies by 1.75 on each confidence drop, caps at 30,000 — six doublings of headroom for a 33K-token trajectory. That budget curve is itself a record of the model's hesitation; the paper's Figure 9 shows it peaking near a quarter of FullKV memory.