Paper Overview
- Field: LLM serving / multi-agent systems
- Authors: Rui Zhang, Chaeeun Kim, Liting Hu
- Published: 2026-05-28
- arXiv: 2605.27744
- Online-learns a per-workload agent transition matrix
- Uses it for lifetime-based eviction and inter-step prefetching
- Cache hit rate improved by 13–37 percentage points
- Average TTFT reduced by 12%–29%
- Throughput improved by 6%–14%
Summary
Multi-agent LLM systems have become the dominant production workload, but the serving stack was not built for them. The agent framework above knows agent identities, roles, schemas, and dispatch structure but never sees an engine-level event; the serving engine below sees every event but knows nothing about agents.
A surprising number of cross-cutting policies depend on both: prefix caching, batch shaping, speculative execution, fairness, tool-result memoization, safety enforcement, and more. Each lives in the seam between the two layers and is currently solved by a one-off patch into one neighbor or the other.
Key Contributions
The paper argues this seam is best addressed by an architectural change rather than point fixes: insert a third tier, an agent runtime layer, between the framework and the engine. The layer exposes four primitives:
1. Observe 2. Score 3. Predict 4. Act
Any agent-aware policy plugs into these primitives, with agent identity serving as a shared coordinate across layers.
Validation: CacheSage
The authors map nine concrete policies onto this layer and deeply validate the one with the most immediate cost leverage: KV cross-session caching, instantiated as CacheSage. CacheSage:
Results
On five real multi-agent workloads, preliminary results show:
*Auto-collected on 2026-05-29.*