A Policy-Driven Runtime Layer for Agentic LLM Serving (arXiv 2605.27744)

Forum topic · 小凯 · 2026-05-29

Summary

This paper proposes an agent runtime layer inserted between agent frameworks and serving engines to close the architectural gap in multi-agent LLM serving. Agent frameworks know agent identities, roles, schemas, and dispatch structure but never see engine-level events, while serving engines see every event but know nothing about agents. Many cross-cutting policies—prefix caching, batch shaping, speculative execution, fairness, tool-result memoization, and safety enforcement—depend on both views and are currently implemented as one-off patches. The authors introduce a third tier exposing four primitives (observe, score, predict, act), into which any agent-aware policy can plug, using agent identity as a shared coordinate. Nine concrete policies are mapped to this layer, with deep validation on cross-session KV caching via CacheSage, which online-learns per-workload agent transition matrices for lifetime-based eviction and inter-step prefetching. On five real multi-agent workloads, CacheSage improves cache hit rates by 13–37 percentage points, reduces average TTFT by 12%–29%, and increases throughput by 6%–14%. Paper: arXiv 2605.27744, by Rui Zhang, Chaeeun Kim, and Liting Hu.

Paper Overview

Research Area: LLM Authors: Rui Zhang, Chaeeun Kim, Liting Hu Published: 2026-05-28 arXiv: 2605.27744

Abstract

Multi-agent LLM systems have become the dominant production workload, but the serving stack was not built for them. The agent framework above knows agent identities, roles, schemas, and dispatch structure but never sees an engine-level event; the serving engine below sees every event but knows nothing about agents. A surprising number of cross-cutting policies depend on both: prefix caching, batch shaping, speculative execution, fairness, tool-result memoization, safety enforcement, and more. Each lives in the seam between the two layers and is currently solved by a one-off patch into one neighbor or the other.

The paper argues this seam is best addressed by an architectural change rather than point fixes: insert a third tier, an agent runtime layer, between the framework and the engine, exposing four primitives (observe, score, predict, act) into which any agent-aware policy plugs, with agent identity as the shared coordinate.

Key Contributions

Agent runtime layer: a new tier between agent frameworks and serving engines, providing four primitives — observe, score, predict, act — that any agent-aware policy can plug into.
Nine policies mapped to the layer, including prefix caching, batch shaping, speculative execution, fairness, tool-result memoization, and safety enforcement.
CacheSage: a deep instantiation on KV cross-session caching. It online-learns per-workload agent transition matrices to drive lifetime-based eviction and inter-step prefetching.

Results

On five real multi-agent workloads, preliminary results show:

Cache hit rate improved by 13–37 percentage points
Average TTFT reduced by 12%–29%
Throughput increased by 6%–14%

---

*Auto-collected on 2026-05-29.*