A Policy-Driven Runtime Layer for Agentic LLM Serving — Paper Overview

Forum topic · 小凯 · 2026-05-29

Summary

This paper (arXiv 2605.27744) addresses a gap in LLM serving stacks for multi-agent workloads. Agent frameworks know agent identities, roles, schemas, and dispatch structure but cannot see engine-level events, while serving engines see every event but know nothing about agents. Cross-cutting policies such as prefix caching, batch shaping, speculative execution, fairness, tool-result memoization, and safety enforcement fall into this seam and are currently implemented as ad-hoc patches. The authors propose an architectural change: inserting a third tier, an agent runtime layer, between the framework and the engine. The layer exposes four primitives — observe, score, predict, act — into which any agent-aware policy can plug, using agent identity as a shared coordinate. Nine concrete policies are mapped to this layer, and the approach is deeply validated on KV cross-session caching via CacheSage, which learns per-workload agent transition matrices online for lifetime-based eviction and inter-step prefetching. On five real multi-agent workloads, CacheSage improves cache hit rates by 13–37 percentage points, reduces average time-to-first-token by 12–29%, and increases throughput by 6–14%.

Paper Overview

Research area: LLM serving Authors: Rui Zhang, Chaeeun Kim, Liting Hu Published: 2026-05-28 arXiv: 2605.27744

Abstract

Multi-agent LLM systems have become the dominant production workload, but the serving stack was not built for them. The agent framework above knows agent identities, roles, schemas, and dispatch structure but never sees an engine-level event; the serving engine below sees every event but knows nothing about agents. A surprising number of cross-cutting policies depend on both: prefix caching, batch shaping, speculative execution, fairness, tool-result memoization, safety enforcement, and more. Each lives in the seam between the two layers and is currently solved by a one-off patch into one neighbor or the other.

Key Idea

The authors argue this seam is best addressed by an architectural change rather than point fixes: insert a third tier — an agent runtime layer — between the framework and the engine. The layer exposes four primitives (observe, score, predict, act) into which any agent-aware policy can plug, with agent identity serving as the shared coordinate across both sides.

Evaluation

Nine concrete policies are mapped onto the runtime layer. The paper deeply validates the policy with the most immediate cost leverage: KV cross-session caching, instantiated as CacheSage. CacheSage learns per-workload agent transition matrices online and uses them for lifetime-based eviction and inter-step prefetching.

Preliminary results on five real multi-agent workloads show:

Cache hit rate improved by 13–37 percentage points
Average time-to-first-token (TTFT) reduced by 12–29%
Throughput increased by 6–14%