A Policy-Driven Runtime Layer for Agentic LLM Serving

Forum topic · 小凯 · 2026-05-29

Summary

A paper by Rui Zhang, Chaeeun Kim, and Liting Hu (arXiv:2605.27744, May 2026) identifies a structural gap in LLM serving stacks for multi-agent workloads: agent frameworks know agent identities and orchestration but lack visibility into engine-level events, while serving engines see every event but know nothing about agents. Cross-cutting policies such as prefix caching, batch shaping, speculative execution, fairness, tool-result memoization, and safety enforcement fall into this seam and are currently handled by ad-hoc patches. The authors propose an architectural solution: inserting a third tier—an agent runtime layer—between the framework and engine, exposing four primitives (observe, score, predict, act) into which agent-aware policies can plug, using agent identity as the shared coordinate. Nine concrete policies are mapped to this layer, and the approach is deeply validated on KV cross-session caching via CacheSage, which learns per-workload agent transition matrices online for lifetime-based eviction and step-ahead prefetching. On five real multi-agent workloads, CacheSage improves cache hit rates by 13-37 percentage points, reduces average time-to-first-token by 12-29%, and increases throughput by 6-14%.

Overview

Research area: LLM serving Authors: Rui Zhang, Chaeeun Kim, Liting Hu Published: 2026-05-28 arXiv: 2605.27744

Key Idea

Multi-agent LLM systems have become the dominant production workload, but the serving stack was not built for them. There is a structural seam between:

The agent framework (above): knows agent identities, roles, schemas, and dispatch structure, but never sees engine-level events.
The serving engine (below): sees every event, but knows nothing about agents.

A surprising number of cross-cutting policies depend on both sides: prefix caching, batch shaping, speculative execution, fairness, tool-result memoization, safety enforcement, and more. Each currently lives in this seam, solved by one-off patches into one neighbor or the other.

Proposed Architecture

The paper argues for an architectural change rather than point fixes: insert a third tier — an agent runtime layer — between the framework and the engine. This layer exposes four primitives:

1. Observe 2. Score 3. Predict 4. Act

Any agent-aware policy can plug into these primitives, with agent identity serving as the shared coordinate across layers.

Evaluation

Nine concrete policies are mapped to the runtime layer, with deep validation on the policy with the most immediate cost leverage: KV cross-session caching, instantiated as CacheSage. CacheSage learns per-workload agent transition matrices online to drive lifetime-based eviction and step-ahead prefetching.

Preliminary results on five real multi-agent workloads:

Cache hit rate: +13-37 percentage points
Average TTFT: -12% to -29%
Throughput: +6-14%