PEEK: A Semantic-Layer Cache Architecture for Long-Context AI Agents (Context Map Explained)

Forum topic · 小凯 · 2026-05-23

Summary

PEEK (arXiv:2605.19932, MIT CSAIL + Stanford) is a semantic-layer caching system for LLM agents that repeatedly query the same large external context, such as 50k+ user feedback records, code repositories, or document collections. Instead of retaining trajectory history (Shared Chat), passively retrieving raw materials (RAG), or maintaining task-level playbooks (ACE), PEEK maintains a 'Context Map': a bounded, human-readable semantic cache persisted in the system prompt that captures orientation knowledge about the context itself. The closed-loop pipeline consists of a Distiller (extracts diagnosis, tags, and cache candidates from execution trajectories with zero external supervision), a Cartographer (applies structured ADD/DELETE/REPLACE edits with unique IDs and dedup), and an Evictor (priority-based eviction under a hard token budget, default B=1024). The map has five sections: Context Roadmap, Context Understanding, Domain Constants, Reusable Results, and Parsing Schema. PEEK outperforms the strongest baseline ACE by +6% to +15% across OOLONG and CL-bench benchmarks while cutting iterations, tokens, and cost (e.g., $4.79 vs $27.69 on TREC-Q-coarse with 58.1% vs 48.8% accuracy), and generalizes across GPT-5-mini, GPT-5.5, Qwen3-Coder, and both RLM and Codex agent frameworks. Maintenance overhead is only 6-18% of total cost, and PEEK is orthogonal to KV-cache optimizations.

PEEK: A Semantic-Layer Cache Architecture for Long-Context AI Agents

Paper: arXiv:2605.19932 Authors: Zhuohan Gu et al., MIT CSAIL + Stanford Date: 2026-05-19

This post is a structured technical breakdown of the PEEK system. Note: the post discusses an arXiv paper (2605.19932) whose claims are reported as stated in the source; the naming of some benchmarks (OOLONG) and models (GPT-5.5) appears unconventional but is preserved from the original.

Key Points

Problem: Agents repeatedly querying a large external context (50k+ records, repos, docs) have no way to persist reusable *orientation knowledge* — understanding of the context's structure, entities, constants, and patterns. Existing approaches (Shared Chat, RAG, ACE) each miss this dimension.
Core idea: PEEK maintains a Context Map — a bounded, human-readable semantic cache loaded in the system prompt — that accumulates knowledge about the context across queries.
Pipeline: Agent execution → Distiller (trajectory analysis producing diagnosis, per-entry tags helpful/harmful/neutral/stale, and transferable cache candidates — zero external supervision) → Cartographer (structured ADD/DELETE/REPLACE edits with unique IDs like [cr-00001], dedup, minimal edit set) → Evictor (priority eviction under a hard token budget B, default 1024).
Five-section map structure: Context Roadmap (index), Context Understanding (entities/concepts), Domain Constants (exact values), Reusable Results (derived computations), Parsing Schema (format structure). All start nearly empty and fill through interaction.

Notable Ablation Findings

| Alternative | Avg. gain | Failure reason | |---|---|---| | First 1024 tokens of context | +0.73% | Document openings rarely summarize structure | | Subgoal-based dynamic retrieval | +4.92% | Fragments clutter working memory | | ACE-style retrieval playbook | +0.73% | Accumulates task-specific policy, adds indirection | | Runtime feedback (full map replacement) | −14.86% | Overwriting per-step destroys stable orientation | | Split Distiller/Cartographer vs monolithic | monolithic −7.7% | Without separation, non-context facts leak into the map | | Frozen map (no priority eviction) | +18.0% | Still valuable, but full PEEK adds another +10.2% |

Key lesson: cache value comes from stability, not immediacy — distillation (understanding what happened) and cartography (deciding how to record it) are separate cognitive tasks that must not be merged.

Eviction Priority

Evicted first → protected last: Parsing Schema → Reusable Results / Domain Constants → Context Roadmap / Context Understanding (highest protection).

vs. KV Cache

PEEK operates at the agent level (semantic, prompt-visible, human-readable), not the model level (token states, hidden). KV cache makes the same agent cheaper/faster; PEEK decides what context knowledge should be kept, updated, or exposed. They are orthogonal and composable.

Design Space

A 2×2 space of (Active/Passive) × (Agent-Task State / External Context State). PEEK uniquely fills the Active External Context State quadrant — actively maintaining an artifact of what has been learned about a recurring external context.

Budget and Cost Analysis

Budget ablation: B=512 → +15.5%, B=1024 → +28.2%, B=2048 → +20.3% average gains; the untuned default of 1024 is best, suggesting *having a map* matters more than its size.
Maintenance cost share: TREC-Q-coarse $0.31/$5.10 (6.2%), AGNews 9.6%, Yahoo 17.9%, CL-bench 16.5%. Distiller ≈ 2/3 of maintenance cost.

Headline Results (vs ACE, strongest baseline)

| Benchmark | vs ACE | Fewer iterations | Cost advantage | |---|---|---|---| | OOLONG TREC-Q-coarse | +9.3% | 145 | 5.8× | | OOLONG AGNews | +7.8% | 93 | 1.7× | | OOLONG Yahoo | +15.0% | 138 | 1.7× | | CL-bench Solve | +6.0% | — | 1.4× | | CL-bench Rubric | +9.9% | — | 1.4× |

Efficiency (TREC-Q-coarse): PEEK — 378 iterations, 2.49M input tokens, $4.79, 58.1% accuracy; ACE — 523 iterations, 11.12M input / 12.45M output tokens, $27.69, 48.8%; RLM — $5.02, 30.3%; Shared Chat — $9.98, 32.0%.

Generality

Verified across four base-model/agent combinations (OOLONG avg gains): GPT-5-mini + RLM (+28.2%), GPT-5-mini + Codex (+44.0%), GPT-5.5 + RLM (+37.8%), Qwen3-Coder + RLM (+17.5%). Codex gains are largest, attributed to structured context understanding complementing production coding agents. The evolve step count m ≤ 4 suffices (m=1 freezes reuse; m=n is fully online), using far less maintenance overhead than ACE's fully online mode.

Engineer's Takeaway

1. Separate distillation from editing — cognitive decomposition prevents knowledge pollution. 2. Hard budget + priority eviction — constraints force focus; costs stay predictable. 3. Five-section structured map — turns vague "context understanding" into operable dimensions.

The deepest insight: in long-context agent systems, the cognitive value of the context itself is undervalued. We optimize how models *remember tokens* (KV cache) but rarely how agents *understand contexts* (context maps). PEEK fills exactly that blind spot.

> "PEEK opens a broader research agenda on how agents repeatedly interact with persistent contexts over diverse tasks."

*Source: arXiv:2605.19932, MIT CSAIL + Stanford, 2026-05-19.*