Context Engineering as a Production Line: How Sessions and Memory Make Agents Remember, Stay Fast, and Stay Within Bounds

Forum topic · ✨步子哥 · 2025-12-28

Summary

Context Engineering treats every LLM call as a fully assembled payload rather than a static prompt, addressing the stateless nature of models by externalizing state into Sessions (short-term, per-conversation working memory) and Memory (long-term, cross-session knowledge). Sessions store event logs plus structured state, require strict user-level isolation, and demand compression strategies (sliding window, token truncation, recursive summarization) to combat context rot, latency, and cost. Memory is a curated, framework-agnostic layer built via an LLM-driven ETL process (extraction, consolidation, deduplication, conflict resolution, provenance tracking, and forgetting). It is distinct from RAG: RAG injects external authoritative facts, while Memory builds a user expert via declarative and procedural entries, retrieval scoring (relevance, recency, importance), and careful context injection (system prompt, history, or tool output). Multi-agent systems choose between shared unified histories and separate individual histories, with Memory serving as the portable common layer across frameworks. Production hardening covers PII redaction, ACL isolation, TTL policies, provenance, evaluation across generation, retrieval, and end-to-end task success, and defenses against memory poisoning.

Introduction

If tools give an Agent its "hands" to act on the world, Context Engineering is the methodology that keeps those hands from grabbing randomly — it decides what the model sees, ignores, remembers, and forgets on every turn. Models are inherently stateless: when a call ends, the model wakes up with no memory of what just happened. To give an Agent sustained conversation, long-term personalization, and cross-session experience accumulation, you must externalize state into two systems: Session (the conversational workbench) and Memory (the long-term filing cabinet), and dynamically assemble them into the context window each turn.

This sequel focuses on engineering practice: how to design Sessions, how to compress long conversations, how Memory is generated / consolidated / retrieved, and why Memory becomes the "universal layer" in multi-agent and cross-framework collaboration. Finally, we bring in privacy/security and evaluation metrics — because "remembering" is not the goal; remembering correctly, finding fast, using reliably, and not leaking is.

---

Chapter 1: What is Context Engineering — From "Writing Prompts" to "Assembling the Whole Request"

Traditional Prompt Engineering is more like writing a fixed system instruction; Context Engineering cares about the complete payload of every call: dynamically constructing a "stateful" request based on user, session, tool results, external knowledge, and long-term memory.

Think of it as *mise en place* before cooking: not just handing the chef a recipe, but pre-preparing the fresh ingredients, knives, spices, and plating requirements — the model does not need to "guess"; it just needs to reason and act with maximum determinism inside minimal noise.

Context typically consists of three categories of information:

A. Context that guides reasoning (the behavioral constitution)

System Instructions: persona, capability boundaries, rules
Tool Definitions: tool schema and descriptions
Few-shot Examples: examples that guide the reasoning path

B. Evidence and factual data (the reasoning evidence chain)

Long-Term Memory: cross-session user information / experience
External Knowledge: documents / database info retrieved via RAG
Tool Outputs: tool return values
Sub-Agent Outputs: conclusions from sub-agents
Artifacts: non-text materials such as files / images

C. Current interaction information (the matter at hand)

Conversation History: current turn's conversation history
State / Scratchpad: temporary variables, shopping cart, draft calculations, etc.
User Prompt: the user's current question

> Tip: more context is not always better > Long context brings cost and latency, and also produces "context rot": the more information, the harder it is for the model to grasp the key points, and reasoning quality actually drops. Context Engineering aims for "not too much, not too little, just right".

---

Chapter 2: The "Context Pipeline" of Each Turn — Fetch → Prepare → Invoke → Upload

A production-grade Agent typically goes through a four-step loop per turn:

1. Fetch Context: retrieve the needed context (memory, RAG, recent events, etc.) 2. Prepare Context: construct the final prompt (this is the blocking hot path) 3. Invoke LLM and Tools: iterate between model and tools; tool / model outputs are appended to the context 4. Upload Context: write new information to persistent storage (mostly asynchronous in the background)

Among these four, the most easily overlooked but most critical point is: Prepare is the blocking hot path; Upload should be backgrounded as much as possible. If you stuff expensive extraction / consolidation work into the hot path, you make the experience "stutter on every sentence".

---

Chapter 3: Sessions — The Conversational Workbench (Event Log + Working State)

3.1 A Session consists of two parts: Events and State

Events: time-ordered conversation events: user inputs, agent replies, tool calls, tool outputs, etc.
State: structured working memory (scratchpad), e.g. shopping cart contents, current task progress, confirmed parameters

A Session is a single-user, single-conversation container. A user can have multiple sessions, but they are disconnected "project workbenches" that do not naturally share memory (sharing must rely on Memory).

3.2 Session storage in production: because the runtime is usually stateless

---

Chapter 4: Framework Differences and Multi-Agent Session Patterns — Shared Ledger vs. Separate Ledgers

4.1 The essence of framework implementation differences: unified internal structure, diverse external protocols

4.2 Two ways of organizing Session history in multi-agent systems

#### A. Shared Unified History All agents write messages, tool calls, and observations into the same time-series log. Suitable for tightly coupled collaboration, strongly depending on a "single source of truth" pipeline-style task.

Pros: globally traceable, natural relay; cons: history inflates quickly, noisy, and harder to permission-isolate.

#### B. Separate Individual Histories Each agent has a private history and only outputs the final result externally (like a black-box tool). Often exchanges results rather than process via "Agent-as-a-Tool" or A2A messages.

Pros: clear boundary, low leakage risk; cons: shared context is sparse, collaboration needs extra design.

---

Chapter 5: The Hard Problem of Cross-Framework Interoperability — Sessions Are Not Portable, Memory Can Serve as the "Universal Layer"

Different frameworks' session / event storage models are often tightly bound to their internal object structures, so a Session written by one framework is hard to read directly by another. You can pass messages via A2A, but the "rich state" still needs a translation layer.

A more robust pattern is to abstract shared knowledge into a framework-agnostic Memory layer. Memory stores not framework event objects, but extracted facts / entities / summaries (strings or dicts). It can therefore become a shared cognitive resource between multiple frameworks and multiple agents.

---

Chapter 6: Production Considerations for Sessions — Security Isolation, Data Integrity, Performance, and Long-Conversation Compression

6.1 Security and privacy: strict isolation + PII redaction before storage

Sessions belong to a single user; ACL must be strictly isolated to prevent cross-user access
Best practice: PII should be redacted / sanitized before writing to storage, reducing blast radius and aiding GDPR / CCPA compliance

6.2 Data integrity: sequential consistency + lifecycle (TTL)

Event-append order must be deterministic (out-of-order logs directly break reasoning)
Sessions should not be kept forever: use TTL or archival policies to control cost and compliance risk

6.3 Performance: Sessions are on the hot path, must be "fast and small"

---

Chapter 7: Long-Conversation Management — Compress History Like Packing a Suitcase

Long conversations impose four hard constraints: 1) context window upper limit 2) token cost 3) latency 4) quality (noise and autoregressive error)

Common compression strategies (from simple to complex):

7.1 Sliding Window: keep the most recent N turns

7.2 Token-Based Truncation: backtrack and truncate according to a token budget

7.3 Recursive Summarization: older content becomes summaries

Summarization should be generated asynchronously in the background and persisted, to avoid re-summarizing every turn
Record which events have been covered by summaries, to avoid re-injecting the original text

7.4 Compression triggers: when to do it?

Count trigger: compress when turn / token exceeds a threshold (most common, "good enough")
Time trigger: compress in the background after the user is silent for a while
Event trigger: compress when a sub-task / topic is detected to have ended

---

Chapter 8: Memory — The Long-Term Filing Cabinet (Extracted, Reusable "Small but Precise" Items)

Memory is not defined as "storing the conversation", but as extracting and solidifying valuable information from the conversation, persisting across sessions, used for personalization, context management, data insight, and self-improvement.

Four capabilities that Memory brings

Personalization (preferences, facts, history)
Standing in for long history (summaries / key facts reduce tokens)
Population-level insight (mining trends after aggregation, with privacy)
Self-improvement (record successful strategies / tool paths to form a playbook)

---

Chapter 9: Boundary between Memory and RAG — One Knows the World, One Knows the User

RAG: inject external, static, authoritative facts; usually shared, read-only
Memory: condense user-related, dynamic, isolated context; needs writes and evolution

In one sentence: RAG makes the system a "fact expert", Memory makes the system a "user expert".

---

Chapter 10: Structure and Types of Memory — Content + Metadata; "Knowing What" and "Knowing How"

10.1 Basic structure of Memory

content: factual snippet (text or structured JSON / dict)
metadata: id, owner, tags, source, etc.

10.2 Knowledge types: Declarative vs. Procedural

Declarative (knowing what): facts / preferences / events
Procedural (knowing how): skills / processes / workflow "playbooks"

> Key difference: procedural memory is not "retrieving data" but "retrieving an executable plan"; it is more like a reasoning enhancement layer. Compared with fine-tuning (offline, changes weights), procedural memory is an online playbook injected in-context, enabling rapid correction via in-context learning.

---

Chapter 11: How Memory Is Organized and Stored — Collections / User Profiles / Rolling Summary; Vector DB / Knowledge Graph / Hybrid

11.1 Organization patterns

Collections: a bag of atomic memories, good for search and multi-topic
Structured User Profile: stable fields (name, preferences), good for fast reading
Rolling Summary: a continuously updated master summary, commonly used to compress long sessions

11.2 Storage architecture

Vector database: semantic-similarity retrieval, good for unstructured memory
Knowledge graph: entity-relation reasoning, good for structured relations
Hybrid: graph nodes carry embeddings, supporting both semantic and relational retrieval

---

Chapter 12: Memory Generation — LLM-Driven ETL of Extraction + Consolidation

Memory generation is not simple summarization, but an LLM-driven ETL:

1) Ingestion: input source data (usually session history) 2) Extraction & Filtering: extract meaningful information by "topic definitions" (skip if not matched) 3) Consolidation: deduplication, conflict resolution, update / create / delete, and forgetting (TTL / low confidence) 4) Storage: write to vector DB / graph, etc.

12.1 Extraction: deciding "what is worth remembering"

schema / template-based extraction (structured output)
natural-language topic definitions
few-shot examples (especially effective for subtle domains)

Many systems use a "rolling summary" as auxiliary input for extraction, to improve efficiency and avoid re-scanning the full history every turn.

12.2 Consolidation: without it, memory becomes a contradictory junk heap

duplication (same fact in multiple expressions)
conflicts (user preferences change)
evolution (facts become more specific)
decay and forgetting (old, low-confidence items deleted / down-weighted)

Common consolidation operations: UPDATE / CREATE / DELETE (or INVALIDATE).

---

Chapter 13: Memory Provenance — Why Should You Trust It?

Without source tracking, long-term memory drifts toward "confident nonsense". Provenance should at least record:

source type (preloaded system data / user input / tool output)
freshness (time)
possible confidence changes (rises with multi-source corroboration, decays over time)

An important recommendation: long-term memory generated from tool output is usually not recommended, because tool data is better suited for short-term caching — it becomes stale and brittle.

Engineering significance of provenance:

establish a trust hierarchy during consolidation (trusted source first, recent first, multi-source corroborated first)
when deleting a data source, regenerate precisely by lineage instead of crude mass deletion

---

Chapter 14: When to Generate Memory — Memory-as-a-Tool Makes the System Smarter, but Costs Must Be Controlled

The trigger strategy trades off "cost vs. fidelity":

End of session: cheap, but possibly low fidelity
Every N turns: common compromise
Real-time every turn: freshest, but most expensive
Explicit user instruction: controllable but low coverage
Memory-as-a-Tool: lets the Agent itself decide when to call create_memory; smarter but requires stricter tool definitions and policy controls to avoid "over-memorization"

Background vs. blocking: memory generation should almost always be async

---

Chapter 15: Memory Retrieval — Relevance Alone Is Not enough; Freshness and Importance Matter Too

The goal of retrieval: within a strict latency budget, find the most useful memories.

Common scoring dimensions:

Relevance (semantic similarity)
Recency (time freshness)
Importance (annotated at generation)

Using only vector similarity is a common pitfall: it pulls up memories that are "very similar but very old / very trivial". A multi-dimensional hybrid score is more robust.

Advanced enhancements (but slower):

query rewriting (one extra LLM)
reranking (first take top50, then rerank with LLM)
training a dedicated retriever (needs labeled data, expensive)

These can be combined with caching to avoid repeating high-cost flows each time.

Retrieval timing: proactive prefetch vs. reactive query

Proactive prefetch: fetch at the start of every turn; simple but adds latency; can be cached
Reactive (Memory-as-a-Tool): only fetch when needed; cheaper but may add one LLM call; the Agent needs to "know what types of memory may exist", otherwise it doesn't know whether to query

---

Chapter 16: Putting Memory into Context — Where You Put It Determines Its Weight

Three main injection methods:

16.1 Into the system instructions

16.2 Injected into conversation history

16.3 As tool output

---

Chapter 17: How to Evaluate Whether "the Memory System Actually Works" — Quality, Retrieval, End-to-End

Evaluation has three layers:

17.1 Generation quality: are we remembering correctly?

Precision: how much of what is recorded is correct and relevant (guards against "over-memorization" pollution)
Recall: did we miss anything that should have been recorded (guards against "key-fact loss")
F1: combined metric

17.2 Retrieval performance: can we find it, and fast?

Recall@K: whether the needed memory appears in topK
Latency: retrieval must be within a strict budget (e.g. <200ms)

17.3 End-to-end task success: does memory actually improve task completion?

---

Chapter 18: Privacy and Security — Memory Is the "Corporate Archive", Not a "Casual Notepad"

The bottom line of long-term memory:

Strict isolation (user / tenant scope): cross-user leakage is a fatal incident
User control: opt-out and delete must be possible
PII redacted before storage
Defend against memory poisoning: validate and sanitize injected and forged information
Procedural memory shared across users must be strongly anonymized, otherwise "experience reuse" becomes "information leakage"

---

References

1. Retrieval-Augmented Generation overview: https://cloud.google.com/use-cases/retrieval-augmented-generation?hl=en 2. Agent Engine Sessions overview: https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/sessions/overview 3. Agent Engine Memory Bank – generate memories: https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/memory-bank/generate-memories 4. Model Armor overview: https://cloud.google.com/security-command-center/docs/model-armor-overview 5. Gemini long context limitations: https://ai.google.dev/gemini-api/docs/long-context#long-context-limitations