MemCon: Memory Is Not a Warehouse, It's a Controlled Process for LLM Agents

An Overlooked Problem

Memory systems have become standard for LLM agents. Voyager uses skill libraries, MemoryBank uses reflective summaries, OAgents uses rule induction—many approaches flourish. But these systems share a common assumption: the way memory is accessed is fixed.

Specifically, they all use "hand-designed heuristics" to decide when to retrieve, what to retrieve, and how much. The most common pattern: run a nearest-neighbor retrieval every turn and stuff the top-k results into context.

In the paper *Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents*, UCLA's Eric Hanchen Jiang, Zhi Zhang, and colleagues argue: this assumption is wrong. Optimal memory behavior is context-dependent, and no fixed set of rules applies to all situations.

Four Counter-Intuitive Scenarios

The paper lists four typical situations, each requiring a different memory strategy:

1. Early in a task: The memory store is still sparse, so retrieval yields mostly irrelevant fragments. The optimal strategy is to retrieve little or not at all. 2. Recurring task types: If a similar task was solved before, the best move is not another fuzzy retrieval but directly reusing the previous plan (plan injection). 3. The agent is stuck: If the current retrieval didn't help, the optimal strategy is not to give up but to re-retrieve with alternative queries. 4. Long task streams: As the memory store grows, retrieval quality degrades. The optimal strategy is periodic consolidation and pruning, and even active forgetting.

These correspond to four entirely different actions. A fixed heuristic must force one strategy onto all scenarios, and will inevitably fail in some of them.

How MemCon Works

MemCon (Memory as a Controlled Process) models memory operations as a Markov Decision Process (MDP):

• State \(\mathcal{S}\): contextual features of the current task (task phase, memory size, whether the agent is stuck, etc.)
• Actions \(\mathcal{A}\): {Retrieve (various depths), PlanInject, Re-Retrieve, Consolidate, Forget, NoOp}—9 actions total
• Reward \(\mathcal{R}\): task success (+1) or failure (-0.5), plus an efficiency bonus
• Transitions \(\mathcal{T}\): determined by the environment and memory state

Then it learns an online policy deciding which action to take in each state.

A key design choice: the policy is not a neural network but a tabular contextual bandit with UCB exploration. Why?

• No pretraining needed: it learns from scratch via task feedback
• No extra LLM calls: the policy itself is lightweight and adds no inference cost
• Fast convergence: converges within tens of tasks (the paper proves a regret bound of \(O(\log T / T)\))
• Interpretable: the Q-value of every state-action pair can be inspected

The policy update uses backward-discounted cumulative reward: \(\gamma^{|\text{ep}|-j-1} r_i\), giving later actions in a task more weight (since they are closer to the final outcome).

An Analogy

Imagine an LLM agent's memory system as a librarian.

The traditional approach: no matter who asks, follow the same procedure—search the catalog, fetch the 5 most relevant books, hand them over. Useful for some readers, wasteful for many.

The librarian trained by MemCon adapts to the situation:

• A first-time visitor (early task): "Just browse around, no need to search the catalog."
• A returning reader with a similar topic (recurring task): "Here's the stack you borrowed last time—take it directly."
• A reader saying "I can't find what I need" (stuck): "Try another keyword."
• Shelves overflowing (long task streams): "Let me archive these old books to free up space."

This librarian doesn't need a PhD—it just needs to learn which action fits which situation. That's exactly what contextual bandits are good at.

Experimental Results

Tested on 6 benchmarks (ALFWorld, ScienceWorld, Blocksworld, TriviaQA, WebWalkerQA, GAIA), 3 agent frameworks, and 3 LLM backbones (Claude Sonnet-4, DeepSeek-V3.2, Qwen):

• Up to 15.2 percentage points improvement in task success rate
• 5-20% reduction in token consumption (because unnecessary retrievals are avoided)
• Consistently outperforms baselines including Voyager, MemoryBank, Generative Memory, OAgents, and G-Memory across all benchmarks

The token efficiency gain is particularly notable: MemCon is not only more accurate but also cheaper—because it learned "when NOT to retrieve."

Why It Matters

For agent design: MemCon reveals a neglected dimension—the control policy of a memory system is itself a learnable problem. Prior work optimized what to store and how to retrieve, but ignored equally important decisions like when to retrieve, how much to retrieve, and when to forget.

For RL in agents: MemCon uses the simplest contextual bandit, not deep reinforcement learning. This shows that for many agent subproblems, lightweight RL suffices—no need for heavy machinery like PPO or DQN. What matters is modeling the problem correctly, not algorithmic complexity.

For sustainable agent operation: Memory bloat over long task streams is a pain point for all agent systems. MemCon's consolidation and forget actions provide an automated solution—no manual cleanup; the policy learns on its own when to slim down.

Honest Limitations

• State design is manual: the policy is learned, but the state features \(\phi\) are hand-crafted. Different tasks may need different features.
• Binary feedback: reward comes only from task success/failure—too sparse for open-ended tasks without clear success criteria.
• Fixed action space: 9 preset actions. If the optimal policy requires actions outside this set, MemCon cannot discover them.
• Tabular bandit limits state space: with high-dimensional states, the table becomes sparse. The paper uses discretised state keys to mitigate this, at the cost of information loss.

Conclusion

MemCon's core insight is not an algorithmic novelty but a perspective shift: memory is not a static warehouse but a dynamic, controlled process.

This echoes "memory control" theory in cognitive science—human memory access is not a fixed pipeline either, but a process dynamically regulated by the prefrontal cortex according to current goals. We sometimes deliberately recall (retrieve), sometimes rely on habits (plan inject), sometimes search with a different cue (re-retrieve), sometimes tidy old memories (consolidate), and sometimes deliberately forget.

LLM agents are finally starting to learn this too.

---

Paper: Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents Authors: Eric Hanchen Jiang, Zhi Zhang, Yuchen Wu, Ying Nian Wu (UCLA) Code: https://github.com/ericjiang18/MemCon