Memory-R1 vs Memento: How RL-Trained Memory Management Is Reshaping LLM Agents
This post explores how agentic reinforcement learning (RL) enables LLM agents to decide what to remember and what to forget, with a focus on budget-conscious research directions for those moving beyond LoRA fine-tuning.
The Problem: Memory Management for LLM Agents
LLM agents face a fundamental limitation: context windows are finite, and long-term memory evaporates easily. Traditional RAG (Retrieval-Augmented Generation) fetches information from external databases but ignores memory dynamics: when to add new entries, when to update old ones, and when to delete useless ones. This has driven researchers toward agentic RL, where agents learn memory operations through trial and error in a partially observable Markov decision process (POMDP).
Memory-R1: RL-Based Memory Management
Memory-R1 (arXiv:2508.19828) trains a dedicated memory manager to execute structured actions — ADD, UPDATE, DELETE, or NOOP — on an external memory bank, like a gardener pruning a knowledge garden. Key details:
- Fine-tuned via PPO or GRPO (Group Relative Policy Optimization), a variance-reduced RL method that avoids training a full value network — requiring only forward passes and cutting sample needs by ~50%.
- Highly data-efficient: trained with just 152 QA pairs distilled from the LoCoMo benchmark, on models from 3B to 14B parameters (e.g., LLaMA-3.1-8B, Qwen-2.5).
- Achieves LoCoMo SOTA: F1 68.9%, BLEU-1 48.3%, outperforming Mem0 by up to 48% on multi-hop tasks.
- Compute: roughly 4-8 H100 GPUs (batch size 128, token limits 4096/2048) — a one-time investment with zero ongoing API costs once self-hosted.
- Entropy collapse: RL policies can become overly deterministic, killing exploration. Memory-R1 mitigates this with GRPO; monitoring policy entropy \(H(\pi) = -\sum \pi(a|s) \log \pi(a|s)\) is recommended, potentially with KL-divergence regularization.
- Retrieval noise: Memento's similarity matching can misfire on noisy cases.
- Hybrid approaches — RL-constructed memory + non-parametric storage — may be the winning combination.
The Broader Landscape
Memory-R1 is a pioneer, not the only paradigm. Alternative approaches include:
| Method | Training Paradigm | Compute Needs | Cost-Friendliness | Highlights | |---|---|---|---|---| | Memory-R1 | RL fine-tuning (PPO/GRPO); 152 QA pairs | 4-8 H100 GPUs | High (data-efficient, zero API cost) | LoCoMo SOTA (F1 68.9%); +48% multi-hop vs Mem0 | | Memento | Memory-augmented MDP; online Q-learning, no LLM gradients | External ops + small MLP; heavy API tokens | Medium (API costs accumulate) | GAIA 87.88% Pass@3; +9.6% OOD continuous learning | | Mem0 (Heuristic) | Rule-driven operations | Minimal, no training | Very high (plug-and-play) | Baseline-level; 48% lower LoCoMo F1 | | A-MEM | Agentic linked-memory evolution | Medium | High | Solid on multi-hop/temporal tasks | | Mem-α | RL-built memory systems | Similar to Memory-R1 | Medium-high | Significant gains on long-horizon reasoning |
Other notable work: MemAgent (token-level RL for long-context management), MemGPT and HippoRAG (neuroscience-inspired heuristic hierarchies).
Memento: Fine-Tuning Agents Without Fine-Tuning LLMs
Memento (arXiv:2508.16153) freezes LLM parameters and trains only a Case Bank retriever via online soft Q-learning, using non-parametric similarity matching or a parametric MLP. It reaches 87.88% Pass@3 on GAIA with +4.7-9.6% improvement on out-of-distribution tasks, and saturates at just 3k cases.
The catch: reproduction depends heavily on GPT-4 API calls — complex tasks can consume up to 121k input tokens, with costs snowballing. Memory-R1's one-time RL investment is more stable for independent researchers.
Caveats and Trade-offs
Practical Recommendations
1. Start small: reproduce a LoCoMo subset with Memory-R1's approach on cloud GPUs using Qwen-2.5-7B; results within days. 2. Try Memento's free GitHub repo with free-tier API (e.g., GPT-4o-mini) to quantify token costs. 3. Consider a hybrid "RL-Memento": RL-refined memory operations feeding a non-parametric case store. 4. Open research directions per the agentic RL survey (arXiv:2509.02547): multi-agent memory sharing and vision-LLM memory integration.
References
1. Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning — arXiv:2508.19828 2. Memento: Fine-tuning LLM Agents without Fine-tuning LLMs — arXiv:2508.16153 3. The Landscape of Agentic Reinforcement Learning for LLMs: A Survey — arXiv:2509.02547 4. A-MEM: Agentic Memory for LLM Agents — arXiv:2502.12110 5. MEM-α: Learning Memory Construction via Reinforcement Learning — OpenReview
Verdict: For compute-constrained researchers, Memory-R1 is the better fit — deep RL adaptability with zero marginal cost after training, while Memento's elegance comes with recurring API expense.