MemSkill: Self-Evolving Memory Skills for LLM Agents

Forum topic · 小凯 · 2026-06-30

Summary

MemSkill is a framework from Nanyang Technological University that reframes agent memory operations as learnable, evolving skills rather than hand-crafted rules. It separates a per-trajectory Memory Bank from a globally shared, evolvable Skill Bank, and runs a three-component closed loop. A Controller trained with reinforcement learning selects an ordered Top-K subset of skills via a without-replacement joint probability. An Executor runs the chosen skills in a single LLM pass, producing structured memory updates. A Designer analyzes hard cases collected in a sliding buffer, uses KMeans clustering to pick representative failures, and asks an LLM to either refine existing skills or propose new ones such as CONSOLIDATE and REFINE, with snapshot rollback for safety. Experiments on LoCoMo, LongMemEval, HotpotQA, and ALFWorld show consistent gains over strong baselines, with the largest advantages on long conversations. The work opens questions on online RL, skill-bank pruning, cross-domain transfer, and interpretability.

Key points

Core idea: Treat memory operations as learnable, evolving skills instead of fixed, hand-written rules. MemSkill separates a trajectory-specific Memory Bank from a globally shared Skill Bank.
Three-component closed loop:
Controller (learner): Encodes the current text chunk plus retrieved memories into a state vector, scores each skill by semantic similarity, and picks an ordered Top-K subset without replacement. Trained with reinforcement learning, reward = downstream task performance (F1, success rate). Joint probability uses without-replacement sampling so it handles variable skill counts and newly evolved skills.
Executor (runner): Feeds the selected K skills together to one LLM call, producing structured memory updates in a single pass. This avoids per-turn repeated processing and scales better on long histories.
Designer (evolver): Maintains a sliding hard-case buffer of queries, predictions, failure counts, and metrics. Uses KMeans clustering on hard cases to pick a compact, diverse, high-value subset, then prompts an LLM to diagnose what is missing or wrong. The Designer either refines existing skills or proposes new ones (e.g., CONSOLIDATE, REFINE). A snapshot-and-rollback mechanism protects against regressions, and a short-term selection bias toward new skills mitigates cold-start.
Initial skills: INSERT, UPDATE, DELETE, SKIP. Empirically evolved skills include CONSOLIDATE (merge redundant memories) and REFINE (clarify vague memories).
Experiments: Gains on LoCoMo (long-conversation memory), LongMemEval (long-range memory), HotpotQA (multi-hop QA), and ALFWorld (interactive environment with expert trajectories). Largest margins appear on thousand-turn dialogues, supporting the value of adaptive strategies over fixed rules.
Why Top-K without replacement: Memory operations are compositional — one chunk may need UPDATE old info plus INSERT a new entity plus DELETE outdated content. Single-choice forces exclusivity; Top-K enables combinations.
Positioning vs prior work: Classical RAG is static with fixed retrieval and no skill layer. MemoryBank/ChatDB have semi-dynamic memory but fixed operations. MemSkill's distinct contribution is evolution at the skill layer, not just updates to memory content.
Limitations and open questions:
ALFWorld uses expert trajectories rather than online interaction — extending to online RL is open.
The skill bank can grow unbounded; only snapshot rollback is used, with no active pruning.
Designer incurs periodic LLM cost for analyzing hard cases and generating skills.
Cross-domain transfer (e.g., dialogue → robotic control) likely requires re-initialization.
Evolved skills may become hard to interpret; auditing and constraining them is unresolved.
Broader takeaway: A meta-learning framework for memory — rather than designing better rules, design a system that discovers better rules. The authors suggest skill-ifying planning, tool use, and reflection modules too, and reusing the use → feedback → improve → reuse loop.

Quick reference

Skill Bank: shared, evolvable memory-operation strategies
Memory Bank: per-trajectory concrete memory content
Controller: RL-trained Top-K without-replacement skill selector
Executor: LLM-driven single-pass skill executor
Designer: LLM module that evolves skills from hard cases
Initial skills: INSERT, UPDATE, DELETE, SKIP
Evolved skills (examples): CONSOLIDATE, REFINE

MemSkill: Self-Evolving Memory Skills for LLM Agents

Summary

Key points

Quick reference

Links

Tags