M*: A Self-Evolving Memory Harness — Every Task Deserves Its Own Memory Architecture

Background and Motivation: Why a Universal Memory Is Elusive

LLM-powered agents performing long-horizon tasks rely on memory systems to accumulate and reuse knowledge. However, mainstream practice uses a one-size-fits-all fixed memory design — semantic retrieval (vector databases) optimized for conversation, or skill libraries for coding. This "universal memory" paradigm tends to fail across tasks: a memory system optimized for one purpose rarely transfers to another.

The reason is that different tasks demand fundamentally different memory capabilities:

• Chat dialogue: needs semantic retrieval of history based on context; fuzzy matching via vector databases works well here.
• Household planning (e.g., robots executing home tasks): needs precise state tracking (e.g., "which room's light is already off") — structured queries like SQL far outperform fuzzy vector retrieval.
• Medical QA: requires extracting structured fields (symptoms, medications, dosages) from dialogue with exact matching, demanding structured extraction and field-level comparison.

The paper (attributed to Microsoft and City University of Hong Kong researchers) states explicitly that a memory system optimized for one purpose often fails at other purposes — fundamentally questioning the feasibility of a single "universal" memory architecture.

The M* Method: Letting AI Evolve Its Own Memory Code

M* addresses this by automatically discovering task-optimized memory architectures (Memory Harnesses) through executable program evolution. The core idea: don't have humans design the memory system — let the AI write and optimize the memory management code itself.

Concretely, M* models the memory system as a Python program encapsulating:

• the memory data schema
• the storage logic
• agent workflow instructions

M* optimizes this program via Reflective Code Evolution: a population-based search strategy evaluates candidate memory programs on training tasks, analyzes failure cases, and has an LLM act as a "reflector" proposing improvements — iteratively modifying and re-testing the code.

After evolution, each task yielded a structurally distinct memory program:

• ALFWorld (household planning): evolved a memory program with a cached SQL reader/writer for precise environment state tracking.
• LoCoMo (long conversation): evolved a hybrid architecture combining vector retrieval with relational tables, balancing semantic and structured queries.
• HealthBench (medical QA): evolved structured information extraction plus a field matcher, extracting key entities and attributes and comparing them field by field.

Experimental Results: The Power of Task-Customized Memory

M* was evaluated on four benchmarks spanning dialogue, planning, and reasoning: LoCoMo (long conversation), ALFWorld (household planning), HealthBench (medical QA), and PRBench (expert reasoning).

Key findings:

• M* beat all fixed memory baselines — including carefully human-designed memory schemes — in 7 of 8 configurations.
• The evolution process used no domain-specific prior knowledge; it was simply instructed to "write a memory management program" and found effective solutions through reflection and iteration.
• Evolved memory programs exhibited "structurally different processing mechanisms in every domain," showing that specializing memory mechanisms per task unlocks a broader design space than any universal paradigm.

Implications: Universal Memory Is a False Premise — Let AI Find the Answer

M*'s real value is the fact it reveals: a universal memory architecture is a false premise. Evidence shows the optimal memory scheme differs per task, so hunting for one "universal" system is futile. The good news: humans don't need to design these schemes one by one — AI can find them automatically via code evolution.

For anyone building LLM agents, the lesson is: rather than agonizing over "which vector database to use," first ask "what memory structure does my task actually need?" M* provides an automated answer to that question, pointing toward a task-driven, adaptive memory paradigm as a new direction for LLM memory system design.