MemCollab: Cross-Agent Memory Collaboration via Contrastive Trajectory Distillation

*An explainer (originally written in Feynman-style Chinese for zhichai.net) of the arXiv paper "MemCollab: Cross-Agent Memory Collaboration via Contrastive Trajectory Distillation" (arXiv:2603.23234).*

The problem: AI memory is locked in silos

LLM-based agents accumulate memory - solution techniques, error lessons, tool tips - from past experience. Today each agent keeps its own memory. This causes:

• Redundant work: ten agents solving the same problem type each repeat trial-and-error independently.
• Capability boundaries: can a small model's (7B) memory help a large model (32B), or vice versa?
• Resource waste: real deployments use small models for cheap tasks and large models for hard ones; each building memory separately is expensive.

Naive sharing makes things worse

Directly copying memory between agents degrades performance:

| Task | Baseline (7B) | With directly transferred 32B memory | |------|---------------|---------------------------------------| | MATH500 | 52.2% | 50.6% | | HumanEval | 42.7% | 34.1% |

The cause is knowledge entanglement: generic solution principles are entangled with model-specific reasoning styles, tool-usage habits, and idiosyncratic failure modes. Transferring memory wholesale transfers these "personal imprints" as noise.

Core idea: contrast across agents

Just as comparing one correct and one incorrect solution to the *same* problem reveals the decisive factors, MemCollab compares trajectories from two agents:

• A weaker agent (Aw), e.g., a 7B model, and a stronger agent (As), e.g., a 32B model, each attempt every task.
• A verifier labels trajectories as preferred (τ+) or dispreferred (τ−) - either agent can produce either outcome.

Formally, a trajectory τ = g(s, b), where s is task structure (shared across agents) and b is agent bias (agent-specific). Contrastive distillation learns a memory m = φ(s) that captures what persists in success and is missing/broken in failure:

• Violation patterns: systematic errors seen in failures (e.g., premature numeric computation before forming equations; assuming independence without verification).
• Reasoning invariants: principles preserved in successes (e.g., establish variable dependencies first; systematically enumerate cases).

Memories are stored as operational rules ("do X, avoid Y"), not example solutions, making them abstract, actionable, and transferable.

Task-aware retrieval

A two-stage pipeline: (1) classify the new problem (algebra, geometry, probability, etc.); (2) rank and retrieve the top-3 relevant memories within that category, avoiding interference from irrelevant rules.

Results

| Setting | No memory | With MemCollab | Gain | |---------|-----------|----------------|------| | Qwen2.5-7B, MATH500 | 52.2% | 67.0% | +14.8% | | Qwen2.5-32B, MATH500 | 68.0% | 73.8% | +5.8% | | Qwen2.5-7B, MBPP | 47.9% | 57.6% | +9.7% | | Qwen2.5-32B, MBPP | 59.4% | 64.3% | +4.9% | | LLaMA3-8B, cross-family | 46.6% | 74.4% | +27.8% |

Additional findings:

• Cross-model contrast beats self-contrast: a model comparing its own successes and failures helps, but far less than comparing across models - perspective differences are what filter out model-specific bias.
• Efficiency: fewer reasoning rounds are needed since known dead-end paths are pruned.
• Inverted-U memory curve: performance improves with 1-3 retrieved memories, then declines beyond ~3, as excessive constraints add noise. Memory acts as search-space pruning.

Why it works

• Contrastive learning view: positive/negative trajectory pairs expose discriminative, task-level features while canceling shared task structure and isolating differing biases.
• Information view: memory distillation maximizes I(task structure) while minimizing I(model bias) and noise.
• Cognitive science echo: experts rely on structured mental models (problem type, core concepts, common pitfalls) rather than memorized steps.

Broader significance

• Small models get a leg up: with MemCollab, a 7B model can approach or exceed a memory-less 32B model - cheaper, more efficient deployments.
• Multi-agent systems: heterogeneous model teams can build a shared collective memory rather than isolated islands.
• Continual learning: shared memory can grow as agents discover new techniques and failure modes.
• Future directions: scaling beyond two agents, meta-learning (learning how to learn problem structure), self-evolving memory, and cross-modal applications (vision, robotics, scientific discovery).

Reference

Chang, Y., Wu, Y., Wu, Q., & Lin, L. (2026). MemCollab: Cross-Agent Memory Collaboration via Contrastive Trajectory Distillation. *arXiv preprint arXiv:2603.23234*.