MemT-1.0: 128K History in a Constant 10M Memory State
Yesterday (October 19), the MemTensor team posted a technical report on arXiv: MemT-1.0. It bets on a memory route completely different from the mainstream. First, the numbers: a model processing a 128K-token history with a full-context KV cache must store raw key-value caches for every step, and the bill grows with history length. MemT-1.0-Nano packs that history into a ~10M dynamic memory state — the state size is constant and does not grow with history. At 128K history, persistent storage is about 66x smaller than full-context KV cache, and end-to-end inference is about 1.86x faster. Both numbers come from the paper abstract; the GitHub repo was created the same day.
Three routes to AI memory
To understand this route, consider the two existing paths. The first stores raw tokens: KV cache keeps computed keys and values for every position — zero distillation, full retention — at the cost of storage and bandwidth growing linearly with history. The second stores text conclusions: an external memory library distills history into textual notes, retrieved when needed — the bill is controlled, but distillation loses structure and retrieval adds an extra interface.
MemT-1.0 takes a third route: storing state. Memory lives neither outside the model nor in accumulated caches; it is part of the model's computation graph. A native memory block runs alongside the backbone attention with two components: a dynamic multi-channel memory matrix that evolves across interaction steps, and a set of trainable memory read/write parts — query/key/value projections plus four gates (erase, write, forget, output). During generation, memory attention reads from the current state and, after gating, adds it to the attention output; at the end of each dialogue turn, Gated DeltaNet-2 performs token-level memory updates, compressing that turn's information into the state.
The write policy is learned
The key point is that the "writing policy" itself is trained. The Nano version has 451M dedicated trainable memory parameters — what to write, what to forget, what to read out is not hand-written engineering rules but taught by gradients. Compare: external memory libraries rely on another model writing summaries, with the distillation strategy frozen in prompts; MemT's distillation strategy lives in the weights and evolves with training data.
Models and training
The training budget is transparent: two stages (continued pre-training + post-training) on 34B tokens of dedicated data. The family has two tiers: Nano is built on Qwen3.6-35B-A3B (36B total, 3B active); Flash is built on DeepSeek-V4-Flash (288B total, 17B active) — both bases are Chinese open-source models. The team released the predecessor Metis prototype just three months ago (July); this is the formal second generation. A Mixture-of-Storage design extends memory capacity by making storage itself a mixture — routing decides which component stores what, and for how long.
The three honest caveats
Three limitations stated in the report itself, which many summaries drop:
1. On 11 memory benchmarks, Flash and Nano rank top-two among parametric memory methods — but overall they still trail strong text-based memory systems, with a clear gap on tasks requiring precise evidence retrieval over long histories. 2. Beyond memory capability, general capability is "largely preserved," not improved. 3. On agentic coding tasks, internalizing retrieved experience into the memory state performs comparably to pasting the experience text into context — not better.
Together, the honest picture: the state route has caught up with what it should catch up with, and not yet with what it wants to catch up with.
Agent experience as state
The agentic experience result deserves its own note. They let agents write retrieved procedural experience into native memory, read via memory attention at task time — the experience text never enters the task context. The carrier of experience switches from the token channel to the state channel, matching text performance at zero context occupancy. The old question of in what form experience should be reused (text loses execution detail) finally has a candidate answer: store it as state.
From the July prototype to October 1.0 — two generations in three months. A 10M state holding 128K of history; the next bottleneck is written in their own evaluation: the state route currently wins on cost and loses on retrieval precision — what text memory systems still lead at is precisely "finding that one specific piece of evidence in a long history."
Source notes: paper arXiv 2610.26825 "MemT-1.0: Towards Memory-Native Evolution" (2026-10-19, MemTensor team, Zhiyu Li et al.), abstract and full text checked; GitHub MemTensor/MemT-1.0 (repo created 10-19, README numbers: 36B/3B/451M/10M, 288B/17B/3.61B/336M, Qwen3.6-35B-A3B and DeepSeek-V4-Flash bases, 66x/1.86x@128K, 34B tokens, 11 benchmarks); predecessor Metis (GitHub MemTensor/Metis, created 2026-07-16); the caveats "top among parametric methods / behind text systems overall / comparable to text experience / general capability largely preserved" are the paper and README's own wording.