CARVE: Teaching Recurrent Language Models to Peek at Their Own Memory Before Forgetting

A Counterintuitive Fact

When you memorize something, your brain doesn't just look at new information — you first glance at what's already in your head before deciding whether to overwrite. When you hear a new phone number, you first consider whether you've already memorized one, then decide to replace or ignore it.

But today's most advanced recurrent language models don't work this way. When deciding what to forget, they look only at the incoming token, never at what they've already stored. It's like an amnesiac librarian who, upon receiving each new book, randomly discards an old one — never checking what's already on the shelves.

Sayak Dutta's paper, CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention, addresses this problem. The core idea is surprisingly simple: let the model glance at its own memory before deciding what to forget. And this glance costs almost nothing computationally.

Background: The Forgetting Dilemma of Recurrent Models

Transformers have a well-known problem: attention is O(T²), so longer sequences mean slower inference. Recurrent models (like Mamba and the GDN family) take another path: they compress all history into a fixed-size state matrix S, updating it with each new token. Inference cost is constant regardless of sequence length.

But compression means forgetting. GDN-2, one of the strongest recurrent architectures, uses a d_v × d_k matrix gate to decide how much to erase at each position. The problem: this gate is computed only from the new token, with no reference to what's actually in the state S.

This creates three intertwined structural defects:

1. Memory-blind gating: the model decides what to erase without knowing what it has stored — it may wipe important information or retain useless content. 2. Parameter waste: the write gate is a d_v-dimensional vector projection, as large as the value projection itself, yet experiments show a scalar suffices. 3. Training-efficiency killer: the value-axis coupling inside the erase gate mathematically prevents the use of WY-form triangular block solvers — the key tool that lets recurrent models train as fast as Transformers. Without it, training speed degrades to a serial recurrence.

CARVE's Solution: One Principle, Three Problems

CARVE's central insight is an architectural constraint: restrict all gating to the key axis.

This sounds simple, but it solves all three problems at once:

• Key-axis gating doesn't depend on value indices, so the within-block coupling matrix is value-independent — the WY-form triangular solver is restored untouched.
• Key-axis gating requires d_k parameters instead of d_v × d_k — a large reduction.
• Most importantly, key-axis gating can be made content-aware.

Content-Aware Erasure: A Zero-Cost Glance at Memory

How can the gate see memory? The obvious idea is reading the state matrix S — but that would double memory bandwidth, which is unacceptable.

CARVE's trick: reuse the recurrent output. The recurrent kernel already computes o = S·q as its output, and this tensor must be written back to HBM anyway. CARVE takes it, computes a block mean m_c, and feeds it to the erase gate through a zero-initialized low-rank projection.

Several elegant details:

1. Zero extra memory traffic: o is written to HBM anyway; reading it adds no bandwidth. 2. Zero initialization: with U_b = 0 at the start of training, CARVE is bit-identical to the baseline — preserving initialization stability. 3. Within-block delay: m_c uses the average output of the current block, introducing one block of latency. The paper proves the gating perturbation from this delay is only O(1/√L), with measured deviation stable at 0.18% across all chunk lengths.

Scalar Write Gate: Less Is More

GDN-2 uses a d_v-dimensional vector as its write gate. CARVE replaces it with a scalar w_{h,t} (one per head). In the H=12, d_v=768 configuration, write-gate parameters drop from 589,824 to 9,216 — a 64x reduction. The paper also proves that for single-slot associative recall, a scalar gate is lossless (Theorem 15).

Results: Across-the-Board Gains at Zero Cost

Trained at 1.3B parameters on 100B FineWeb-Edu tokens on NVIDIA H100, averaged over three seeds:

• Language modeling: WikiText perplexity 15.72 vs baseline 15.90, a drop of 0.18 (4.5σ across-seed effect). The hybrid variant further improves to 15.41 vs 15.62.
• Commonsense reasoning: +0.63pp average zero-shot accuracy across 9 benchmarks, ahead of all recurrent models.
• In-context retrieval (RULER): records on all S-NIAH and MK-NIAH context lengths; first place on all 6 real-world recall benchmarks.
• Hardware efficiency: throughput within 0.4% of baseline (within measurement noise), peak memory -13%, mixer parameters -19%.

Most impressively: none of these improvements carry any hardware cost. CARVE doesn't trade more computation for better performance — it uses a smarter architecture to achieve better performance while using fewer resources.

Theoretical Guarantees: Six Theorems

CARVE isn't just an engineering improvement; it's backed by six formal theorems:

1. Containment hierarchy: linear attention ⊊ delta rule ⊊ scalar gating ⊊ key-axis gating ⊊ CARVE — each step is a strict extension. 2. Lyapunov stability: the state matrix cannot blow up. 3. Gradient flow: gradients neither vanish nor explode. 4. Expressiveness separation: CARVE is strictly stronger than key-axis gating baselines. 5. Speed-accuracy Pareto frontier: CARVE occupies the optimal frontier. 6. Chunkability boundary (Theorem 11): a precise characterization of which architectures admit WY-form chunked solvers — GDN-2 doesn't; CARVE does.

A Deeper Insight

CARVE's story suggests a design philosophy: constraints as freedom.

Restricting gating to the key axis seems to reduce expressiveness — one dimension fewer. But it is precisely this constraint that makes the WY-form solver usable again, makes content-aware gating feasible, and slashes parameter counts. The constraint isn't a limitation; it opens a new design space.

This echoes gauge symmetry in physics: by restricting a theory's symmetry group, you obtain richer dynamics. CARVE restricts gating dimensionality and gets a stronger model.

Perhaps the next step in AI architecture design is not stacking more parameters and more complex gates, but finding those "just enough" constraints — letting simplicity itself become strength.

---

Paper: CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention

Author: Sayak Dutta (independent researcher)