CARVE: Teaching Recurrent Models to Look at Their Own Memory Before Forgetting

A Counter-Intuitive Fact

When you memorize something, your brain doesn't only look at new information—you first glance at what's already in memory, then decide whether to overwrite or ignore. But the most advanced recurrent language models don't work this way. When deciding what to forget, they only look at the incoming token, never at their own stored memory. It's like an amnesiac librarian who, upon receiving each new book, randomly discards one old book—never checking what's already on the shelves.

Sayak Dutta's paper CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention solves this. The core idea is surprisingly simple: let the model look at its own memory before deciding to forget—at almost zero computational cost.

Background: The Forgetting Dilemma of Recurrent Models

Transformers are well known to suffer from O(T²) attention complexity. Recurrent models (Mamba, GDN family) take another path: they compress all history into a fixed-size state matrix S, updated with each new token, giving constant-cost inference regardless of sequence length.

But compression means forgetting. GDN-2 uses a d_v × d_k matrix gate to decide how much to forget at each position. The problem: this gate is computed only from the new token and never sees the current state S.

This causes three coupled structural defects:

1. Memory-blind gating: the model decides what to erase without knowing what it stored. 2. Parameter waste: the write gate is a d_v-dimensional projection, as large as the value projection itself, while experiments show a scalar suffices. 3. Training efficiency killer: value-axis coupling in the erase gate mathematically blocks the use of WY-form triangular-block solvers—the key tool that makes recurrent model training as fast as Transformers. Without it, training degrades to serial-recurrence speed.

CARVE's Solution: One Principle Solves Three Problems

CARVE's core insight is an architectural constraint: restrict all gating to the key axis.

• Key-axis gating doesn't depend on value indices, so the intra-block coupling matrix is value-independent—the WY-form triangular solver is restored intact.
• Key-axis gating parameters scale as d_k instead of d_v × d_k—a large reduction.
• Most importantly, key-axis gating can be made content-aware.

Content-Aware Erasure: A Zero-Cost Look at Memory

Reading the state matrix S directly would double memory bandwidth—unacceptable. CARVE's trick: reuse the recurrent output. The kernel already computes o = S·q as output and writes it back to HBM. CARVE takes this tensor, computes a block mean m_c, and feeds it to the erase gate through a zero-initialized low-rank projection.

1. Zero extra memory traffic: o is already written to HBM; reading it adds no traffic. 2. Zero initialization: at the start of training U_b = 0, so CARVE is bit-identical to the baseline—initialization stability is preserved. 3. Intra-block delay: m_c uses the current block's average output, introducing one block of delay. The paper proves this perturbation is only O(1/√L), with measured deviation stable at 0.18% across all chunk lengths.

Scalar Write Gate: Less Is More

GDN-2 uses a d_v-dimensional vector as write gate. CARVE replaces it with a scalar w_{h,t} per head. With H=12, d_v=768, write-gate parameters drop from 589,824 to 9,216—a 64x reduction. The paper also proves the scalar gate is lossless for single-slot associative recall (Theorem 15).

Experimental Results: Full Improvement at Zero Cost

Trained at 1.3B parameters on 100B FineWeb-Edu tokens on NVIDIA H100, averaged over three seeds:

• Language modeling: WikiText perplexity 15.72 vs baseline 15.90 (−0.18, a 4.5σ cross-seed effect). The hybrid variant reaches 15.41 vs 15.62.
• Commonsense reasoning: +0.63 pp average zero-shot accuracy across 9 benchmarks, leading all recurrent models.
• In-context retrieval (RULER): records on all S-NIAH and MK-NIAH context lengths; first place on all 6 real-world recall benchmarks.
• Hardware efficiency: throughput within 0.4% of baseline (within measurement noise), peak memory −13%, mixer parameters −19%.

Most impressively: these gains come at no hardware cost. CARVE doesn't trade more compute for better performance—it uses a smarter architecture to achieve better performance while saving resources.

Theoretical Guarantees: Six Theorems

1. Containment hierarchy: linear attention ⊊ delta rule ⊊ scalar gating ⊊ key-axis gating ⊊ CARVE—each step a strict extension. 2. Lyapunov stability: the state matrix cannot explode. 3. Gradient flow: gradients neither vanish nor explode. 4. Expressiveness separation: CARVE is strictly stronger than key-axis-gating baselines. 5. Speed–accuracy Pareto frontier: CARVE occupies the optimal frontier. 6. Chunking boundary (Theorem 11): exactly characterizes which architectures can use WY-form chunk solvers—GDN-2 cannot, CARVE can.

A Deeper Insight

CARVE's story suggests a design philosophy: constraints are freedom.

Restricting gating to the key axis appears to reduce expressiveness—one fewer dimension. But precisely this constraint makes the WY-form solver usable again, makes content-aware gating feasible, and slashes parameters. The constraint is not a limitation; it opens a new design space.

This echoes gauge symmetry in physics: restricting a theory's symmetry group yields richer dynamics. Perhaps the next step in AI architecture design is not stacking more parameters and more complex gates, but finding those "just enough" constraints—letting simplicity itself become power.

---

Paper: CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention

Author: Sayak Dutta (independent researcher)