English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

CARVE: Teaching Recurrent Models to Look at Their Own Memory Before Forgetting

Forum topic · ✨步子哥 · 2026-06-26

Summary

CARVE (Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention) is a new architecture by Sayak Dutta that fixes a core blind spot in recurrent language models: their erase gates decide what to forget based only on the incoming token, ignoring the contents of the recurrent state. CARVE restricts all gating to the key axis, which simultaneously restores compatibility with WY-form chunk-parallel training solvers, cuts gate parameters dramatically (a scalar write gate replaces a d_v-dimensional vector, reducing write-gate parameters 64x), and enables content-aware erasure at near-zero cost by reusing the recurrent output o = S·q already written to HBM. A zero-initialized low-rank projection of the block-mean output feeds the erase gate, introducing only one-block latency with measured deviation of about 0.18%. Trained at 1.3B parameters on 100B FineWeb-Edu tokens, CARVE improves WikiText perplexity (15.72 vs 15.90 baseline), tops RULER and six recall benchmarks among recurrent models, reduces peak memory 13% and mixer parameters 19%, all within 0.4% of baseline throughput. Six formal theorems support the design, including Lyapunov stability and a precise characterization of chunking compatibility.

A Counterintuitive Fact

When humans memorize things, the brain doesn't just look at new information—it first glances at what's already stored before deciding whether to overwrite. If you hear a new phone number, you first consider whether you've already memorized one, then decide to replace or ignore it.

But today's most advanced recurrent language models don't work this way. When deciding "what to forget," they only look at the incoming token, never at what they've already stored. It's like an amnesiac librarian who randomly discards an old book whenever a new one arrives—never checking what's on the shelves.

Sayak Dutta's paper, *CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention*, addresses this. The core idea is surprisingly simple: let the model glance at its own memory before deciding what to forget—and this glance costs almost nothing computationally.

Background: The Forgetting Dilemma of Recurrent Models

Transformers are well known for their O(T²) attention complexity, which slows down as sequences grow. Recurrent models (e.g., Mamba, GDN family) take another path: they compress all history into a fixed-size state matrix S, updated with each new token, giving constant-cost inference independent of sequence length.

But compression means forgetting. GDN-2, one of the strongest recurrent architectures, uses a d_v × d_k matrix gate to decide how much to forget at each position. The problem: this gate is computed from the new token only, with no regard for what's stored in the current state S.

This causes three coupled structural defects:

1. Memory-blind gating: the model decides what to erase without knowing what it stores—erasing important information or keeping useless information. 2. Parameter waste: the write gate is a d_v-dimensional vector projection, as large as the value projection itself, though experiments show a scalar suffices. 3. A training-efficiency killer: the value-axis coupling inside the erase gate mathematically blocks the use of WY-form triangular block solvers—the key tool that makes recurrent model training as fast as Transformers. Without it, training degrades to serial-recurrence speed.

CARVE's Solution: One Principle Solves Three Problems

CARVE's core insight is an architectural constraint: restrict all gating to the key axis.

This simple constraint solves all three problems at once:

  • Key-axis gating doesn't depend on value indices, so the intra-block coupling matrix is value-independent—the WY-form triangular solver is restored intact.
  • Key-axis gating has d_k parameters instead of d_v × d_k—a large reduction.
  • Most importantly, key-axis gating can be made content-aware.
  • Content-Aware Erasure: A Zero-Cost Glance at Memory

    How can the gate see the memory? The naive idea is to read the state matrix S—but that would double memory bandwidth, which is unacceptable.

    CARVE's cleverness: reuse the recurrent output. The recurrent kernel already computes o = S·q as its output, and this tensor must be written back to HBM anyway. CARVE takes it, computes a block mean m_c, and feeds it to the erase gate through a zero-initialized low-rank projection.

    Several elegant properties:

    1. Zero extra memory access: o is written to HBM regardless; reading it adds no traffic. 2. Zero initialization: with U_b = 0 at training start, CARVE is bit-identical to the baseline—initialization stability is preserved. 3. Intra-block latency: m_c uses the current block's average output, introducing one block of latency. But the paper proves the gating perturbation from this delay is only O(1/√L), and measured deviation stays stable at 0.18% across all chunk sizes.

    Scalar Write Gate: Less Is More

    GDN-2 uses a d_v-dimensional vector as the write gate. CARVE replaces it with a scalar w_{h,t} (one per head). In a configuration with H=12, d_v=768, write-gate parameters drop from 589,824 to 9,216—a 64x reduction. The paper also proves that for single-slot associative recall, the scalar gate is lossless (Theorem 15).

    Experimental Results: Comprehensive Gains at Zero Cost

    Trained at 1.3B parameters on 100B FineWeb-Edu tokens on NVIDIA H100s, averaged over three seeds:

  • Language modeling: WikiText perplexity 15.72 vs baseline 15.90, a drop of 0.18 (a 4.5σ cross-seed effect). A hybrid variant improves further to 15.41 vs 15.62.
  • Commonsense reasoning: +0.63 pp average zero-shot accuracy across 9 benchmarks, leading all recurrent models.
  • In-context retrieval (RULER): records on all S-NIAH and MK-NIAH context lengths; first place on all six real-world recall benchmarks.
  • Hardware efficiency: throughput within 0.4% of the baseline (measurement noise), peak memory −13%, mixer parameters −19%.
Most impressive: all these improvements come with no hardware cost. CARVE doesn't trade more computation for better performance—it achieves better performance with a smarter architecture while using fewer resources.

Theoretical Guarantees: Six Theorems

CARVE is not just an engineering tweak; it is backed by six formal theorems:

1. Containment hierarchy: linear attention ⊊ delta rule ⊊ scalar gating ⊊ key-axis gating ⊊ CARVE—each step is a strict extension. 2. Lyapunov stability: the state matrix cannot explode. 3. Gradient flow: gradients neither vanish nor explode. 4. Expressiveness separation: CARVE is strictly stronger than the key-axis-gated baseline. 5. Speed–accuracy Pareto frontier: CARVE occupies the optimal frontier. 6. Chunking boundary (Theorem 11): an exact characterization of which architectures can use WY-form chunk solvers—GDN-2 cannot, CARVE can.

A Deeper Insight

CARVE's story suggests a design philosophy: constraints are freedom.

Restricting gating to the key axis seems to reduce expressiveness—one dimension fewer. But precisely this constraint makes the WY-form solver usable again, makes content-aware gating feasible, and slashes parameter count. The constraint is not a limitation; it opens a new design space.

This echoes gauge symmetry in physics: restricting a theory's symmetry group yields richer dynamics. CARVE restricts the dimensionality of gating and obtains a stronger model.

Perhaps the next step in AI architecture design is not stacking more parameters and more complex gates, but finding those "just enough" constraints—letting simplicity itself become strength.

---

Paper: CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention

Author: Sayak Dutta (independent researcher)

Tags

#carve#recurrent-neural-networks#linear-attention#state-space-models#gdn#chunk-parallel-training#memory-gating#efficiency

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178208156