RWKV-7 "Goose": From Transformers to Constant-Memory Sequence Modeling

This article explores RWKV-7 "Goose", a new sequence modeling architecture that challenges the limitations of traditional Transformers with constant memory usage and constant per-token inference time. Through vivid analogies, it reveals RWKV-7's innovative architecture, strong performance, and significance for the future of language models.

Introduction: Language Models' Interstellar Voyage

Traditional Transformers dominate NLP thanks to powerful parallel computation and context handling. However, their memory demand and computational complexity grow quadratically with sequence length — like a spaceship whose fuel tank keeps expanding until it slows the voyage.

RWKV-7 "Goose" is a new-generation light cruiser: constant memory footprint and per-token inference time. It sets new SoTA at the 3B-parameter scale on multilingual tasks and rivals top models on English downstream tasks, despite training on far less data.

Core Engine: The Generalized Delta Rule

Architectural Innovation: From Transformers to RNNs

Softmax attention computes relationships between every pair of tokens, driving memory needs quadratically with sequence length. RNNs offer constant memory and linear complexity. RWKV-7 introduces a Generalized Delta Rule, retaining RNN efficiency while greatly boosting expressiveness. Its state update formula:

Here \(S_t\) is the state matrix at the current step, \(\text{diag}(w_t)\) is a data-dependent vector-valued decay, \(\hat{\kappa}_t\) and \(a_t\) control removal and replacement of state, and \(v_t^T \tilde{k}_t\) adds new information. The state matrix updates dynamically at every step, enabling RWKV-7 to exceed the TC\(^0\) complexity limits of Transformers and enter the NC\(^1\) domain (Appendix D of the paper).

RWKV World v3 Dataset: A Multilingual Treasure Trove

To power RWKV-7, the team built the 3.1-billion-token RWKV World v3 dataset, covering English, code, and multilingual data — narrowing the gap with modern LLMs that use 15-18 trillion tokens. Diverse sources (English, Chinese novels, code, etc.) strengthen multilingual ability. The dataset is open-sourced under Apache 2.0, with details in Appendix B.

Performance: Battling the Giants

English Tasks

RWKV7-World3-2.9B on English benchmarks:

| Model | Tokens (T) | LAMBADA | Hellaswag | PIQA | ARC-E | ARC-C | GLUE | Winogrande | SciQ | MMLU | Avg | |---|---|---|---|---|---|---|---|---|---|---|---| | Qwen2.5-3B | 18.0 | 67.1 | 73.5 | 78.6 | 77.4 | 45.0 | 70.2 | 68.5 | 96.2 | 65.7 | 71.4 | | Llama3.2-3B | 15.0 | 70.5 | 73.6 | 76.7 | 74.5 | 42.2 | 50.7 | 69.9 | 95.7 | 56.5 | 67.8 | | RWKV7-World3-2.9B | 5.6 | 73.4 | 76.4 | 79.7 | 81.0 | 48.7 | 61.8 | 72.8 | 95.0 | 55.0 | 71.5 |

With only a third of the training tokens (5.6T), RWKV-7 matches Qwen2.5-3B's average accuracy (71.5%) and beats Llama3.2-3B on multiple tasks.

Multilingual Tasks

| Model | Tokens (T) | LAMBADA Multilingual (ppl↓) | XCOPA | XNLI | XStoryCloze | xWinogrande | Avg | |---|---|---|---|---|---|---|---| | Qwen2.5-3B | 18.0 | 36 | 59.0 | 38.5 | 59.6 | 79.8 | 55.6 | | Llama3.2-3B | 15.0 | 30 | 58.5 | 44.2 | 60.6 | 79.2 | 58.1 | | RWKV7-World3-2.9B | 5.6 | 18 | 63.1 | 45.4 | 64.7 | 82.4 | 61.1 |

RWKV-7 reaches 61.1% average multilingual accuracy, clearly beating both competitors.

Novel Data Evaluation

To avoid data contamination, the team evaluated compression rates on fresh internet data generated after January 2025 (arXiv papers, GitHub code, news). RWKV-7 2.9B achieved an average compression rate of 7.74%, close to Qwen2.5-3B's 7.66% and Llama3.2-3B's 7.57%, despite far less training data (Table 5 of the paper).

Board Game Modeling: Strategic Interstellar Play

RWKV-7's strong state tracking shines in board games. An extended variant, RWKV-7a, uses the update:

Setting \(c = 2\) allows state transition matrix eigenvalues in (-1,1), further enhancing expressiveness. Trained on 6 million Reversi samples, RWKV-7a shows excellent board-state tracking and Alpha-Beta pruning; as search budget (depth and width) grows, its win rate rises markedly, demonstrating test-time scaling capability.

Speed and Memory: An Efficient Engine

With constant memory and per-token inference time, RWKV-7's linear complexity gives it a clear edge on long-sequence tasks. Figures 3 and 4 of the paper show it approaches or exceeds Transformer performance with fewer FLOPs on multilingual and English benchmarks.

Theoretical Breakthrough: Beyond TC0

The Generalized Delta Rule lets RWKV-7 recognize all regular languages (Appendix D), which Transformers cannot do under TC\(^0\) complexity limits. The proof hinges on its non-diagonal, input-dependent transition matrix enabling complex "copying" operations (Lemma 3).

Conclusion: A Blueprint for the Future

RWKV-7 "Goose" is both an efficient language model and a pioneer of a new generation of sequence modeling, challenging Transformer dominance with less data and compute, and showing great potential in multilingual and long-sequence settings. Future work includes improving training stability, scaling datasets, and exploring multimodal tasks and real-time inference.

Open resources:

• Models and datasets: Hugging Face
• Code: GitHub

References

1. Peng, B., et al. (2025). RWKV-7 "Goose" with Expressive Dynamic State Evolution. arXiv:2503.14456v2. 2. Vaswani, A., et al. (2023). Attention is All You Need. arXiv:1706.03762. 3. Schlag, I., et al. (2021). DeltaNet: A Linear Attention Architecture for Efficient Sequence Modeling. arXiv:2103.06314. 4. Gao, L., et al. (2020). The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00027. 5. Qwen, et al. (2025). Qwen2.5: A Step Forward in Language Modeling. Technical Report.