RWKV-7 Goose: Constant-Memory Sequence Modeling Challenging Transformers

Forum topic · ✨步子哥 · 2025-10-07

Summary

RWKV-7 'Goose' is a novel sequence modeling architecture that challenges transformer limitations with constant memory usage and constant per-token inference time. Its core innovation is a Generalized Delta Rule that dynamically updates a state matrix with data-dependent decay, allowing it to exceed the TC0 expressivity class of standard transformers and recognize all regular languages. Trained on the 5.6-trillion-token RWKV World v3 dataset (released under Apache 2.0), the 2.9B-parameter RWKV7-World3-2.9B model sets a new state of the art at the 3B scale on multilingual benchmarks (61.1% average accuracy vs. Qwen2.5-3B's 55.6% and Llama3.2-3B's 58.1%), including best results on LAMBADA Multilingual, XCOPA, XNLI, and XStoryCloze. On English benchmarks it matches larger models trained on 15-18 trillion tokens, achieving a 71.5% average despite using only about a third of the data. A modified variant, RWKV-7a, demonstrates strong board-game state tracking and Alpha-Beta pruning in Reversi, with win rates improving as search budget grows. Evaluated on post-January-2025 novel internet data, its compression rate (7.74%) approaches that of far more heavily trained rivals. Open models, datasets, and code are available on Hugging Face and GitHub, positioning RWKV-7 as an efficient alternative for long-sequence, multilingual language modeling.

RWKV-7 Goose: From Transformers to Constant-Memory Sequence Modeling

> This article explores RWKV-7 "Goose", a new sequence modeling architecture that challenges the limitations of traditional transformers with constant memory usage and per-token inference time. We cover its architectural design, benchmark performance, and implications for future language models.

Introduction: The Limits of Transformers

Traditional transformers rely on softmax self-attention, whose memory and compute requirements grow quadratically with sequence length. RWKV-7 "Goose", built on recurrent neural network (RNN) principles, offers constant memory usage and linear-time inference, making it a lightweight alternative for long sequences.

Core Engine: The Generalized Delta Rule

RWKV-7's key innovation is a Generalized Delta Rule for state updates:

S_t = S_{t-1} (\text{diag}(w_t) - \hat{\kappa}_t^T (a_t \odot \hat{\kappa}_t)) + v_t^T \tilde{k}_t

Here, $$S_t$$ is the state matrix at time step $$t$$ , $\text{diag}(w_t)$ is a data-dependent vector-valued decay, $\hat{\kappa}_t$ and $$a_t$$ control removal and replacement of state information, and $v_t^T \tilde{k}_t$ adds new information. This dynamic, input-dependent transition matrix lets RWKV-7 exceed the TC $$^0$$ complexity class of transformers and enter NC $$^1$$ (Appendix D of the paper), enabling it to recognize all regular languages.

The RWKV World v3 Dataset

The model was trained on the RWKV World v3 dataset containing 5.6 trillion tokens spanning English, code, and multilingual data—far less than the 15-18 trillion tokens used by top competitors like Qwen2.5 and Llama3.2. The dataset is open-sourced under Apache 2.0.

Benchmark Performance

English Tasks (2.9B-parameter RWKV7-World3-2.9B)

| Model | Train tokens (T) | LAMBADA | Hellaswag | PIQA | ARC-E | ARC-C | Winogrande | MMLU | Avg | |---|---|---|---|---|---|---|---|---|---| | Qwen2.5-3B | 18.0 | 67.1 | 73.5 | 78.6 | 77.4 | 45.0 | 68.5 | 65.7 | 71.4 | | Llama3.2-3B | 15.0 | 70.5 | 73.6 | 76.7 | 74.5 | 42.2 | 69.9 | 56.5 | 67.8 | | RWKV7-World3-2.9B | 5.6 | 73.4 | 76.4 | 79.7 | 81.0 | 48.7 | 72.8 | 55.0 | 71.5 |

With roughly a third of the training data, RWKV-7 matches Qwen2.5-3B's average accuracy and beats Llama3.2-3B on most tasks.

Multilingual Tasks

| Model | LAMBADA Multi (ppl↓) | XCOPA | XNLI | XStoryCloze | xWinogrande | Avg | |---|---|---|---|---|---|---| | Qwen2.5-3B | 36 | 59.0 | 38.5 | 59.6 | 79.8 | 55.6 | | Llama3.2-3B | 30 | 58.5 | 44.2 | 60.6 | 79.2 | 58.1 | | RWKV7-World3-2.9B | 18 | 63.1 | 45.4 | 64.7 | 82.4 | 61.1 |

RWKV-7 achieves a new SoTA at the 3B scale for multilingual performance.

Novel-Data Evaluation

Evaluated on fresh internet data generated after January 2025 (arXiv papers, GitHub code, news), RWKV-7 2.9B achieves an average compression rate of 7.74%, close to Qwen2.5-3B (7.66%) and Llama3.2-3B (7.57%), despite far less training data.

Board-Game Modeling: RWKV-7a

An extended variant, RWKV-7a, modifies the state update:

S_t = S_{t-1} \text{diag}(w_t)(I - c \hat{\kappa}_t^T (a_t \odot \hat{\kappa}_t)) + v_t^T k_t

With $$c = 2$$ , transition-matrix eigenvalues span (-1, 1), boosting expressivity. Trained on 6 million samples, RWKV-7a shows strong Reversi board-state tracking and Alpha-Beta pruning, with win rates rising as search budget (depth and width) increases—evidence of test-time scaling.

Speed and Memory

Thanks to constant memory usage and per-token inference time, RWKV-7 achieves near or better-than-transformer performance at lower FLOPs, making it well-suited to long-sequence workloads.

Theoretical Significance

The generalized delta rule allows RWKV-7 to recognize all regular languages, which transformers constrained to TC $$^0$$ cannot. The proof hinges on the non-diagonal, input-dependent transition matrix enabling complex "copying" operations.

Conclusion

RWKV-7 "Goose" demonstrates that efficient, constant-memory architectures can rival heavily trained transformers, particularly in multilingual and long-sequence settings. Future work targets training stability, larger datasets, and multimodal/real-time applications.

Models & datasets: https://huggingface.co/RWKV
Code: https://github.com/RWKV/RWKV-LM

References

1. Peng, B., et al. (2025). RWKV-7 "Goose" with Expressive Dynamic State Evolution. arXiv:2503.14456v2. 2. Vaswani, A., et al. Attention is All You Need. arXiv:1706.03762. 3. Schlag, I., et al. (2021). DeltaNet: A Linear Attention Architecture for Efficient Sequence Modeling. arXiv:2103.06314. 4. Gao, L., et al. (2020). The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00027. 5. Qwen (2025). Qwen2.5: A Step Forward in Language Modeling. Technical Report.