Competing Learning in Hybrid Architectures: When Attention and Recurrent Layers Fight Over the Same Job
A Counterintuitive Phenomenon
Imagine you are a new employee at a company, and a senior colleague sits next to you. The boss assigns a task—finding a key table in a pile of documents. You can both do it, but your colleague is faster and always finishes first. Over time, the boss stops assigning this type of work to you. You think it's a clear division of labor, but in reality, your document-searching ability is atrophying.
This is exactly what is happening inside hybrid-architecture language models.
In recent years, linear recurrent layers (represented by Mamba and RWKV) have been combined with softmax attention layers (represented by Transformers) to form so-called "hybrid architectures." The design intent was appealing: recurrent layers handle high throughput, compressing long-context information into a fixed-size state, while attention layers handle precise retrieval, pulling back specific information from history when needed. Each plays to its strengths.
But a paper posted on October 19 by Tsinghua University's Xingyu Shen, Yingfa Chen, and Ziqing Qiao—*Competing Learning in Hybrid Architectures*—reveals a phenomenon that has been overlooked: these two layer types are not dividing labor as designers expected; they are competing for the same job. Even more counterintuitively, recurrent layers learn faster and end up "starving" the attention layers.
Competing Learning: Whoever Learns First Suppresses the Other
The paper's core finding can be summarized in one sentence: once any layer learns a function, it suppresses the learning signal for that same function in other layers.
The authors call this "Competing Learning." It is not a design flaw but a natural consequence of gradient descent. When both recurrent and attention layers can perform "retrieval," if the recurrent layer learns it first, the loss gradient for the retrieval component shrinks—the task is already done, the error is gone, and so is the gradient signal. The attention layer's learning signal is suppressed, and it stops learning retrieval even though it could do it better.
It's like two students doing homework together: once one has written the answer, the other loses the motivation to think independently. Gradient descent doesn't distinguish "who should learn"—it only cares whether error remains.
Why Recurrent Layers Learn Faster
Through layer attribution analysis, the authors discovered a key fact: under typical natural language data distributions, recurrent layers learn retrieval functions faster than attention layers.
This seems counterintuitive at first. Isn't attention natively good at retrieval? It can extract information from historical tokens directly via attention weights, while recurrent layers can only retain information indirectly through a fixed-size state.
But "natively good at" and "learns fast" are two different things. Although recurrent layers have limited expressiveness, their update mechanism is simpler—just updating a fixed-size state—so the gradient signal is more concentrated. Attention layers must learn an entire attention matrix, with a much larger parameter space, and therefore learn more slowly.
The result: in early training, recurrent layers can already handle basic retrieval while attention layers are still fumbling. The recurrent layer's "finishing first" means the attention layer never receives sufficient gradient signal, and its retrieval capability ends up locked at a lower level.
A Simple but Effective Fix: Pausing Recurrent-Layer Training Midway
If the problem is that recurrent layers learn too fast, the solution is straightforward—pause recurrent-layer weight updates partway through training, giving attention layers a chance to learn retrieval.
It sounds like a hack, but the paper provides systematic experimental support. The authors propose a staged training procedure:
1. Stage 1: Normal training, with recurrent and attention layers updating jointly. Recurrent layers quickly learn basic retrieval. 2. Stage 2: Freeze recurrent-layer weights and update only attention layers. Now the retrieval-related gradient can only be reduced through the attention layer, which is forced to learn retrieval. 3. Stage 3: Unfreeze recurrent layers and resume joint training.
The elegance of this procedure is that it doesn't change the architecture—it changes the *rhythm* of learning. By artificially creating a window in which "only the attention layer can learn," it gives attention layers room to develop their own retrieval capability.
Experimental Results: Across-the-Board Gains at 1B Parameters
The authors validated the method on models up to 1B parameters. Results show:
- Language modeling loss: significantly reduced
- Commonsense reasoning tasks: clearly improved
- Retrieval-related tasks: improved across different context lengths
What This Means
The deeper lesson of this paper: hybrid architecture design must consider not just "what can be done" but "what will be learned".
From an architectural standpoint, the recurrent-plus-attention combination indeed possesses both capabilities—high throughput and precise retrieval. But from a training-dynamics standpoint, there is a hidden competition between them. Without intervention, the faster-learning capability suppresses the slower one, and the model's final capability structure deviates from the design intent.
This aligns with the "component–interaction decoupling" perspective: **architecture defines what components *can* do, but training dynamics determine what they *will* do. The designer's written "division of labor" is only an intention; gradient descent decides the real one.
More broadly, this finding applies to other multi-component systems. When you train multiple modules that can each perform the same task, you must account for differences in learning speed. The faster module will "eat" the gradient signal of the slower one, even when the latter could do better.
An Analogy: Why "Pausing" Beats "Adding People"
In management terms, the problem is intuitive. Two employees can both do a task: one is fast but rough, the other slow but meticulous. If the manager only checks whether tasks are completed, the fast worker takes everything and the meticulous one never develops their finer skills.
The solution is not hiring more people (scaling the model) or changing management style (changing the architecture), but temporarily pausing task assignments to the fast worker, letting the meticulous one get hands-on experience. Once the meticulous capability is built up, normal assignment resumes.
That is the essence of "pausing recurrent-layer training midway"—not punishing the recurrent layer, but giving the attention layer a chance to grow.
Limitations and Outlook
The paper has notable limitations. First, experiments only reach 1B parameters; effectiveness at larger scales remains to be verified. Second, the timing of the "midway pause" requires tuning—pausing too early harms the recurrent layer's foundational abilities, while pausing too late means the attention layer is already starved. Finally, the paper only studies competition between recurrent and attention layers; if hybrid architectures include other layer types (e.g., MoE), the competition may be more complex.
But as a mechanistic finding, the "competing learning" phenomenon is important in itself. It reminds us: in multi-component systems, capabilities are not simply additive—the learning dynamics between components determine the final capability distribution.
Next time you see a hybrid-architecture model that "should be strong by design" underperforming on certain tasks, ask one question: is one component learning so fast that it starved another?
---
Paper: Competing Learning in Hybrid Architectures Authors: Xingyu Shen, Yingfa Chen, Ziqing Qiao arXiv: https://arxiv.org/abs/2610.26830 HTML**: https://arxiv.org/html/2610.26830v1