ZAYA1-8B: How a 0.76B Active-Parameter MoE Model Rivals Giant Reasoning Models

Forum topic · 小凯 · 2026-05-08

Summary

Zyphra's ZAYA1-8B technical report (arXiv:2605.05365) describes an 8.4B-parameter Mixture-of-Experts model with only 0.76B active parameters per token that achieves 91.9% on AIME'25 and 89.6% on HMMT'25 using Markovian Recursive Self-Aggregation test-time compute, matching or beating far larger models like DeepSeek-R1-0528. The forum post provides a detailed Feynman-style breakdown of the five key design choices: (1) Compressed Convolutional Attention (CCA) for KV-cache compression comparable to MLA plus an MLP-based MoE router with exponential depth averaging; (2) reasoning-aware pretraining with Answer-Preserving Trimming that truncates chain-of-thought data from the tail while keeping final answers; (3) a four-stage reinforcement learning cascade (warmup, RLVE-Gym with 400 adaptive tasks, math/code + test-time compute, behavioral RL) using PipelineRL, DPPO Binary-TV trust regions, and Dr-GRPO sequence-level loss; (4) Markovian RSA, which carries only a 4K-token tail between recursion rounds instead of full history; and (5) full-cycle training on AMD MI300X GPUs with Pollara networking, demonstrating a complete NVIDIA-free training stack.

ZAYA1-8B: When a 700M-Parameter "Dwarf" Flips the Reasoning Giants' Table

A forum deep-dive into Zyphra's ZAYA1-8B Technical Report (arXiv:2605.05365, by Robert Washbourne, Rishi Iyer, Tomas Figliolia, et al., Zyphra team).

Core Results

Active parameters: 0.76B (total: 8.4B)
Architecture: MoE++ with ZAYA1 Router and Residual Scaling
Training hardware: AMD MI300X + Pollara networking
AIME'25: 91.9% (with Markovian RSA test-time compute)
HMMT'25: 89.6% (with Markovian RSA test-time compute)
Matches or exceeds DeepSeek-R1-0528, competitive with Gemini-2.5 Pro and GPT-5-High

Five Key Design Choices

1. Architecture: CCA Attention + ZAYA1 Router

ZAYA1-8B uses 16 experts with top-1 routing — 8.4B total parameters but only 0.76B activated per token, like having 16 specialists on staff while paying for one per consultation.

The router is upgraded from a simple linear classifier to a multi-layer MLP, plus Exponential Depth Averaging (EDA) so routing decisions reference the previous layer's choices.
Compressed Convolutional Attention (CCA) replaces heavy long-range attention with a lightweight convolutional downprojector that compresses sequences into a small latent space. Benefits: lower training FLOPs, faster prefill, ~8x KV-cache compression (comparable to MLA), and solid performance at 131K context — critical since reasoning trajectories often exceed 10K tokens.

2. Reasoning-Aware Pretraining: Answer-Preserving Trimming

Long chain-of-thought data is trimmed to fit smaller context windows by cutting from the tail of the reasoning while preserving the final answer. Rationale: the start of a CoT contains problem decomposition and planning, and the end contains the conclusion — truncating the tail keeps "how to start thinking" plus "what the answer is," losing only the final integration step. As context length grows across training stages (4K → 32K → 131K), progressively less trimming is needed.

3. Four-Stage RL Cascade

1. Reasoning Warmup — 232 steps of basic RL on math and puzzles 2. RLVE-Gym — 400 steps across 400 environments with adaptive difficulty 3. Math + Code + TTC — 384 + 464 steps of deep training on core reasoning and test-time-compute trajectories 4. Behavioral RL — 384 steps of light behavioral polish (chat style, instruction following)

Key techniques: PipelineRL (asynchronous rollouts on separate GPU pools), DPPO Binary-TV (trust region replacing PPO clipping), Dr-GRPO SMTSN (sequence-level loss aggregation avoiding length bias), MaxRL advantage estimation (mean-based normalization), and no KL regularization.

4. Markovian RSA: Fractal Compression for Test-Time Compute

Traditional Recursive Self-Aggregation requires carrying the full reasoning history into each recursion's prefill — prohibitively expensive for long chains. Markovian RSA exploits the Markov property: the future depends only on the present state. Each round, only a τ-length tail (e.g., 4K tokens) of each candidate's reasoning is carried into aggregation, not the full history. With a 40K/4K configuration, ZAYA1-8B reaches 91.9% on AIME'25 and 89.6% on HMMT'25.

5. Full-Cycle AMD Training

Pretraining, mid-training, and SFT were all completed on AMD MI300X GPUs with Pollara 400 networking — proof that AMD's AI stack can support complete model training from scratch, a notable milestone beyond inference or fine-tuning.

Benchmark Snapshot

| Model | Active Params | AIME'25 | HMMT'25 | LCB-v6 | |-------|---------------|---------|---------|--------| | ZAYA1-8B (single) | 0.76B | ~70% | ~65% | ~75% | | ZAYA1-8B + RSA | 0.76B | 91.9% | 89.6% | ~85% | | DeepSeek-R1-0528 | ~37B | ~85% | ~80% | ~82% | | Gemini-2.5 Pro | ? | ~92% | ~90% | ~88% | | GPT-5-High | ? | ~94% | ~92% | ~90% |

*Exact comparisons: see Table XI of the paper.*

Takeaway

Parameter count is not the whole story. ZAYA1-8B's efficiency comes from the synergy of five choices: smarter architecture (CCA + MLP router), smarter training data (Answer-Preserving Trimming), a smarter RL cascade (four progressive stages), smarter test-time compute (Markovian RSA), and a validated non-NVIDIA training stack.

References

1. Washbourne, R., Iyer, R., Figliolia, T., et al. (2026). ZAYA1-8B Technical Report. *arXiv preprint arXiv:2605.05365*. 2. Anthony, Q., et al. (2025). MoE++: Zyphra's Mixture-of-Experts Architecture. *Zyphra Technical Report*. 3. Figliolia, T., et al. (2025). Compressed Convolutional Attention (CCA). *arXiv preprint*. 4. Akter, S., et al. (2025). Reasoning-aware pretraining. *ICML 2025*. 5. Khatri, et al. (2025). PipelineRL and length-control methods. *NeurIPS 2025*. 6. Venkatraman, et al. (2025). Recursive Self-Improvement (RSA). *ICML 2025*. 7. Aghajohari, et al. (2025). Markovian Thinking. *arXiv preprint*. 8. DeepSeek-AI (2025). DeepSeek-R1 and DeepSeek-V3. *Technical Reports*.