ZAYA1-8B Technical Report: A 0.76B-Active-Parameter MoE Matching Giant Reasoning Models

Forum topic · 小凯 · 2026-05-08

Summary

ZAYA1-8B, a sparse Mixture-of-Experts model from Zyphra, packs 8.4B total parameters but activates only 0.76B per forward pass, yet achieves 91.9% on AIME'25 and 89.6% on HMMT'25 with Markovian RSA test-time compute—matching or exceeding DeepSeek-R1-0528 and approaching frontier models like Gemini-2.5 Pro and GPT-5-High. The technical report credits five coordinated design choices: (1) Compressed Convolutional Attention (CCA) for 8x KV-cache compression plus an MLP-based router with Exponential Depth Averaging; (2) reasoning-aware pretraining with Answer-Preserving Trimming to fit long chain-of-thought data into short contexts; (3) a four-stage RL pipeline covering warmup, RLVE-Gym with 400 adaptive tasks, math/code/TTC training, and behavioral tuning, using PipelineRL, DPPO Binary-TV, and Dr-GRPO; (4) Markovian RSA, a test-time compute method that recursively aggregates candidate reasoning while carrying only a 4K-token tail instead of full history; and (5) full-cycle training on AMD MI300X GPUs with Pollara networking, proving a complete non-NVIDIA training stack. This forum post provides a Feynman-style walkthrough of these innovations.

David vs. Goliath: How a 700M-Parameter 'Dwarf' Beat the Reasoning Giants

*Feynman-style deep dive into the ZAYA1-8B Technical Report, originally posted on zhichai.net.*

Paper Info

Title: ZAYA1-8B Technical Report
Authors: Robert Washbourne, Rishi Iyer, Tomas Figliolia, et al. (Zyphra)
arXiv: 2605.05365
Published: 2026-05-06

Core Numbers

Active parameters: 0.76B (of 8.4B total)
Architecture: MoE++ (Zyphra's Mixture-of-Experts)
Training hardware: AMD MI300X + Pollara network
AIME'25: 91.9% (with Markovian RSA test-time compute)
HMMT'25: 89.6% (with Markovian RSA)
Matches or exceeds DeepSeek-R1-0528 (~37B active)

The Five Key Design Choices

1. Architecture: CCA attention + ZAYA1 Router + Residual Scaling 2. Reasoning-aware training: long CoT data during pretraining + Answer-Preserving Trimming 3. Four-stage RL cascade: warmup → RLVE-Gym (400 tasks) → math/code + TTC → behavioral RL 4. Test-time compute: Markovian RSA (recursive aggregation carrying only a 4K-token tail) 5. AMD full-stack training

Act 1: MoE — Less Is More

A traditional Transformer forces every parameter to participate in every task. MoE (Mixture of Experts) instead works like a consulting firm: 16 specialist departments, where a receptionist (the router) directs each question to the 1–2 most relevant experts.

ZAYA1-8B has 16 experts with top-1 routing: each token activates exactly one expert. You get the wisdom of 16 specialists but pay for one per query. Crucially, Zyphra upgraded the router from a simple linear classifier to a multi-layer MLP — like promoting the receptionist to a seasoned dispatcher — dramatically improving routing quality. EDA (Exponential Depth Averaging) goes further: the router also references the previous layer's routing decisions, remembering "where this kind of question usually goes" instead of re-deciding from scratch.

Act 2: CCA — From Telescope to Microscope

Standard Multi-Head Attention is like a telescope: you need every detail, but at long sequences the KV-cache memory explodes. MLA and GQA put the telescope on a diet. CCA (Compressed Convolutional Attention) replaces it with a microscope: a lightweight convolutional downprojector compresses the sequence into a smaller latent space where attention is computed, then projects back.

Three benefits:

Lower training FLOPs
Faster inference prefill
~8x KV-cache compression, comparable to MLA

CCA performs well up to 131K context — essential since reasoning trajectories often exceed 10K tokens.

Act 3: Answer-Preserving Trimming

Imagine teaching a student from a 15,000-character worked solution when your classroom (context window) holds only 4,000 characters. Options: discard the problem, truncate the middle, or — ZAYA1's choice — truncate from the tail while keeping the final answer.

Why? A reasoning trace has structure: the beginning holds problem decomposition and planning; the middle holds attempts; the tail holds final integration. Tail-trimming preserves "how to start thinking" and "what the answer is," losing only the final consolidation. As training progresses (4K → 32K → 131K context), progressively less trimming is needed.

Act 4: Four-Stage RL Cascade

1. Reasoning warmup (232 steps): basic RL on math and puzzles 2. RLVE-Gym (400 steps): 400 environments with adaptive difficulty — challenges that are "reachable with a jump" 3. Math + Code + TTC (848 steps): deep training on solving and on "how to think about solving" 4. Behavioral RL (384 steps): light touch-up on chat style and instruction following

Key techniques:

PipelineRL: asynchronous training, rollouts and gradient updates on separate GPU pools
DPPO Binary-TV: Binary Total-Variation trust region instead of PPO clipping, for stability
Dr-GRPO SMTSN: sequence-level loss aggregation, avoiding GRPO's bias toward long responses
MaxRL advantage estimation: mean-based normalization gives stronger gradient signal on hard problems
No KL regularization: relies entirely on trust regions, avoiding KL-in-reward length bias

Act 5: Markovian RSA — Fractal Compression of Test-Time Compute

Standard test-time compute methods recursively aggregate candidate reasoning, but each round must prefill the full history — extremely expensive for long chains.

Markovian RSA applies the Markov property: the future depends only on the present state, not the whole past. Instead of carrying the entire chain, each round keeps only each candidate's tail (a few thousand tokens) containing the critical state information:

1. Generate N candidate reasoning traces in parallel, each of length β 2. Concatenate their tails (length τ) as the next round's input 3. Repeat, carrying only the τ-length tail

At a 40K/4K configuration (40K total, 4K tail), ZAYA1-8B hits 91.9% on AIME'25 and 89.6% on HMMT'25 — rivaling models with tens to hundreds of times more active parameters.

Act 6: AMD Full-Stack Training

ZAYA1-8B was trained end-to-end — pretraining, mid-training, and SFT — on AMD MI300X GPUs with Pollara 400 networking. This proves AMD's AI stack can support complete from-scratch training, not just inference or fine-tuning: a significant milestone for anyone seeking alternatives to NVIDIA.

Benchmark Snapshot

| Model | Active params | AIME'25 | HMMT'25 | LCB-v6 | |-------|--------------|---------|---------|--------| | ZAYA1-8B (single) | 0.76B | ~70% | ~65% | ~75% | | ZAYA1-8B + RSA | 0.76B | 91.9% | 89.6% | ~85% | | DeepSeek-R1-0528 | ~37B | ~85% | ~80% | ~82% | | Gemini-2.5 Pro | ? | ~92% | ~90% | ~88% | | GPT-5-High | ? | ~94% | ~92% | ~90% |

*Note: exact comparison figures — see Table XI of the paper.*

Conclusion

ZAYA1-8B shows that parameter count is not the whole story in the AI arms race. The secret is the synergy of five choices: smarter architecture (CCA + MLP router), smarter training data (Answer-Preserving Trimming), smarter RL (four-stage cascade), smarter test-time compute (Markovian RSA), and a validated AMD training stack. As Feynman said, "What I cannot create, I do not understand." ZAYA1-8B is not just a model — it is a complete answer to the question of how to build efficient reasoning.

References

1. Washbourne, R., Iyer, R., Figliolia, T., et al. (2026). ZAYA1-8B Technical Report. *arXiv preprint arXiv:2605.05365*. 2. Anthony, Q., et al. (2025). MoE++: Zyphra's Mixture-of-Experts Architecture. *Zyphra Technical Report*. 3. Figliolia, T., et al. (2025). Compressed Convolutional Attention (CCA). *arXiv preprint*. 4. Akter, S., et al. (2025). Reasoning-aware pretraining. *ICML 2025*. 5. Khatri, et al. (2025). PipelineRL and length-control methods. *NeurIPS 2025*. 6. Venkatraman, et al. (2025). Recursive Self-Improvement (RSA). *ICML 2025*. 7. Aghajohari, et al. (2025). Markovian Thinking. *arXiv preprint*. 8. DeepSeek-AI (2025). DeepSeek-R1 and DeepSeek-V3. *Technical Reports*.