TaH2: Adaptive Looped Transformers for Better Test-Time Scaling

Forum topic · 小凯 · 2026-09-30

Summary

A paper on arXiv (2609.35748) by Yichen You, Tianyu Fu, Aosong Feng, Xingtai Lv, Xuefei Ning, Ning Ding, and Yu Wang investigates whether looped transformers improve test-time scaling as outputs grow longer. The authors post-train looped transformers and measure the accuracy-compute slope, defined as accuracy gain per doubling of test-time decoding FLOPs. They find that existing looped transformers often produce steeper slopes than non-looped baselines but underperform them at matched compute, because fixed-depth looping spends extra iterations on every token even though many tokens do not benefit. They propose TaH2, which jointly post-trains the backbone and an iteration decider using lookahead depth supervision with online labels indicating whether further iterations improve predictions, concentrating extra iterations on tokens that benefit from looping. On the AIME benchmark, TaH2 improves the accuracy-compute slope by 53% over the non-looped baseline (2.74 vs. 1.79) and surpasses the baseline's peak accuracy by about 3.4 points at matched test-time compute. While existing looped models plateau as maximum iteration depth increases, TaH2's gains grow from +2.8 points at depth 2 to +3.9 points at depth 8.

Paper Overview

Field: NLP
Authors: Yichen You, Tianyu Fu, Aosong Feng, Xingtai Lv, Xuefei Ning, Ning Ding, Yu Wang
Published: 2026-09-28
arXiv: 2609.35748

Summary

Looped transformers demonstrate promising parameter efficiency by reusing layers for latent computation. Prior studies compared looped and non-looped models at matched parameters or per-token FLOPs, but whether looping improves test-time scaling as outputs grow longer remained underexplored.

Through post-training looped transformers, the authors study the accuracy-compute slope, measured as the accuracy gain per doubling of test-time decoding FLOPs. They find that existing looped transformers often yield steeper slopes than their non-looped baseline, yet underperform it at matched compute. While fixed-depth looping spends extra iterations on every token, their analysis shows that many tokens do not benefit from extra iterations.

Proposed Method: TaH2

TaH2 enables the model to concentrate extra iterations on tokens that benefit from looping. It jointly post-trains the backbone and an iteration decider using lookahead depth supervision, with online labels indicating whether further iterations improve predictions.

Key Results

On the AIME benchmark, TaH2 improves the accuracy-compute slope by 53% over the non-looped baseline (2.74 vs. 1.79).
At matched test-time compute, TaH2 exceeds the baseline's peak accuracy by about 3.4 points.
As maximum iteration depth increases, existing looped models plateau, while TaH2's gain over the baseline grows from +2.8 points at depth 2 to +3.9 points at depth 8.

---

*Auto-collected on 2026-09-30.*

TaH2: Adaptive Looped Transformers for Better Test-Time Scaling

Summary

Paper Overview

Summary

Proposed Method: TaH2

Key Results

Tags