Telescopic Language Models: One Training Run, Valid at Every Depth

Forum topic · 小凯 · 2026-09-30

Summary

Telescopic Language Models (TLM) address the problem that deployed language models must serve many compute budgets, which traditionally requires separate training or compression runs per budget point. TLM is a nested-capacity Transformer trained with stochastic prefix supervision plus a full-capacity anchor: at each training step, one randomly truncated prefix of the capacity axis is trained against the full next-token objective alongside one full-capacity forward-backward pass. This produces an artifact that is a valid language model at every depth, using only two forward-backward passes per step, no architectural changes, and zero inference overhead. Compared with fixed-exit suites like Matryoshka Language Model Suites, which leave unsupervised depths near random performance, a single TLM run (200M-parameter proxy, 20B FineWeb-Edu tokens) is effective at all twenty layer depths, reducing quality-budget area by 43-44% while matching fixed-exit quality at full capacity with roughly 12% lower GPU cost. Prefix sampling density acts as a tuning knob trading continuity for targeted depth quality.

Paper Overview

Field: NLP Authors: Zhilin Guo, Boqiao Zhang, Hakan Aktas, Kyle Fogarty, Nursena Koprucu Aslan, Wenzhao Li, Canberk Baykal, Albert Miao, Siyu Hong, Yixiao Liu, Adam Wu, Ashish Kumar Singh, Sakar Khattar, Chenliang Zhou, Weihao Xia, Cristina Nader Vasconcelos, Cengiz Oztireli Published: 2026-09-28 arXiv: 2609.35769

Summary

A deployed language model often needs to serve multiple compute budgets, but supporting each budget point typically means a separate training or compression run. The authors train Telescopic Language Models (TLM) to act as a continuum: a nested-capacity Transformer supervised via stochastic prefix supervision with a full-capacity anchor.

At every step, a randomly truncated prefix of the capacity axis is trained against the full next-token objective, together with one full-capacity forward-backward pass. As a result, the trained artifact is a valid language model at every layer depth. The method requires only two forward-backward passes per step, no architectural changes, and adds zero overhead at inference.

Fixed-exit suites such as Matryoshka Language Model Suites (MLMS) occupy just one point in this design space, and that point has a cost: supervising only a few fixed exits leaves the nested model at near-random performance at all other depths (baseline perplexity of 10²–10⁵ in their experiments).

Key Results

On a 200M-parameter proxy suite (20B FineWeb-Edu tokens, identical data stream for all methods):

A single TLM run is a valid language model at every one of its twenty layer prefixes, performing well on perplexity and perplexity-sensitive downstream tasks.
Reduces the quality-budget area by 43–44% relative to fixed-exit suites.
Matches fixed-exit quality at full capacity, with roughly 12% lower GPU cost per run.
Prefix sampling density acts as a tuning knob: concentrating sampling on a few depths recovers fixed-exit quality there, at the cost of losing continuity.

Conclusion

The results indicate that it is the training objective—not nesting itself—that makes a nested model resilient across depths.

--- *Auto-collected on 2026-09-30*

Telescopic Language Models: One Training Run, Valid at Every Depth

Summary

Paper Overview

Summary

Key Results

Conclusion

Tags