Let the Backbone Predict the Future Itself

After DeepSeek-V3, multi-token prediction (MTP) has gradually become a standard part of large-model training: a small module is bolted next to the backbone to predict several future tokens. During training this gives the model a denser supervision signal; at inference, the module can double as a draft model, feeding speculative decoding. It sounds like pure profit.

But there is an awkward flaw in this design. Most of the multi-view supervision is absorbed by those bolt-on auxiliary modules — the modules learn to predict the 2nd, 3rd token, while the backbone itself barely changes. Each additional prediction depth means one more module and one more forward pass. The supervision lands on the "transfer student," not the main act.

A new paper from Baidu's ERNIE team flips this around: no more modules — let the backbone's own last layer recursively predict the future. They call it ECHO.

Reusing Down to the Last Parameter

The mechanism is simple. In a standard autoregressive model, each step runs the final transformer layer, applies the output head, and emits the next token. In ECHO, when predicting the second token, the first step's hidden state and the sampled token are concatenated and fed back into the same final layer and the same output head — the third and fourth predictions repeat this. Apart from one extremely lightweight fusion layer, all parameters for multi-step prediction are the backbone's own parameters.

This way, the multi-view supervision has nowhere to hide — it all lands on the weights "already responsible for generating the next token." As the paper's abstract puts it: future prediction is no longer a task attached to the autoregressive objective; it has become part of the autoregressive objective.

What about cost? Recursing three steps means, in theory, running the final layer three extra times. ECHO's answer is dynamic depth sampling: each optimization step rolls dice on how deep to predict, with probabilities 0.6, 0.3, and 0.1 for depths 1, 2, and 3. On average, that's only ~1.5 layers of overhead while capturing supervision up to depth 3.

Controlled Comparison: Same Recipe, Only MTP Design Differs

The backbone is a 10B-total-parameter, 1B-active mixture-of-experts model (24 layers, MLA, 256 experts with 8 active), pretrained on 600B tokens on 24×8 H800 GPUs with the Muon optimizer. Six configurations share exactly the same recipe — only the MTP design differs; everything else is locked down.

Results:

• At depth 1, ECHO already leads on math, reasoning, and code (+0.79, +0.57, +1.35).
• At depth 3, the reasoning lead grows to +2.63 and code to +2.23; Q&A and multiple-choice categories are on par.
• A full rerun reproduces the lead (+0.52 and +0.78), with the reasoning/code advantage drifting only 0.11 points — this is not single-run luck.

The supervision placement makes the SFT story more subtle. One configuration (average depth 2) trails the 1.5-average-depth version during pretraining — deep predictions accumulate errors, making the supervision noisier. But after SFT it overtakes by 0.49 points, and its lead over DeepSeek MTP-Instruct reaches the largest in the whole study at +2.10: deep supervision is noise during pretraining, but a savings account once fed into downstream fine-tuning.

Inference: Drafting with Itself

Speculative decoding needs a draft model — the traditional approach is to fine-tune a drafter after the main model is done, costing tens of wall-clock hours (in the paper's comparison table, EAGLE-3 takes about 22 hours; comparable methods range from 19 to 55 hours). Worse, post-trained drafters tend to overfit academic benchmarks and falter on real high-entropy generation.

ECHO skips this step entirely. The draft model is the pretrained weights themselves: since each draft step reuses the same layer, and training only reached depth 3, inference can keep recursing to step 5, step 8, with no acceptance-rate collapse. In experiments it achieves the longest acceptance length, and the freer the generation (higher entropy), the bigger its lead — exactly where post-trained drafters are weakest. The drafter fine-tuning stage is deleted from the pipeline altogether.

Bottom Line

The backbone is a 10B-total / 1B-active MoE pretrained on 600B tokens, with the largest dense evaluation scale among comparable methods. At this scale, folding an attached task back into the backbone's own objective yields gains in three places at once: pretraining, SFT, and inference acceleration. Those extra billions of tokens were not wasted — they were spent on the token after next.

---

*Source: ECHO: Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction (arXiv:2610.24766, 2026-10-17, ERNIE Team at Baidu + independent researchers; code open-sourced at github.com/ernie-research/ECHO). Backbone: 10B-A1B Latent MoE (24 layers, MLA, 256 experts with 8 active), 600B-token pretraining on 24×8 H800, Muon optimizer, six same-recipe controlled configurations. Figures cited come from the abstract, Figures 1-3, and Tables 1/3; the double-run reproduction and ECHO-2's post-SFT turnaround are in Sections 5-6; speculative decoding comparisons (EAGLE-3/DFlash/Domino/DSpark post-training wall-clock) are in the related-work section and Table 4.*