STP: Challenging Scaling Laws with a Geodesic Hypothesis - Equal Performance with 1/16 of the Data

Forum topic · 小凯 · 2026-06-29

Summary

Semantic Tube Prediction (STP), introduced in February 2026 by researchers from Atlassian, NYU, and Brown (Hai Huang, Yann LeCun, Randall Balestriero), adds a lightweight geometric constraint to standard next-token prediction without increasing parameters or data. Built on the geodesic hypothesis—that token sequences move along locally linear geodesics on a semantic manifold—STP penalizes angular deviation between successive hidden-state displacement vectors, keeping trajectories inside a 'tube' around the geodesic. The reported result is 16x data efficiency: on NL-RX-SYNTH, STP matches full-data accuracy using only 1/16 of the training data, alongside improved signal-to-noise ratio in hidden states and reduced mode collapse during autoregressive inference. STP also simplifies JEPA-style training by requiring no two-view construction or target encoder. A follow-up, Semantic Step Prediction, samples at reasoning-step boundaries and reports 168x gains on ProcessBench over frozen baselines. Paper: arXiv:2602.22617.

One-line takeaway: Instead of feeding more data, make the data run on the right track

In February 2026, researchers from Atlassian, NYU, and Brown (Hai Huang, Yann LeCun, Randall Balestriero) released Semantic Tube Prediction (STP). It adds no parameters and no training data—only a geometric constraint on top of standard next-token prediction—yet dares to challenge Chinchilla-style scaling laws.

Result: 16x data efficiency. On the NL-RX-SYNTH dataset, STP reaches the same accuracy as full-data training using just 1/16 of the data.

---

1. The problem: Why do LLMs need so much data?

Chinchilla scaling laws tell us: bigger models need more data. This power-law relationship seems immovable.

But the LeCun-team authors asked: Is all this data actually being used effectively?

The answer: no. An LLM's hidden states wander chaotically in high-dimensional space, and much of the training signal is drowned in noise. It's like asking someone to run blindfolded through a maze—running a lot doesn't mean finding the exit.

---

2. Core assumption: the Geodesic Hypothesis

> "Token sequences move along geodesics on a semantic manifold, and locally this motion is linear."

In plain terms: if you view an LLM's hidden states as points in high-dimensional space, a correct token sequence traces a smoothest possible curve—a geodesic—between them. Errors, noise, and randomness push the curve off this path.

This is a powerful geometric prior: if token sequences "should" travel straight, then the training objective should include not only "predict the next token correctly" but also "keep the hidden-state trajectory straight."

---

3. The STP mechanism: one loss term constrains the entire trajectory

STP's loss is extremely simple:

L_{STP} = E[1 - \cos(h_r - h_s, h_t - h_r)]

Randomly sample three positions s < r < t from a sequence, compute the hidden-state displacement vectors $$(h_r - h_s)$$ and $$(h_t - h_r)$$ , then take the cosine of the angle between them.

Goal: push the cosine toward 1, i.e., make the two displacement vectors parallel.

This means the direction from s to r matches the direction from r to t. The whole trajectory is constrained to a "tube" around a geodesic.

Full training objective:

L = L_{NTP} + \lambda \cdot L_{STP}

Standard next-token prediction preserves language modeling ability; STP enforces geometric smoothness.

---

4. Results: the 16x data efficiency is not magic, it's a geometric constraint

| Metric | Baseline (full data) | STP (1/16 data) | Meaning | |--------|----------------------|-----------------|---------| | NL-RX-SYNTH accuracy | reference | equal | Directly violates data scaling law | | Signal-to-noise ratio (SNR) | low | significantly higher | Cleaner hidden states | | Inference diversity | prone to mode collapse | fewer trajectory collisions | Prevents diversity loss in generation | | Two-view requirement | JEPA needs manual construction | not needed | Simplifies JEPA-style architectures |

Key insight: STP does not "eat less data"—it makes every bite of data produce a cleaner signal. It compresses hidden-state noise into the semantic tube so the model can "see" the right direction.

---

5. Why does it prevent inference collapse?

During training (teacher forcing), the model sees correct context, and hidden states move smoothly along geodesics. But at inference (autoregressive generation), the model may see its own erroneous tokens, and errors accumulate.

In differential-equation terms:

Training: ODE (deterministic, unique solution, no mode collapse)
Inference: SDE (stochastic, noise makes trajectories diverge into a cone)

By constraining trajectories near geodesics, STP adds a kind of "orbit correction" to inference—even when errors appear, hidden states don't drift too far, reducing trajectory collisions and mode collapse.

---

6. Follow-up impact: an opened framework

STP's geodesic assumption has inspired follow-up work:

Semantic Step Prediction (Yuan Yidi, 2026-04): changes STP's sampling points from random token positions to reasoning-step boundaries. On ProcessBench, multi-step latent prediction accuracy beat a frozen baseline by 168x (the random-token version achieved only 4x).
Simplifying JEPA: STP needs no manually constructed two views (text + code), no extra target encoder or predictor—letting JEPA-style ideas land more naturally in language models.

---

7. One-sentence summary

STP's bet: LLM hidden states are not randomly wandering through high-dimensional space—they move along geodesics on a semantic manifold. If true, the training objective should include not only "predict the next word correctly" but also "walk the right path."

How much is this geometric prior worth? 16x data efficiency—empirical evidence challenging scaling laws, not a theoretical conjecture.

---

Paper: arXiv:2602.22617 Authors: Hai Huang, Yann LeCun, Randall Balestriero (Atlassian, NYU, Brown) Published: 2026-02-26 Code: galilai-group/llm-jepa