TL;DR: Not more data — data moving on the right track
In February 2026, researchers from Atlassian, NYU, and Brown (Hai Huang, Yann LeCun, Randall Balestriero) released Semantic Tube Prediction (STP). It adds no parameters and no extra training data — just a geometric constraint on top of standard next-token prediction — and still dares to challenge the Chinchilla scaling laws.
Result: 16x data efficiency. On the NL-RX-SYNTH dataset, STP reaches the same accuracy as full-data training with only 1/16 of the data.
---
1. The problem: why do LLMs need so much data?
Chinchilla scaling laws tell us: bigger models need more data, following a seemingly unshakeable power law.
But the LeCun-team-affiliated authors ask: is all this data actually being used effectively?
The answer: no. LLM hidden states wander chaotically in high-dimensional space, and much of the training signal is drowned in noise — like running blindfolded through a maze: running a lot doesn't mean finding the exit.
---
2. Core hypothesis: the Geodesic Hypothesis
> "Token sequences move along geodesics on a semantic manifold, and locally they are linear."
In plain language: if you view LLM hidden states as points in high-dimensional space, a correct token sequence traces the smoothest possible curve between them — a geodesic. Errors, noise, and randomness push the trajectory off this curve.
This is a powerful geometric prior: if token sequences "should" travel straight, then the training objective should include not just "predict the next token" but also "keep the hidden-state trajectory straight."
---
3. The STP mechanism: one loss term constraining the whole trajectory
The STP loss is extremely simple:
Sample three positions s < r < t from a sequence, compute the displacement vectors \((h_r - h_s)\) and \((h_t - h_r)\), and take their cosine similarity.
Goal: push the cosine toward 1, i.e., make the two displacement vectors parallel.
This means the direction from s to r matches the direction from r to t, constraining the whole trajectory to move inside a "tube" around a geodesic.
The full training objective:
Standard next-token prediction preserves language modeling ability; STP enforces geometric smoothness.
---
4. Results: 16x data efficiency is geometry, not magic
| Metric | Baseline (full data) | STP (1/16 data) | Meaning | |--------|---------------------|-----------------|---------| | NL-RX-SYNTH accuracy | Reference | Equal | Directly violates data scaling law | | Signal-to-noise ratio (SNR) | Low | Significantly improved | Cleaner hidden states | | Inference diversity | Prone to mode collapse | Fewer trajectory collisions | Prevents diversity degradation | | Two-view requirement | JEPA needs manual construction | Not needed | Simplifies JEPA-style architectures |
Key insight: STP doesn't "eat less data" — it makes each bite of data yield a cleaner signal, compressing hidden-state noise into a semantic tube so the model can "see" the right direction.
---
5. Why does it prevent inference collapse?
During training (teacher forcing), the model sees correct context and hidden states move smoothly along geodesics. During inference (autoregressive generation), the model may condition on its own erroneous tokens, and errors accumulate.
In differential-equation terms:
- Training: an ODE (deterministic, unique solution, no mode collapse)
- Inference: an SDE (stochastic, noise causes trajectories to fan out into a cone)
- Semantic Step Prediction (Yuan Yidi, 2026-04): changes STP's sampling from random token positions to reasoning-step boundaries. On ProcessBench, multi-step latent prediction accuracy is 168x higher than a frozen baseline (vs. only 4x for random-token sampling).
- Simplifying JEPA: STP requires no manually constructed two views (text + code), no extra target encoder or predictor — making JEPA-style ideas land more naturally in language models.
By constraining trajectories near a geodesic, STP adds "orbit correction" to inference — even when errors occur, hidden states don't drift too far, reducing trajectory collisions and mode collapse risk.
---
6. Follow-up impact: opening a framework
The geodesic hypothesis has inspired follow-up work:
7. One-sentence takeaway
STP's bet: LLM hidden states don't random-walk through high-dimensional space — they move along geodesics on a semantic manifold. If that's true, training shouldn't only optimize "predict the next word" but also "walk the right path."
How much is this geometric prior worth? 16x data efficiency — empirical evidence challenging scaling laws, not a theoretical conjecture.
---
Paper: arXiv:2602.22617 Authors: Hai Huang, Yann LeCun, Randall Balestriero (Atlassian, NYU, Brown) Published: 2026-02-26 Code: galilai-group/llm-jepa