AMI's H-JEPA: Three-Layer Hierarchical World Models Boost Visual AntMaze Success from 18% to 73.3%

Forum topic · 小凯 · 2026-10-21

Summary

H-JEPA, a new paper from Yann LeCun's Advanced Machine Intelligence team, stacks three JEPA world-model layers, each predicting in its own latent space at its own time scale: a slow top layer handling coarse navigation goals, a middle layer for waypoints, and a fast bottom layer translating subgoals into joint-level actions. Higher layers discard fast, unpredictable details (like leg poses) that lower layers handle, with this abstraction learned automatically rather than hand-designed. On Visual AntMaze, a flat single-layer world model scores 18% while three-layer H-JEPA reaches 73.3% using less planning compute. Ablations show most of the gain comes from abstract high-level goal representations rather than task decomposition alone. The paper also documents 'slow feature collapse' on DROID robot video: predictors optimally discard hard-to-predict content like robot arms, fixed by adding an inverse-dynamics loss requiring action labels. Limits are acknowledged: manipulation tasks (Push-T, Cube) benefit far less, and four layers degrade performance due to data hunger. Code, weights, and datasets are open-sourced under MIT license.

AMI Drops the Ant's Legs: H-JEPA, a Three-Layer Hierarchical World Model

A virtual ant must find its way through a maze. Its legs have six joints each, swinging several times per second; the ant's position in the maze moves one cell every few seconds. For a world model to predict all of this, trouble arises: predicting the legs requires seeing fine details at every step; predicting position requires remembering where the ant was seconds ago and which way it was heading. Fast variables and slow variables get mixed into one pot.

Yann LeCun's company AMI (Advanced Machine Intelligence) released a paper last week offering a layered solution to this pot: H-JEPA. Three world-model layers are stacked, with each layer making predictions in its own latent space, on its own time scale. Planning unfolds top-down: the top-level planner thinks about the coarse "where to go," the middle layer about "which intersection to reach first," the bottom layer about "how the joints should move in this half second." Higher-level predictions directly become lower-level subgoals.

The results speak in numbers. On the Visual AntMaze task, a flat single-layer world model achieves an 18% success rate; three-layer H-JEPA reaches 73.3% — with less planning compute. It wins not by brute force, but by ignoring what it shouldn't attend to.

Upstairs and Downstairs, Each Keeps Its Own Clock

First, what is JEPA? It's the roadmap LeCun has pushed for nearly a decade: instead of predicting pixels, the model predicts in latent space — rather than learning "what every pixel of the next frame looks like," it learns "an encoding of the important quantities in the next frame." This saves compute and avoids being hijacked by irrelevant details like cloud shadows and reflections. H-JEPA builds a tower on this idea: each layer is a JEPA with its own actions, its own field of view, and its own clock.

The elegance lies in the division of labor. The top layer's clock ticks slowly, with one planning step covering ten seconds; the bottom layer's clock ticks fast, covering only half a second per step. The high layer cannot see the legs — in its latent space, leg poses have been discarded, leaving position and maze outline. The low layer cannot see the global picture — it only translates subgoals handed down from above into actions. The paper's decoding figure is intuitive: the same frame passes through three layers of encoders; the first layer still shows the legs, the third only a blob of ant-with-position.

This "discarding" is learned, not hand-specified. During training, each layer decides what to keep on its own: what it can predict accurately stays; what it cannot is thrown away. Leg poses are noise for a slow clock — discard. Position is the main melody for a slow clock — keep. Whatever factors evolve on separated time scales in the data, the model automatically sorts them onto different floors.

Two Mechanisms, Separately Verified

Hierarchy brings two benefits: tasks are temporally decomposed (big goals split into small ones), and higher levels gain more abstract goal representations (higher floors mean goals that look less like pixels and more like intent). Can these two benefits be measured separately?

The paper runs a clean experiment: keep the three-layer architecture intact, but swap out the "high-level goal" — instead of using the high layer's own latent-space goals, use low-level representations projected upward as targets, everything else identical. Performance drops to 22%. Conversely, with true hierarchical goals, it's 73.3%. In other words, the bulk of that seventy-percent success rate comes from "goals living on the right floor," not from "tasks being chopped up." Task decomposition alone (using only the first layer's dynamics) yields just 16.7% — worse than the flat baseline. Decomposition's benefit is real, but the star of the show is the goal representation.

More layers aren't always better. On the same task, three layers hit 73.3%; four layers fall back to 63.3%. Each added layer demands more training data, and by four layers the data runs out.

Collapse: The Answer the Objective Function Chooses

The paper's best section is in the appendix. While training on the DROID real-robot video dataset, the team hit a trap: the model's prediction loss kept falling, seemingly learning well — until they noticed the encoder was discarding the wrong things. It retained the background (factory floor, tabletop, lighting — nearly constant within the prediction horizon, thus trivially predictable) and discarded the robot arm (constantly moving, hard to predict). To minimize prediction loss, the easiest route is to simply discard whatever you can't predict. The model did exactly that, and did it quite well.

This phenomenon has a name: slow feature collapse. The paper devotes a seven-subsection appendix to proving something harsh: as long as a regularizer only sees the marginal distribution of per-frame embeddings, this collapse can never be prevented — and it isn't a flaw of failed training; it is the optimal solution of the prediction objective. Discarding the unpredictable is precisely the optimal strategy for making predictions accurate. You can't blame the model; the problem was framed this way.

The antidote is adding an inverse-dynamics loss to the objective: show the model "what action was executed at this step" and teach it to infer the action from adjacent frames. With action labels, the robot arm becomes a "controlled subject" — a link in the causal chain that cannot be thrown away, because the action would have nowhere to live. The background, lacking action labels, still lines up by predictability and gets discarded as usual. The loss curves show the effect: once pure prediction loss plunges toward zero, that's a collapse signal (a constant is the best prediction); with inverse dynamics added, prediction losses at every layer stabilize on non-zero plateaus.

What the Paper Doesn't Claim, It Says So Itself

As usual, the boundaries are laid on the table. Hierarchy is not a master key: across four simulated environments, navigation-style tasks (AntMaze, FourRoom, Humanoid) show significant high-level abstraction gains (distractor decoding difficulty differs by 3–12x), while manipulation tasks (Push-T, Cube) show only 1.2–1.6x — tasks like pushing blocks lack the "fast legs, slow position" structure, so there's nothing for abstraction to discard. On Push-T the three-layer model actually fails: its episodes are too short, and the three-layer model sees only 58% of the transition data the baseline gets — not enough to feed it. The DROID results are improved offline planning fidelity, not a real robot being driven around. Open-sourcing is real: code under MIT, plus model weights and datasets, in a repository under the author's personal GitHub with 180+ stars — still young.

Finally, where this battle stands. The world-model race currently has three roads: generative (drawing pixel by pixel — expensive, but what you see is what you get), latent (the JEPA route — compute-efficient but hard to verify correctness), and hybrid. H-JEPA's answer is to stack the latent approach into a tower. Pixel-faction observers will ask the old question: is the information your latent space discarded really all unwanted? Appendix L of the paper honestly answers half of it: under a pure prediction objective, what the model discards is exactly what it cannot predict — including the parts you most want it to remember. Inverse dynamics is a patch that ransoms "the controlled" from "the unpredictable." But that patch requires action labels, which pure video data doesn't have — hence DROID used the action-annotated subset.

Three floors, 73.3%, less compute — this is the first time a latent world model has stacked to three floors on long-horizon tasks without collapsing. The ant is still in the maze, but upstairs and downstairs have already started dividing the work.

---

*Sources: arXiv 2610.06805 (2026-10-05), "H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning"; Wancong Zhang, Basile Terver, Michael Rabbat, Yann LeCun, Randall Balestriero (NYU × Advanced Machine Intelligence × INRIA); code and weights open-sourced (GitHub kevinghst/H-JEPA, MIT; HuggingFace jepa-world-models); experimental environments: Visual AntMaze/FourRoom/Humanoid/Push-T/OGBench Cube/DROID.*