Based on a video explainer by bycloud combined with deep reading of recent papers (LeJEPA, EchoJEPA, V-JEPA 2).
Video: What Is Yann LeCun Cooking? JEPA Explained Simply
A Heretic in the AI World
In September 2025, at a NYU seminar, Meta's Chief AI Scientist Yann LeCun — Turing Award winner and co-inventor of deep learning — made a statement that shook the AI community:
"Autoregressive large language models are doomed to fail."
His proposed alternative is JEPA (Joint-Embedding Predictive Architecture), a framework he has been pushing for years. The problem: JEPA involves latent spaces, representation collapse, isotropic Gaussians — a pile of abstract jargon that leaves most listeners confused. This article explains JEPA's core logic, recent progress, and real-world results in plain language.
Forget Everything You Know About AI Training
To understand JEPA, first understand what it opposes. There are two mainstream training approaches:
1. Autoregressive prediction (the LLM approach): predict the next token given the prefix. It generates at pixel/character-level precision. The problem: imagine watching a movie where someone pauses every second and asks "what does the next frame look like?" — lighting changes, camera shake, and random details are impossible to predict precisely. But asked "what happens next in the story?" you'd answer well.
2. Pixel-level reconstruction (MAE, VAE): fill in missing pixels of an image. The problem: it over-focuses on detail and ignores semantics. Fur direction and lighting angle tell you nothing about "understanding a cat," yet reconstruction forces the model to learn them.
JEPA is completely different: it predicts neither pixels nor tokens, but abstract descriptions.
Views: JEPA's Basic Unit
JEPA's core concept is the View — a different "take" on the same thing: a crop, a blur, a brightness change, a partial occlusion of a cat photo. All are semantically identical: a cat.
JEPA's training logic: given one view (context), predict another view's representation in latent space.
The keyword is latent space. JEPA doesn't ask you to reproduce the masked region pixel-by-pixel; it asks what the region's abstract features should be. Show someone the left half of a painting: reconstruction requires reproducing every stroke; JEPA only requires "there should be a bird flying on the right." It's about semantics, not detail — filtering irrelevant noise and focusing on meaning.
JEPA's Three Components
1. Context Encoder: encodes the visible part (e.g., first half of a video) into a latent vector. 2. Target Encoder: encodes the part to be predicted into a latent vector (the "correct answer"). 3. Predictor: given the context encoding, predicts the target encoding; training minimizes the gap.
Key detail: the target encoder's gradients are stopped (stop-gradient). The predictor learns "how to predict," while the target encoder learns "how to encode" unaffected — like a student guessing a teacher's answers while the teacher's grading standard stays fixed.
Three Application Scenarios
1. Representation Extraction (I-JEPA)
The basic use: train a visual encoder, then plug it into downstream tasks (classification, detection) without retraining from scratch. I-JEPA achieved SOTA linear-probe accuracy on ImageNet among self-supervised methods at the time.2. World Modeling (V-JEPA / V-JEPA 2)
LeCun's most valued direction. V-JEPA learns to predict masked spatiotemporal regions of video in latent space — essentially learning a world model of physics. V-JEPA 2 (2025) went further: self-supervised pretraining on internet-scale video plus a small amount of robot interaction data achieved zero-shot robot control — a model that has only watched videos can control a robot on new tasks. Strong evidence that understanding the world ≠ generating text.3. Latent-Space Robot Planning
Traditional planning in pixel or action space is extremely costly. JEPA enables planning in latent space — simulating consequences of actions in an "abstract world," picking the best, then executing. Like a chess master who reasons in an abstract strategy space rather than simulating every piece's trajectory.The Biggest Enemy: Representation Collapse
JEPA has a fatal trap: representation collapse. Like a student answering "I don't know" to every exam question — never too wrong, but learning nothing. In JEPA, encoders map all inputs to nearly identical vectors, making the task meaningless. Two forms:
- Total collapse: all inputs map to one point
- Dimensional collapse: all inputs map to a low-dimensional subspace
- Patch 1 — EMA (exponential moving average): target encoder updated slowly via EMA (SimSiam, BYOL), so even if the predictor "cheats," the target doesn't immediately cooperate.
- Patch 2 — Sample contrast (SimCLR): push different samples' representations apart, same-sample views together; requires huge batches (4096+), costly.
- Patch 3 — Dimension contrast (Barlow Twins, VICReg): constrain per-dimension statistics — high variance, low correlation. VICReg is currently the most popular.
- Linear complexity in compute and memory
- Theoretical guarantees from statistical testing theory
- No heuristics: no stop-gradient, no teacher-student, no hyperparameter schedulers
- ImageNet-1k linear probing: ViT-H/14 reaches 79%, matching heuristic-heavy methods
- Training stability: stable even on a 1.8B-parameter ViT-g with no special tricks
- Architecture-agnostic: works across 60+ architectures (ResNet, ViT, ConvNeXt, MaxViT, Swin Transformer)
- Domain transfer: on Galaxy10 (astronomical images), LeJEPA's in-domain pretraining fully outperforms transfer from DINOv2/v3 — frontier models trained on billions of natural images
- Zero-shot generalization to pediatric patients: pretrained on adult data, EchoJEPA outperforms baselines fully fine-tuned on pediatric data
- Robustness to acoustic degradation: under simulated artifacts (depth attenuation, shadowing, speckle), performance drops only 17% vs 40%+ for competitors
- Data efficiency: competitive performance with just 1% of labels
- Vision/video: information is continuous and redundant. A million pixels compress into a few dimensions of "cat." JEPA's latent prediction naturally suits this "high-redundancy → low-dimensional semantics" compression.
- Language: information is discrete and dense. Every word carries precise semantics; there's no intermediate state between "cat" and "dog." Next-token prediction *is* semantic prediction — no pixel noise to filter.
- Code: github.com/facebookresearch/eb_jepa
- Every example trains on a single GPU within hours
- Includes full implementations of I-JEPA, V-JEPA, and AC-JEPA (action-conditioned)
- Paper: arxiv.org/abs/2506.09985
- Code: github.com/facebookresearch/jepa
- Paper: arxiv.org/abs/2511.08544
Known "patches":
All share one problem: they are heuristic — known to work, but not known why or whether optimal.
LeJEPA: Ending the Patch Era with Mathematics
In late 2025, LeJEPA (Latent-Euclidean JEPA) arrived from Meta FAIR's Randall Balestriero and Yann LeCun. Its contribution isn't another patch — it mathematically proves the optimal solution.
Core theorem: latent representations should be isotropic Gaussian
LeJEPA proves that for best downstream performance, encoder outputs should follow an isotropic Gaussian (zero mean, identity covariance). Intuition: like organizing a library so books are evenly spread across all dimensions — no corner-piling (dimensional collapse), no stacking (total collapse). Isotropic Gaussian is the ideal "uniform and dispersed" state.
SIGReg: efficiently reaching the ideal distribution
SIGReg (Sketched Isotropic Gaussian Regularization): 1. Randomly choose direction vectors 2. Project latent representations onto them 3. Check if projected distributions approach standard normal 4. Adjust the encoder if not
Like a quality inspector sampling from many angles rather than checking every detail.
Advantages:
Results: simplicity is power
Most striking: LeJEPA's core code is ~50 lines. No stop-gradient, no EMA, no teacher-student, one hyperparameter. When you truly understand a problem, the solution is often surprisingly simple.
EchoJEPA: JEPA's Striking Landing in Medical Imaging
If LeJEPA proves JEPA's theory, EchoJEPA (ICML 2026) proves its practical value: JEPA applied to echocardiography — ~30 million exams worldwide annually.
Ultrasound's unique challenge is speckle noise: random noise unrelated to cardiac anatomy, which traditional methods force models to learn. EchoJEPA's core advantage fits perfectly: JEPA predicts in latent space, naturally filtering pixel-level noise.
Results are stunning:
When you stop forcing a model to predict irrelevant details, it learns something more essential.
Why JEPA Isn't for LLMs
An important caveat from the video: JEPA isn't a replacement for LLMs, due to modality differences:
So LeCun's "LLMs are doomed" is better understood as: autoregressive LLMs are not the only or optimal path to AGI. For understanding the physical world, planning, and reasoning, JEPA-style world models may be the better choice.
So What Is LeCun Cooking?
He's doing something counter-trend: while everyone chases bigger LLMs, he pursues a completely different AI paradigm built on:
1. Understanding the world requires world models, not more text data 2. Prediction should happen at an abstract level, not at pixel or token level 3. Good theory beats good engineering — LeJEPA proves it with 50 lines of code
JEPA may not replace ChatGPT, but it could become the foundational architecture for robotics, autonomous driving, medical imaging, and scientific discovery.
Open Source: Try It Yourself
EB-JEPA (Feb 2026): Meta FAIR's official open-source library with JEPA implementations for images, video, and action-conditioned video, plus JEPA-based planning algorithms.
V-JEPA 2 (June 2025): video world model with zero-shot robot control.
LeJEPA: theoretically optimal JEPA, ~50 lines of core code.
Final Thoughts
What moves me most about JEPA is its answer to the fundamental question: what makes a good representation? LeJEPA shows the best representations are isotropic Gaussians — an ideal foundation model spreads all inputs evenly across latent space, with no wasted dimensions, no collapsed subspaces, every dimension carrying unique information.
This raises a deeper question: might the human brain be doing something similar? When we see a cat, we don't memorize pixels — we extract the concept of "cat." How is that concept represented in our neural networks? Is it also close to some "uniformly dispersed" state?
Perhaps JEPA isn't just an engineering method — it's a window into the nature of intelligence.
---
Papers | LeJEPA (arxiv.org/abs/2511.08544) | EchoJEPA (ICML 2026) | V-JEPA 2 (arxiv.org/abs/2506.09985) | EB-JEPA (arxiv.org/abs/2602.03604) Code | facebookresearch/eb_jepa Video | What Is Yann LeCun Cooking? by bycloud