Yann LeCun's Secret Kitchen: What Is JEPA Really Cooking?
> Based on bycloud's video walkthrough, combined with deep reading of LeJEPA, EchoJEPA, V-JEPA 2, and related papers > Original video: What Is Yann LeCun Cooking? JEPA Explained Simply
A Heretic in the AI World
In September 2025, at a New York University seminar, Meta's Chief AI Scientist Yann LeCun dropped a bombshell:
"Autoregressive LLMs are doomed to fail."
This wasn't coming from a nobody — he is one of the deep learning triumvirate, a 2018 Turing Award winner, and the inventor of convolutional neural networks. His proposed alternative is the JEPA (Joint-Embedding Predictive Architecture) he has been championing for years.
The problem: every time LeCun explains JEPA, listeners scratch their heads. Latent space, representation collapse, isotropic Gaussians — a pile of abstract jargon.
This article explains JEPA's core logic, latest progress, and real-world results in plain language.
First, Forget Everything You Know About AI Training
To understand JEPA, you must understand what it opposes. There are two mainstream training approaches:
1. Autoregressive prediction (the LLM approach)
Given the first half of a text, predict the next token. That's how GPT is trained: token-by-token generation, precise down to the character/pixel level.
The problem? Imagine watching a movie while someone pauses every second to ask "what does the next frame look like?" You can't guess precisely — lighting changes, camera shake, and minute details are simply unpredictable. But if asked "what happens next in the story?", you'd answer well.
2. Pixel-level reconstruction (MAE, VAE approach)
Given part of an image, fill in the missing pixels — precisely, down to each pixel's color.
The problem? Too much attention to detail, ignoring semantics. A cat photo's fur direction and lighting angle don't help with "understanding cats," but pixel-level reconstruction forces the model to learn these irrelevant details.
JEPA is completely different: it predicts neither pixels nor tokens, but abstract descriptions.
"Views": JEPA's Basic Unit
JEPA's core concept is the View — a different "look" at the same thing. For a cat photo:
- A cropped version is a view
- A blurred version is a view
- A brightness-adjusted version is a view
- A partially occluded version is a view
- Total collapse: all inputs map to one point
- Dimensional collapse: all inputs map to a low-dimensional subspace
- Linear complexity in dimension for compute and memory
- Theoretical guarantees backed by rigorous statistical testing theory
- No heuristics: no stop-gradient, no teacher-student, no hyperparameter schedulers
- ImageNet-1k linear probe: ViT-H/14 reaches 79%, on par with methods requiring heavy heuristic tricks
- Training stability: stable even on a 1.8B-parameter ViT-g with no special tricks
- Cross-architecture generality: works on ResNet, ViT, ConvNeXt, MaxViT, Swin Transformer, and 60+ architectures
- Domain transfer: on Galaxy10 (astronomical images), LeJEPA's in-domain pretraining fully surpasses DINOv2/v3 transfer learning — even though those are frontier models trained on billions of natural images
- Zero-shot generalization to pediatric patients: pretrained on adult data, EchoJEPA applied directly to pediatric patients outperforms baselines fully fine-tuned on pediatric data
- Robustness to acoustic degradation: under simulated artifacts (depth attenuation, acoustic shadowing, speckle), performance drops only 17%, versus 40%+ for competitors
- Data efficiency: competitive performance with only 1% of labeled data
- Vision/video: information is continuous and redundant. A cat photo has millions of pixels, but "cat" needs only a few dimensions. JEPA's latent prediction naturally suits this "high redundancy → low-dimensional semantics" compression.
- Language: information is discrete and dense. Every word carries precise semantics; there's no "intermediate state" between "cat" and "dog." In language, predicting the next token *is* semantic prediction — there's no "pixel noise" to filter.
- Code: github.com/facebookresearch/eb_jepa
- Every example trains on a single GPU within hours
- Includes complete I-JEPA, V-JEPA, and AC-JEPA (action-conditioned) implementations
- Paper: arxiv.org/abs/2506.09985
- Code: github.com/facebookresearch/jepa
- Paper: arxiv.org/abs/2511.08544
These views look different but share the same semantics: they are all cats.
JEPA's training logic: given one view (context), predict another view's representation in latent space.
Note the key phrase: latent space. JEPA doesn't ask you to reconstruct what the masked region looks like (that's pixel-level reconstruction) — it asks you to predict the region's abstract features.
Analogy: shown the left half of a painting, pixel-level reconstruction demands you repaint every stroke; JEPA only requires you to say "the right half should show a bird flying." It cares about semantics, not details.
This is JEPA's core advantage: filter irrelevant noise, focus on semantic information.
JEPA's Three Components
1. Context Encoder
Encodes "what you see" (e.g., the first half of a video) into a latent vector — an abstract description of the observed content.
2. Target Encoder
Encodes "what you must predict" (e.g., the second half) into a latent vector — the abstract description of the correct answer.
3. Predictor
Given the context encoder's output, predicts the target encoder's output. Training minimizes the gap between prediction and the target's actual representation.
Key detail: the target encoder's gradients are truncated (stop-gradient). The predictor learns "how to predict," while the target encoder learns "how to encode" independently. Like an exam: the student (predictor) guesses the teacher's answers, but the teacher doesn't change the grading standard based on the student's guesses.
Three Application Scenarios
1. Representation Extraction (I-JEPA)
The most basic use: train a visual encoder to extract meaningful features from images, then plug it into downstream tasks (classification, detection) without training from scratch. I-JEPA achieved state-of-the-art linear-probe accuracy on ImageNet among self-supervised methods at the time.
2. World Modeling (V-JEPA / V-JEPA 2)
The direction LeCun cares about most. V-JEPA trains on video, learning to predict latent representations of masked spatiotemporal regions — essentially learning a world model of physics.
2025's V-JEPA 2 went further: self-supervised pretraining on internet-scale video plus a small amount of robot interaction data enabled zero-shot robot control — a model that has only watched videos, never operated a robot, can directly control one on new tasks. The strongest evidence for the JEPA route: understanding the world ≠ generating text.
3. Latent-Space Planning for Robots
Traditional planning operates in pixel or action space — computationally expensive. JEPA enables planning in latent space: simulate consequences of different actions in an "abstract world," pick the best, then execute. Like chess masters who don't simulate every piece's trajectory but reason in an abstract strategy space.
The Biggest Enemy: Representation Collapse
JEPA has a fatal pitfall: representation collapse.
Analogy: on an exam asking you to complete sentences, if you always answer "I don't know," you're never very wrong — but you learn nothing. In JEPA, collapse means the encoder maps all inputs to nearly identical vectors. If every image has the same representation, the predictor just outputs a constant and the task becomes meaningless.
Two types:
Patch 1: EMA (Exponential Moving Average)
Two encoders: one updated online, one slowly via EMA. The target encoder uses the EMA version, so even if the predictor finds a "cheat," the target doesn't immediately cooperate. Core trick behind SimSiam and BYOL.
Patch 2: Sample Contrast (SimCLR)
Push representations of different samples apart, pull views of the same sample together. Requires large batch sizes (typically 4096+) — costly.
Patch 3: Dimension Contrast (Barlow Twins, VICReg)
Don't compare samples; instead constrain each latent dimension's statistics — sufficient variance, low inter-dimension correlation. VICReg is currently among the most popular.
All share a common problem: they're heuristic — we know they work, but not why, or whether they're optimal.
LeJEPA: Ending the Patch Era with Mathematics
In late 2025, LeJEPA (Latent-Euclidean JEPA) arrived from Meta FAIR's Randall Balestriero and Yann LeCun himself.
Its core contribution isn't a new patch, but a mathematical proof of what the optimal solution is.
Core Theorem: Latent Representations Should Be Isotropic Gaussian
LeJEPA proves that for the best downstream performance, the encoder's latent representations should follow an isotropic Gaussian distribution — zero mean, identity covariance.
The intuition is clear: like organizing a library, the best arrangement spreads information evenly across every dimension — books shouldn't all crowd one corner (dimensional collapse) or stack in one pile (total collapse). The isotropic Gaussian is this ideal of "uniform and dispersed."
SIGReg: Efficiently Reaching the Ideal Distribution
LeJEPA proposes SIGReg (Sketched Isotropic Gaussian Regularization):
1. Randomly choose a set of direction vectors 2. Project latent representations onto these directions 3. Check whether the projected distributions are close to standard normal 4. If not, adjust the encoder accordingly
Like a quality inspector checking products from multiple angles — spot checks from enough angles guarantee quality.
SIGReg's advantages:
Results: Simplicity Is Power
Most striking: LeJEPA's core code is ~50 lines. No stop-gradient, no EMA, no teacher-student, one hyperparameter.
A plain truth confirmed: when you truly understand a problem, the solution is often surprisingly simple.
EchoJEPA: JEPA's Striking Landing in Medical Imaging
If LeJEPA proves JEPA's theoretical foundation, EchoJEPA proves its practical value.
EchoJEPA (ICML 2026) applies JEPA to echocardiography — roughly 30 million exams per year worldwide.
Ultrasound poses a unique challenge: speckle noise. Random noise points unrelated to cardiac anatomy, which traditional methods force the model to learn. EchoJEPA's core advantage fits precisely: JEPA predicts in latent space, naturally filtering pixel-level noise.
The results are stunning:
Lesson: when you don't force a model to predict irrelevant details, it learns something more essential.
Why Doesn't JEPA Suit LLMs?
The video makes an important point: JEPA isn't a replacement for LLMs. The core reason is modality difference:
So "LLMs are doomed," more precisely, means: autoregressive LLMs are not the only — nor the optimal — path to AGI. For tasks requiring physical-world understanding, planning, and reasoning, JEPA-style world models may be the better choice.
What Is LeCun Actually Cooking?
He is doing something counter-trend: while everyone chases bigger LLMs, he pursues a completely different AI paradigm, built on three beliefs:
1. Understanding the world requires world models, not more text data 2. Prediction should happen at an abstract level, not at the pixel or token level 3. Good theory beats good engineering — LeJEPA proved it with 50 lines of code
JEPA may not replace ChatGPT, but it could become the foundational architecture for robotics, autonomous driving, medical imaging, and scientific discovery. When AI needs to understand the physical world and make planning decisions, JEPA-style world models may be the better tool than autoregressive generation.
Open Source: Try It Yourself
EB-JEPA (February 2026): Meta FAIR's official open-source library with JEPA implementations for images, video, and action-conditioned video, plus JEPA-based planning algorithms.
V-JEPA 2 (June 2025): video world model with zero-shot robot control.
LeJEPA: theoretically optimal JEPA with ~50 lines of core code.
Final Thoughts
What moves me most about JEPA is its answer to the fundamental question: what makes a good representation?
LeJEPA proves the best representation is an isotropic Gaussian. An ideal foundation model should spread all inputs uniformly across latent space — no wasted dimensions, no collapsed subspaces, every dimension carrying unique information.
This raises a deeper question: does the human brain do something similar? When we see a cat, we don't memorize pixels — we extract the concept "cat." How is that concept represented in our neural networks? Is it also close to some "uniformly dispersed" state?
Perhaps JEPA isn't just an engineering method — it's a window into the nature of intelligence.
---
Papers | LeJEPA (arxiv.org/abs/2511.08544) | EchoJEPA (ICML 2026) | V-JEPA 2 (arxiv.org/abs/2506.09985) | EB-JEPA (arxiv.org/abs/2602.03604) Code | facebookresearch/eb_jepa Video | What Is Yann LeCun Cooking? by bycloud