Learn · Intermediate

JEPA: teaching a model to predict its own understanding

A joint-embedding predictive architecture, or JEPA, trains a neural network to predict its own internal representation of a part of the input it has not been shown, instead of predicting the raw pixels or tokens directly. The point of this apparently small change is data efficiency: most of the detail in a photograph or a sentence is unpredictable noise, and a model that is not forced to reproduce it can spend all its capacity on the structure that actually matters. The approach was pushed hardest by Yann LeCun's group at Meta and is now backed by a formal proof that it can be exponentially more sample-efficient than learning from tokens.

The problem it solves

Suppose you want a model to understand images without labelling millions of them. The obvious self-supervised trick is masking: hide part of the picture, ask the model to fill it in, and grade it on how close the reconstruction is. This works, and it is how masked autoencoders learn.

But look at what you are actually asking. To reconstruct a masked patch of a photograph, the model must predict the exact texture of the wall, the precise grain of the wood, the individual blades of grass. Almost none of that is determined by the surrounding context -- it is essentially random. The model burns enormous capacity learning to hallucinate plausible noise, and the fraction of its effort spent on genuinely meaningful structure is small.

JEPA changes the question. Instead of "what pixels are behind the mask?", it asks "what would my own representation of that region look like?" The target is not the image; it is the model's own encoding of the image, produced by a second copy of the encoder that did see the hidden region.

The analogy is a friend describing a film you missed. If you had to predict their exact words, sentence by sentence, you would fail constantly and learn little, because word choice is mostly arbitrary. If you predict the gist -- who betrayed whom, how it ended -- you can succeed, and succeeding requires understanding the story. JEPA predicts the gist.

How it works

Three pieces. A context encoder sees the visible part of the input and produces a representation. A target encoder sees the hidden part and produces the representation you are trying to match; its weights are usually a slowly-updated moving average of the context encoder rather than a separately trained network. A predictor takes the context representation plus some indication of where the missing region is, and produces a guess at the target representation. Training minimises the distance between the guess and the target, in embedding space.

The obvious danger is collapse. If the goal is to make two representations match, the trivial winning strategy is for the encoder to output the same constant vector for everything -- a perfect score, zero information. Preventing collapse is the central engineering problem of the whole family, and the standard solution is asymmetry: the target encoder updates slowly and receives no gradient from the loss, so there is no direct path for the network to learn the degenerate answer.

This is where JEPA differs sharply from contrastive learning, which solves collapse a different way -- by explicitly pushing non-matching pairs apart. That works but requires large batches of carefully chosen negative examples. JEPA needs none, which is a substantial practical simplification.

I-JEPA, published by Mahmoud Assran and colleagues in 2023, is the canonical image version: predict the representations of several target blocks from a single context block, using a vision transformer as the encoder. data2vec, from Alexei Baevski and colleagues at Meta a year earlier, applied essentially the same idea across speech, vision and text with one recipe -- a striking result, since those three modalities normally demand entirely different tricks.

The theoretical backing

Until recently the case for JEPA was empirical and philosophical: it works well, and predicting abstractions feels more like what brains do. In 2026, Daniel Korchinski, Alessandro Favero and Matthieu Wyart supplied a proof. Using a grammar that generates visible tokens by recursively applying rules down a hidden tree of depth L, they showed that token-level learning needs a number of training samples exponential in L, while latent prediction needs a number constant in L, up to logarithmic factors.

Their analysis also found that data2vec implicitly performs hierarchical latent prediction without having been designed to -- and concluded that explicitly stacking hierarchical modules, as in the proposed H-JEPA, "is largely redundant." The hierarchy falls out of the objective; you do not have to build it into the architecture. We covered the result in full in predicting your own latents cuts the sample cost from exponential to flat.

Where it fits

JEPA is a way of learning representations, which makes it a cousin of world models rather than the same thing: a world model is about predicting how an environment evolves, and JEPA is a strong candidate for the representation such a model predicts in. Predicting future states in latent space rather than in pixels is exactly the argument, applied along the time axis.

The honest limitation is that JEPA representations are not directly inspectable. A masked autoencoder produces an image you can look at and judge. A JEPA produces a vector, and the only way to evaluate it is downstream -- fine-tune something on it and see whether it works. That makes debugging harder and progress noisier, which is part of why generative pretraining remains dominant despite the efficiency argument. See also sample complexity for the formal way of asking how much data a method needs.

Ground Truth. (2026, August 11). JEPA: teaching a model to predict its own understanding. Ground Truth. https://groundtruth.day/learn/joint-embedding-predictive-architectures.html

@misc{groundtruth:joint-embedding-predictive-architectures,
  title  = {JEPA: teaching a model to predict its own understanding},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/joint-embedding-predictive-architectures.html}
}

Topics: jepa · self-supervised-learning · representation-learning · architecture · sample-efficiency