News · 2026-08-11

Predicting your own latents cuts the sample cost from exponential to flat

Researchers at EPFL and Cambridge have proved that predicting your own internal representations is exponentially more sample-efficient than predicting tokens, on data with hidden hierarchical structure. Where token-level learning needs a number of training examples that grows exponentially with the depth of the hidden hierarchy, latent prediction needs a number that stays constant in that depth. It is the first clean theoretical account of why the JEPA family of methods closes part of the gap between machine and biological learning.

Key facts

• On a depth-L hierarchical grammar, token-level learning requires samples exponential in L; latent prediction requires a number constant in L, up to logarithmic factors.
• The paper delivers the first sample-complexity analysis of data2vec, showing it implicitly performs hierarchical latent prediction.
• Its conclusion: "explicit stacking such as H-JEPA is largely redundant."
• Primary source: Learn from your own latents and not from tokens: A sample-complexity theory by Daniel J. Korchinski, Alessandro Favero and Matthieu Wyart, discussed by Wyart in a 10 August 2026 Machine Learning Street Talk interview.

The gap the paper is about

The authors open with the discrepancy that motivates the whole line of work: generative models "achieve remarkable performance but at a cost in training data orders of magnitude larger than what biological learners require." A child learns language from a vanishingly small fraction of the text a large model consumes. Something about how the machine learns is inefficient, and the question is what.

One candidate answer has been circulating for years, mostly as an argument rather than a proof: models learn at the wrong level. Predicting the next token means modelling surface detail -- exact word choices, precise pixel values -- most of which is noise relative to the structure that actually matters. The alternative, as the paper describes it, is that "networks are trained to predict their own latent representations of related views or masked regions, as in data2vec and JEPA." Predict the gist, not the wording.

The analogy is studying for an exam. Memorising a textbook verbatim is expensive and generalises poorly. Reading a chapter, forming an understanding, and then predicting what the next chapter will be about is cheaper and travels further. See contrastive learning for the other major self-supervised family and embeddings for what a representation is.

Turning the argument into a theorem

To prove anything you need data whose hidden structure you know. The authors use a probabilistic context-free grammar: a system that "generates strings of visible tokens by recursively applying production rules along a tree of hidden symbols of depth L." It is a deliberate stand-in for the compositional structure of language and images -- letters make words make phrases make sentences.

The result is sharp. Supervised or token-level self-supervised learning "require a number of samples exponential in L to recover the latent tree." Latent prediction "achieves this with a number of samples constant in L, up to logarithmic factors."

Read that gap again. Not a constant-factor improvement. Exponential versus flat in the depth of the structure. Every extra layer of hierarchy multiplies the data a token-level learner needs, and costs a latent learner essentially nothing.

They confirm it three ways rather than one: a hierarchical clustering algorithm, an end-to-end network "whose predictor-clusterer modules predict their own latents at each level via gradient descent," and an analysis of data2vec -- which they show is already doing hierarchical latent prediction internally without being designed to.

The finding that will annoy people

The paper's closing line is the one architects should sit with: "This suggests that explicit stacking such as H-JEPA is largely redundant."

A reasonable design instinct says hierarchical data needs a hierarchical architecture -- build the levels in, one module per level. The analysis says the hierarchy emerges from the objective on its own. A single latent-prediction loss recovers the tree; wiring the tree into the network buys little. That is a claim about where structure should live, and it points away from architectural complexity rather than toward it. Compare with I-JEPA, which predicts target block representations from context, and with world models, the applied end of the same argument.

Why it matters

Almost every scaling debate assumes the data cost of learning is roughly fixed and the remedy is more data. This says the data cost is a function of the objective, and that a different objective changes it by an exponential factor on structured data. If that transfers even partially out of the synthetic setting, it reframes what scaling laws are measuring: not the price of intelligence, but the price of one particular way of buying it.

The honest caveat

The authors are explicit about the limits. The Random Hierarchy Model has a fixed tree topology, no recursion and no context-dependent rules -- all of which real language has. Extending the analysis to variable topologies and context dependence is future work, and the paper says so. This is a clean proof about a synthetic grammar designed to be tractable, not a theorem about English. What it provides is a mechanism and a direction, which is more than the argument had yesterday.

Primary source, verified: read the paper → (arXiv 2605.27734)

Ground Truth. (2026, August 11). Predicting your own latents cuts the sample cost from exponential to flat. Ground Truth. https://groundtruth.day/news/predicting-your-own-latents-cuts-the-sample-cost-from-exponential-to-flat.html

@misc{groundtruth:predicting-your-own-latents-cuts-the-sample-cost-from-exponential-to-flat,
  title  = {Predicting your own latents cuts the sample cost from exponential to flat},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/predicting-your-own-latents-cuts-the-sample-cost-from-exponential-to-flat.html}
}

Topics: theory · self-supervised-learning · jepa · sample-efficiency · representation-learning