sample-efficiency
Sample complexity: how many examples does learning actually take? Lesson
Sample complexity is the number of training examples a learning method needs to reliably learn a given pattern. It is the formal way of asking why a child learns a word from three exposures while a language model needs the internet, and it is what separates a method that scales from one that merely works.
Predicting your own latents cuts the sample cost from exponential to flat News
A new proof shows that on hierarchically structured data, learning from tokens needs training examples growing exponentially with the depth of the hidden structure while predicting your own representations needs a number that stays constant.
JEPA: teaching a model to predict its own understanding Lesson
A joint-embedding predictive architecture trains a network to predict its own internal representation of a missing part of the input, rather than predicting the missing pixels or tokens themselves. Skipping the surface detail is what makes it dramatically more data-efficient than generative self-supervised learning.