self-supervised-learning
Pretraining: how a model learns almost everything it knows before anyone teaches it anything Lesson
Pretraining is the phase where a model reads an enormous quantity of text and learns only to predict the next word, and it is where essentially all of a language model's knowledge and fluency comes from. Everything afterwards adjusts behaviour, not knowledge.
Predicting your own latents cuts the sample cost from exponential to flat News
A new proof shows that on hierarchically structured data, learning from tokens needs training examples growing exponentially with the depth of the hidden structure while predicting your own representations needs a number that stays constant.
JEPA: teaching a model to predict its own understanding Lesson
A joint-embedding predictive architecture trains a network to predict its own internal representation of a missing part of the input, rather than predicting the missing pixels or tokens themselves. Skipping the surface detail is what makes it dramatically more data-efficient than generative self-supervised learning.
Contrastive learning: teaching models by pulling likes together and pushing unlikes apart Lesson
Contrastive learning is a self-supervised training method that learns useful representations without labels by pulling matching pairs closer together in an embedding space and pushing mismatched pairs apart - the technique behind SimCLR and CLIP, and the classic alternative to generation-based approaches for teaching a model to perceive.
A video generator, repurposed as a perception model, matches specialists with up to 500x less data News
GenCeption repurposes a pre-trained video generative diffusion model as a feed-forward perception system, matching specialist vision models on depth, surface normals, pose and segmentation while using 7x to 500x less training data - and generalizing from synthetic-only training to real footage.