representation-learning
Predicting your own latents cuts the sample cost from exponential to flat News
A new proof shows that on hierarchically structured data, learning from tokens needs training examples growing exponentially with the depth of the hidden structure while predicting your own representations needs a number that stays constant.
JEPA: teaching a model to predict its own understanding Lesson
A joint-embedding predictive architecture trains a network to predict its own internal representation of a missing part of the input, rather than predicting the missing pixels or tokens themselves. Skipping the surface detail is what makes it dramatically more data-efficient than generative self-supervised learning.
Autoencoders and VAEs: teaching a model to compress the world Lesson
An autoencoder is a network trained to squeeze data through a narrow bottleneck and rebuild it, and a variational autoencoder makes that bottleneck a smooth space of probabilities you can sample from, which is why nearly every modern image and video generator does its work inside one.
Vector Quantization: Turning Continuous Data Into a Vocabulary Lesson
Vector quantization forces a neural network's continuous internal representations to snap to a finite set of learned reference vectors, converting images, audio, or video into sequences of discrete symbols that a language model can predict just like words.
Seven new papers cannot agree what a world model is made of News
PhiZero compresses video into a learned alphabet of state transitions, but six competing late-July papers argue the substrate should instead be executable simulator code, explicit game state, paired re-renderings, distribution coverage, architecture, or action intent.
Contrastive learning: teaching models by pulling likes together and pushing unlikes apart Lesson
Contrastive learning is a self-supervised training method that learns useful representations without labels by pulling matching pairs closer together in an embedding space and pushing mismatched pairs apart - the technique behind SimCLR and CLIP, and the classic alternative to generation-based approaches for teaching a model to perceive.
Orca proposes a single 'world latent space' to replace next-token, next-frame, and next-action prediction News
Researchers introduced Orca, a world foundation model that learns one unified latent space from multimodal signals and predicts the next world state rather than the next token or frame, outperforming similar-sized specialists on text, image, and action tasks.