Learn · Beginner

Vision Transformers: what happens when you feed a picture to a language architecture

A Vision Transformer, usually shortened to ViT, is an image model that does not contain a single image-specific component. It slices a picture into a grid of small square patches, flattens each patch into a vector, and feeds that sequence of vectors into the same Transformer stack that powers language models. Google Research introduced it in 2020 in a paper whose title is the whole idea: An Image is Worth 16x16 Words. It matters because it collapsed vision and language onto one architecture, which is the reason a single model today can read a document, watch a video, and drive a robot arm.

The problem it solved

For most of the 2010s, image models were convolutional neural networks. A convolution slides a small filter across the image, looking for edges, then textures, then shapes. Its great virtue is a built-in assumption: a cat is a cat whether it sits in the top-left or bottom-right of the frame, and nearby pixels matter more than distant ones. That assumption is called an inductive bias, and it means convolutional networks learn efficiently from modest amounts of data.

The same assumption is also a ceiling. A convolution can only relate two distant parts of an image after many stacked layers have slowly widened its field of view. Meanwhile the language world had discovered that attention - letting every token look directly at every other token in one step - scaled remarkably well. The obvious question was whether images could be reformatted as a sequence and handed to the same machinery.

How it works

Take a 224-by-224 pixel image. Cut it into a grid of 16-by-16 pixel squares. That gives you 196 patches. Flatten each patch into a list of numbers and multiply it by one learned matrix to get a vector of the model's working width - this is exactly the embedding step a language model applies to a word.

One problem: attention has no built-in sense of order or location. Shuffle the patches and the raw math gives the same answer. So each patch vector gets a positional encoding added to it - a learned signature saying "I am the patch from row 3, column 7." Without it, a ViT sees a bag of textures rather than a picture.

From there it is a plain Transformer: alternating attention and feed-forward blocks, with residual connections and layer normalization. A useful way to picture the attention step is a room of 196 people, each holding one jigsaw piece. Every round, everyone shows their piece to everyone else and updates their own understanding based on what looked relevant. After a dozen rounds, each person knows roughly what the whole puzzle depicts. That is the crucial difference from convolution: a patch in the corner can consult a patch on the far side in the very first layer, not the twentieth.

The catch, and how the field fixed it

The original paper was blunt about the cost. Dosovitskiy and colleagues wrote that Vision Transformers "yield modest accuracies of a few percentage points below ResNets of comparable size" when trained on mid-sized datasets, and attributed this directly to the missing inductive bias. Trained on ImageNet alone, ViT lost. Trained on a 300-million-image internal dataset, it won - and kept winning as the data grew, exactly the pattern scaling laws predict.

That data appetite was the obstacle to everyone outside Google, and three lines of work removed it. Facebook AI Research's DeiT used heavy augmentation and a distillation token that learns from a convolutional teacher, reaching competitive accuracy on ImageNet alone. Microsoft Research Asia's Swin Transformer restored some locality by computing attention inside shifted local windows, which also made cost grow linearly rather than quadratically with image size - the same instinct behind sparse attention in language models. And self-supervised pre-training, most influentially Kaiming He's Masked Autoencoders, removed the need for labels entirely by masking out most patches and training the model to reconstruct them. Meta's DINOv2 later pushed unlabeled pre-training to features good enough to use off the shelf.

Why it still matters

The deepest consequence was not accuracy. It was that images became sequences of tokens, and sequences of tokens compose. OpenAI's CLIP trained an image encoder and a text encoder to agree using contrastive learning, producing one shared space where a picture of a dog and the word "dog" land in the same neighbourhood. Almost every multimodal system since runs a ViT-style encoder and pipes its patch tokens straight into a language model's context. That is the plumbing underneath vision-language-action models that control robots, and underneath every model that reads a chart or a screenshot.

The honest caveat is that patch tokens are not free and not magic. Attention cost grows with the square of the token count, so doubling the resolution roughly quadruples the bill - a large part of why long-video understanding is still hard, and why so much engineering goes into compressing frames before the model reads them. And a ViT that has seen millions of photographs can still be brittle in ways a person is not: recent benchmarks have repeatedly shown models scoring in the single digits on visual tasks humans find trivial. Treating an image as 196 words is a powerful trick. It is not the same as seeing.

Ground Truth. (2026, July 30). Vision Transformers: what happens when you feed a picture to a language architecture. Ground Truth. https://groundtruth.day/learn/vision-transformers.html

@misc{groundtruth:vision-transformers,
  title  = {Vision Transformers: what happens when you feed a picture to a language architecture},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/vision-transformers.html}
}

Topics: vision-transformers · computer-vision · transformers · architecture · multimodal · fundamentals