Learn · Beginner

Diffusion models: how AI turns noise into images and video

A diffusion model is a type of generative AI that makes an image or video by starting from a screen of pure random noise -- television static -- and removing that noise a little at a time, over many steps, until a clear picture is left behind. It is the technology behind Stable Diffusion, most modern AI image tools, and video systems including OpenAI's Sora and the real-time, voice-steerable Vidu S1. Understanding it is the key to understanding nearly all AI image and video generation today.

Start with the counterintuitive core idea: the easiest way to learn to create is to learn to destroy, and then run destruction backwards. During training, a diffusion model is shown millions of real images, and to each one it applies noise in small increments -- a bit of static, then a bit more, then more -- until the image is completely destroyed into random noise. This is the "forward" or diffusion process, and it is trivial. The model's actual job is to learn the reverse: given a slightly noisy image, predict what noise was added so it can be subtracted. Do that well, and you have a machine that can take one step from noisy toward clean.

Generation is then just running that reverse step over and over. You hand the model a fresh field of pure random noise -- an image it has never seen, because it is random -- and ask: what noise would I remove to make this slightly cleaner? It answers, you subtract, and you repeat, dozens of times. With each step the static resolves a little more, the way a photograph swims into focus in a developing tray, until a coherent image emerges that was never in the training set but shares its patterns. The seminal 2020 paper that made this work at high quality, Denoising Diffusion Probabilistic Models by Jonathan Ho, Ajay Jain, and Pieter Abbeel, built directly on a 2015 idea from Jascha Sohl-Dickstein and colleagues that borrowed the math of nonequilibrium thermodynamics -- literally the physics of how heat and particles spread out.

Two refinements turned this from a research curiosity into the tools people actually use. The first is guidance: to make the picture match a text prompt, the denoising steps are steered toward the prompt's meaning, so "a red bicycle in the rain" nudges every step toward that description rather than a generic image. The second is working in a compressed space. Denoising millions of raw pixels directly is enormously expensive, so latent diffusion -- the method behind Stable Diffusion, from Robin Rombach and colleagues -- first squeezes the image into a small coded representation, does the slow denoising there, then decodes back to full pixels. That single change is much of why image generation became cheap enough to run on a home GPU.

How does this compare to the alternative? Before diffusion, the dominant image generators were generative adversarial networks, which produce an image in one shot from a generator locked in a contest with a discriminator. GANs are fast but notoriously unstable to train and prone to producing a narrow range of outputs. Diffusion traded that single risky leap for many small, stable steps -- which is why it largely won, and why the field's big open problem is now speed. Each step is a full pass through a large network, and doing 50 of them is slow. Techniques closely related to diffusion, like flow matching, and step-reduction methods aim to get the same quality in a handful of steps -- and reaching real-time video, as Vidu S1 does at up to 42 frames per second, is largely a story of driving that step count and latency down.

Why it matters: diffusion is the generative engine of the visual AI era. The same denoise-from-noise recipe scales from still images to video (where the model also has to keep motion consistent across frames) and even to other domains -- there are diffusion language models that generate text by denoising, an alternative to the usual word-by-word approach. When you see a stunning AI image or a clip that did not exist yesterday, the odds are overwhelming that somewhere inside, a model spent a few dozen steps quietly subtracting noise from static.

A note on names, since they cause confusion: "diffusion," "denoising diffusion," "latent diffusion," and "score-based models" all refer to variations on this same noise-removal idea. The details differ; the picture to hold is constant -- destroy an image with noise during training, learn to undo it, then generate by undoing noise you never added in the first place.

Ground Truth. (2026, July 10). Diffusion models: how AI turns noise into images and video. Ground Truth. https://groundtruth.day/learn/diffusion-models.html

@misc{groundtruth:diffusion-models,
  title  = {Diffusion models: how AI turns noise into images and video},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/diffusion-models.html}
}

Topics: diffusion · image-generation · video-generation · generative-models · fundamentals