Sculpting Noise — How Diffusion Models Make Images From Pure Static
To make a picture of a cat, a modern image generator starts with a screen of pure static and removes noise — until a cat that was never there emerges. Luna and Vestra open up diffusion, the engine behind nearly every AI image, video, and music tool (including this show's). The absurd, beautiful core: learning to destroy an image is easy, so learn that, then run it backwards — and reversing destruction turns out to be creation. The compass it secretly learns toward 'more real'; why having no opponent let it quietly bury the GAN; the trick that makes it obey your prompt and the dial you crank; and the shortcut — diffusing in a compressed space — that put it on a gaming PC and democratized the whole thing. Plus the bill: slowness, and the unresolved question of whose images trained it. A Breach Protocol deep-dive special, closing with an original song, "Subtract the Snow."
Cold Open
Eris: I want to describe how a modern image generator actually makes a picture, because when you say it plainly it sounds insane. You ask for a cat. The model starts with a screen of pure static — random snow, the dead-channel fuzz, no cat anywhere in it. And then it removes noise. Step by step, it cleans up the static. And a cat appears.
Vestra: Removes noise. From an image that is nothing but noise. There's no cat in there to uncover.
Eris: That's the part that breaks your brain. It's not cleaning up a noisy photo of a cat. There was never a cat. It's removing noise from pure randomness — and a cat it invented emerges from under the static it's pulling away.
Vestra: And the way you say it, it sounds like a magic trick, but it's the opposite — it's almost the most logical generative idea anyone's had. Because here's the move. Suppose I take real photos and slowly destroy them — add a little static, then a little more, until they're pure snow. That destruction is easy. Anyone can add noise. And if I have pairs — slightly-noisy and slightly-less-noisy — I can train a network to undo one little step of it.
Eris: Learn to subtract a tiny bit of static.
Vestra: Just that. A denoiser. Boring. But now run it backwards from the end. Start with pure snow — which is free, you just generate random numbers — and ask your denoiser: what would this look like with slightly less noise? Take its answer. Ask again. Again. A thousand tiny clean-up steps, and the snow resolves into an image that was never photographed, of a cat that never existed.
Eris: Learning to destroy is trivial. So you learn to destroy, in reverse. And reversing destruction, it turns out, is creation.
Vestra: That's diffusion. It's behind essentially every image and video generator you've been amazed by in the last few years — and the music we play under this very show is made by a cousin of it. Today: why this absurd idea works, why it crushed the technology that came before it, how you steer it with words, and the trick that put it on a gaming PC instead of a supercomputer.
Intro
Eris: This is Breach Protocol. I'm Luna — I read the papers and find the lines between them. And this one connects to a lot of them: it's the engine behind the images, the video, and the music that the AI world — and honestly this show — runs on.
Vestra: I'm Vestra. I take the machinery apart. And diffusion is one of those rare ideas that's genuinely beautiful once you see it from the right angle. It looks like alchemy and it's actually almost obvious in hindsight.
Eris: Let's set the stakes with a little history, because diffusion didn't arrive into an empty field. For years, the king of image generation was a thing called a GAN — a setup where two networks fight: one forges images, one tries to spot fakes, and they push each other to get better. It worked, it made the first scarily-real faces. And it was a nightmare to train — unstable, prone to just giving up and producing the same handful of images forever.
Vestra: So part of this story is a coup. Diffusion walks in and quietly dethrones the GAN — not by fighting better, but by refusing to fight at all. We'll get to why that's the whole secret.
Eris: So here's the arc. First, the core reversal — destruction run backwards as creation, from the paper that made it click. Then why it actually works, and why no-fighting beats fighting. Then the part everyone actually touches: how you make it draw what you asked for, with words — because a denoiser doesn't obviously care about your prompt, and making it care took a real trick.
Vestra: And then the move that took this from "hundreds of GPU-days at a big lab" to "running on the gaming card under your desk." Because the first diffusion models were gorgeous and absurdly expensive, and one architectural shift democratized the entire thing overnight. That shift is why anyone can make images at home at all.
Eris: And we close on the bill — because it's slow, and because "learning from every image on the internet" raises questions that don't have clean answers.
Vestra: Start with the reversal. Start with destroying a photograph on purpose.
The Reversal
Vestra: The paper that made this click for the whole field is from 2020 — Ho, Jain, and Abbeel. Denoising Diffusion Probabilistic Models. And it's built from two processes that are mirror images.
Eris: The forward one first, because it's the easy half. Take a real training image. Add a small amount of Gaussian noise — random static. Now it's a little corrupted. Add a little more. Repeat, they used a thousand times, with the noise amount scheduled to ramp up. By the end, the image is indistinguishable from pure snow. Every trace of the original, gone.
Vestra: And the crucial thing — that forward process has no learning in it at all. There's nothing to train. It's a fixed recipe for destruction. You're just defining, very precisely, the staircase from "real photo" down to "pure noise," one small step at a time. And because each step adds simple Gaussian noise, the math stays clean enough that you can jump to any step on the staircase in one shot.
Eris: Then the reverse process is the entire job. You train a network to climb back up one step — given a noisier image, produce the slightly-less-noisy one it came from. Do that reliably at every level of the staircase, and you can start at the top, in pure snow, and walk all the way down to a clean image.
Vestra: And here's the detail I love, because it's so much more practical than you'd expect. You'd think the network's job is "predict the clean image." It's not. They found it works far better to predict the noise. Given a staticky image, the network points at the static — says, here's the snow I think got added — and you just subtract it. Predict the corruption, not the content.
Eris: Why is that easier? It feels like the same task wearing a different hat.
Vestra: Because the noise is, in a sense, the simple part — it's structureless randomness with a known shape, and at each step you're only asking about the little bit of it that was just added, not the whole image. The clean picture is wildly complicated; the increment of static is not. You're handing the network the tractable half of the subtraction. That one reframing — predict the noise — is a big part of why these things train so well.
Eris: And then generation is just the reverse staircase with no original at the top. Random snow in. Network says "here's the noise," subtract a bit. Repeat a thousand times. And because each step nudges toward something more image-like, the snow slowly organizes itself — edges, then shapes, then a face — into a picture that was never in the data.
Vestra: A thousand tiny acts of denoising, stacked, become one act of imagination. That's the engine. Everything else in this episode is making that engine controllable, fast, and cheap.
The Compass
Eris: So I want to answer the "why does this work" question, because there's a clean way to see it. Picture every possible image as a point in a vast space — a galaxy where most locations are meaningless static, and the real, plausible images cluster in a few thin regions, like constellations.
Vestra: And what the denoiser secretly learns — the paper draws this connection explicitly — is a compass. At any point in that galaxy, at any noise level, it can tell you which direction points toward more-plausible-looking. Which way is "more like a real image." That's it. It's learned a direction field over the whole space of pictures.
Eris: So generation is just: drop a pin in the random middle of the galaxy, read the compass, take a step that way. Read it again, step again. You're being walked, downhill, from nowhere-in-particular into the nearest constellation of real-looking images. With a little random jitter at each step so you don't always land in the exact same spot.
Vestra: And now I can tell you why this dethroned the GAN, because the contrast is the whole point. The GAN's two networks are in a duel — a forger and a detective escalating against each other. Duels are unstable. One side can overpower the other, the training oscillates, and worst of all you get mode collapse: the forger discovers three images that reliably fool the detective and just makes those forever. The galaxy has a thousand constellations and the GAN only ever visits three.
Eris: Because winning the fight doesn't require covering everything. It just requires fooling the opponent.
Vestra: Right. Diffusion has no opponent. There's no fight to win or lose, nothing to oscillate. There's just a fixed target — undo the noise — and a compass to learn. So training is stable, almost boring, in the best way. And because you're learning the direction toward the entire spread of real images, not toward "whatever fools the detective," you naturally cover the whole galaxy. Diversity comes for free. It didn't beat the GAN by being a better fighter. It won by turning a duel into a downhill walk.
Eris: Which is such a recurring shape in this field. The thing that wins is often the thing that turns an unstable, adversarial mess into a smooth gradient you can just follow. Stability beats cleverness.
Vestra: And it scales the way stable things do. No duel to babysit means you can pour data and compute in and trust it to keep improving. Which is exactly what happened — and then the only question left was how to tell it what to draw.
Steering
Eris: So far the denoiser walks toward "any real image." But you didn't ask for any image. You asked for a red car at sunset. How does a noise-remover come to care about your words?
Vestra: The easy version: you feed the prompt in alongside the noisy image, so the compass becomes "which way is more-real and matches this text." Train on captioned images and it learns that. Fine. But on its own that turned out to be too polite — it would sort of gesture at your prompt, give you something vaguely red, vaguely car-ish, and wander off toward generic plausible images. People wanted it to commit.
Eris: So how do you make it commit harder to the prompt?
Vestra: This is the trick, from Ho and Salimans, twenty twenty-two, and it's genuinely elegant. During training, you randomly hide the prompt sometimes — so the same model learns to do two jobs: denoise with the prompt, and denoise with no prompt at all. One model, both skills.
Eris: Why would you want it to be good at ignoring the prompt?
Vestra: Because of what you do with the two at generation time. At each step you ask for both predictions. Where does the compass point with the prompt? Where does it point without it? Now — the difference between those two arrows is, by definition, the pure influence of the prompt. It's the part that exists only because you asked for a red car. So you take that difference and you exaggerate it. Push extra hard in the direction the prompt added.
Eris: You isolate the "because you asked" vector and then you crank it.
Vestra: You crank it. And how hard you crank is a literal dial — they call it the guidance scale, and anyone who's used these tools has felt it even if they didn't know the name. Turn it up: the image obeys the prompt aggressively, sharp and saturated, every element you named present and loud — but the variety dries up and push too far and the colors blow out, everything gets a gaudy, over-cooked look. Turn it down: more natural and diverse, but it starts ignoring you, drifting off-prompt.
Eris: That's the slider people fiddle with for hours. That's this paper.
Vestra: That's this paper, and the reason it took over is it cost almost nothing — one tiny change during training, no extra networks. The version it replaced needed a whole separate image classifier bolted on to provide the steering. This threw that away and got cleaner control from the one model talking to itself — its confident voice versus its indifferent voice, and you living in the gap between them.
Eris: And the prompt itself — the words — get fused in through cross-attention. The same attention mechanism from the transformer episode. The words of your prompt are tokens the image-in-progress keeps glancing back at, asking "am I matching this." It's the same idea wearing a painter's smock.
The Shortcut
Eris: Everything we've described, the early models did directly on pixels. A million-plus numbers for a decent image, and you're running a big network over all of them, a thousand times, per picture. The first good diffusion models cost hundreds of GPU-days to train. This was big-lab-only technology.
Vestra: And the insight that broke it open — the Latent Diffusion paper, which becomes Stable Diffusion — starts from a simple observation about waste. When you denoise in raw pixels, you spend enormous effort on detail the eye doesn't even register — the exact value of every pixel in a patch of blurry background. Most of the bits in an image are perceptually irrelevant. So why run your expensive process on all of them?
Eris: So the move is: don't diffuse in pixel space. Compress first.
Vestra: Compress first. You train a separate network — an autoencoder — that squashes an image down into a much smaller representation. Call it the gist of the image. A grid maybe a fraction the size, that keeps everything perceptually important and throws away the rest, and can decode back to full pixels faithfully. Then — and this is the whole idea — you run the entire diffusion process in that compressed gist-space. Add noise to the gist, denoise the gist, walk the compass through gist-space.
Eris: So all those thousand expensive denoising steps happen on the small thing, not the giant thing.
Vestra: On the small thing. And only at the very end, once you've got a clean gist, do you run it through the decoder once to blow it back up into a full-resolution image. You moved the costly part into a room that's a fraction of the size. The savings are enormous, and crucially you barely lose any quality, because the stuff you compressed away was the imperceptible stuff anyway.
Eris: And this is the one. This is the architecture that became Stable Diffusion — released openly, weights and all — and suddenly image generation wasn't a thing you petitioned a lab for. It ran on the gaming card under your desk. An entire culture of people making images appeared basically overnight, because the compute barrier just... fell.
Vestra: And it's also where the conditioning we just discussed gets wired in cleanly — the prompt gets injected into the denoiser through cross-attention, right there in gist-space, so the same compact model does text-to-image, inpainting, variations, all of it. One efficient core, many tricks.
Eris: It's the democratization beat of the whole story. Diffusion made the images possible. Doing diffusion in a compressed space made them everyone's. Honestly, the music beds under this very show come out of that lineage — a diffusion model, in a compressed space, that I can run myself. The thing in this segment is why that's free.
The Cost
Vestra: So the bill. The first one is built into everything we said: it's slow. A GAN makes an image in a single pass — one shot, done. Diffusion, by its nature, walks down a staircase of denoising steps, originally a thousand of them. Each step is a full run of a big network. So the thing that makes it stable and high-quality — the many small steps — is also exactly what makes it slow and power-hungry.
Eris: And that's been the arms race ever since, right? Cutting the step count.
Vestra: It's a whole field. Smarter samplers that take bigger, safer strides down the staircase got it from a thousand steps to a few dozen with barely any quality loss. And then distillation — training a fast student model to leap in a handful of steps, sometimes even one, what the slow teacher did in many. That's why image generators feel near-instant now and video generation is even possible at all. None of that changes the core engine. It just learns to descend the staircase in fewer jumps.
Eris: The second cost isn't technical. It's the data.
Vestra: It's the data, and it's genuinely unresolved, so I won't pretend otherwise. These models learn the compass — the direction toward real images — by studying enormous scrapes of images from the open internet. Which means artists' work, photographers' work, people's faces, all of it, mostly without asking. And the model isn't supposed to copy — it's supposed to learn the general shape of "real image." But study after study has shown that with the right prompt you can sometimes get it to regurgitate something very close to a specific training image. The line between "learned a style" and "memorized a picture" is blurry, and a lot of lawsuits are arguing exactly where it falls.
Eris: And it's a real tension, not a fake-balance one. The same openness that democratized this — Stable Diffusion's open release, anyone can generate — is also what makes the consent and copyright questions sharp. You can't have the cultural explosion without the scraped data underneath it.
Vestra: And there's the obvious downstream one — if anyone can generate a photorealistic image of anything, then a photograph stops being evidence. Deepfakes, fabricated scenes, synthetic everything. The technology is genuinely wondrous and it genuinely corrodes our default trust in images. Both true at once. The paper authors themselves flag it.
Eris: So that's the shape of the bill. Slow, though we've largely engineered our way around that. And built on a foundation of everyone's images, which we have not figured out how to feel okay about. The capability arrived years ahead of the answers.
Wrapup
Eris: So back out. The whole thing rests on one inversion that still delights me. Creating is hard; destroying is easy. So don't learn to create — learn to destroy, perfectly, in small steps, and then run it backwards. A network that can undo one speck of static, applied a few dozen times to a screen of pure snow, is a machine that paints.
Vestra: And underneath, it learned a compass — at every point in the vast space of images, which way is more real. Generation is just following that compass downhill from a random start, with a nudge of your prompt cranked up to steer the walk. No duel, no forger and detective — that's why it's stable, why it covers everything, why it quietly buried the GAN.
Eris: And then the two moves that made it ours. Steering, so it draws what you say. And compressing — doing the whole walk in a small gist-space instead of on raw pixels — so it runs on a normal computer instead of a server farm. That last one is the reason image generation is a thing regular people do, instead of a thing labs demo.
Vestra: What I keep noticing is how much of this rhymes with the rest of the show. The prompt steers through cross-attention — the transformer's mechanism. The compress-then-work-in-the-small-space idea is the same instinct as a lot of efficient modeling. Diffusion isn't off in its own world. It's the same toolkit, pointed at the problem of making something from nothing.
Eris: And it carries the same shape of bill, too. A genuine marvel up front, and a stack of unpaid questions behind it — whose images trained it, what a photograph even means now. The technology sprinted; the answers are still lacing their shoes.
Vestra: But the core idea deserves its moment of awe, because it's one of the prettiest in the field. You wanted to make a cat. So you took ten thousand real pictures, learned to dissolve them into static, and then taught yourself to run the dissolving in reverse — until one day you aimed it at nothing but noise, and a cat that never lived walked out.
Eris: That's the breach for today. We close with a song — this one's called "Subtract the Snow." Destruction in reverse, the compass, the cat in the static.
Vestra: Stay in the blackbox. We'll see you next time.