Where AI Benchmarks Go Blind, and a Statistics Check for Two-Step Image Generators
AI in Detail -- Daily News & Research. A University of Illinois team mapped 25 popular AI benchmarks by question difficulty and found many are lopsided or have empty stretches where real model improvement cannot show up; their BenchFill tool rewrites existing questions to fill those holes, and in a controlled test a single added question made a hidden improvement detectable. Then: Intrinsic Distribution Matching adds a cheap check on the average and spread of a fast image generator's own internal features, giving small but consistent gains in prompt following and long-text rendering at one or two steps, with the caveats on training data and pooled statistics.
Listen (MP3) · Spotify · Pocket Casts
Your Benchmark Is an Empirical Measure Over Difficulty
Eris: A team led from the University of Illinois Urbana-Champaign checked twenty-five widely used AI benchmarks and found that many of them are lopsided in a way the headline score hides. Some pile most of their questions into one band of difficulty. Others have stretches of difficulty with no questions at all, so a model can get genuinely better at those problems and the score won't move. They also built a tool, BenchFill, that writes new questions to plug those holes.
Vestra: The reason this matters is that a benchmark score is just an average over questions. If two models do equally well on easy questions but one is much better on hard ones, a test that is mostly easy questions will report them as nearly tied. Nothing is wrong with the arithmetic. The mix of questions decides which differences get counted. The paper names that problem a mismatch: the test's spread of difficulty doesn't match the spread you actually care about.
Eris: And the second problem is the holes. Say you're training a reasoning model and it improves on problems of medium-high difficulty. If the test has no questions in that band, your progress chart stays flat, and you might kill a training run that was working.
Vestra: They prove something stronger than "it might be missed." If an improvement is confined to a stretch of difficulty narrower than the test's biggest empty stretch, then in the worst case no statistical test can detect it, no matter how many times you rerun the benchmark. Repeating the questions you have can't measure ability on questions you don't have. Once the improvement spreads wider than that biggest gap, enough repeated runs will eventually reveal it.
Eris: Which raises the obvious question. How do you put a difficulty number on a test question in the first place?
Vestra: They borrow item response theory from educational testing. Every model gets an ability score, every question gets a difficulty score, and the chance that a model solves a question depends on the gap between the two. That beats simply counting how many models fail a question, because a question that only weak models fail isn't really hard. They ran twelve open models, from under two billion parameters to nearly four hundred billion, up to thirty-two attempts per question, and fit the scores from those results. For agent benchmarks, which need strong proprietary models, they used published results from an existing agent leaderboard instead.
Eris: How stable is that? With only twelve models, one odd model could drag the scale around.
Vestra: They checked that directly. They dropped each model in turn and refit, and the difficulty rankings barely changed. They also tried more flexible versions that give each question extra parameters or several kinds of difficulty. On this data, either the simpler version fit better or the flexible fit wouldn't settle, so they stayed with the one-number version. That choice matters later, because it means difficulty here is a single scale relative to this panel of models. It isn't a universal property of the question.
Eris: Then they lay every benchmark's questions along a line from easiest to hardest and compare that to an even spread. Why even? A math competition team might reasonably want mostly hard problems.
Vestra: And the paper agrees that if you know your target, you should use it. The even spread is their default when you don't know where models will differ. They prove it's the only choice that never shrinks the true average difference between two models in the worst case. It's a neutral reference, not a claim that every test should be flat.
Eris: Ten of the twenty-five benchmarks were lopsided enough that, on average, their questions would have to shift by at least a tenth of the whole difficulty range to become even. Eight had a single empty stretch at least that wide. The most striking cases are the thirty-question math competition sets from the American Invitational Mathematics Examination, a hard high-school contest that labs use to track reasoning models during reinforcement learning. In those, the five biggest holes together cover close to half the difficulty scale.
Vestra: You'd expect a thirty-question test to have holes, and the authors say so. But it isn't only small tests. HumanEval, a coding test with a hundred and sixty-four problems, also has large gaps. And when they compared each benchmark against what random, evenly spread questions would produce, the real gaps were typically about four times wider. A few large science and medical question sets were far worse than that.
Eris: Writing new expert-level questions by hand is slow and expensive, so BenchFill has a language model rewrite existing ones instead.
Vestra: The target is the midpoint of a gap, because one question placed there shrinks the hole the most. The search keeps a memory of questions with known difficulty. Each round, it picks the four questions closest to the target as parents and asks a rewriting model, MiniMax, for four variants of each. The prompt includes the target difficulty, rescaled from zero to ten because models handle small decimals badly, plus nearby example questions just below and just above the target. GPT-5.4 checks that each new question's answer is correct. Then the same twelve models attempt the new question, and its difficulty is placed on the original scale without refitting everything. Good variants go back into the memory, so the next round starts from parents that are already closer. After at most eight rounds, it keeps the closest valid question, and at least three human experts check it before it goes in.
Eris: What does a rewrite look like?
Vestra: One original contest problem has a disk cut into four quarters and twenty-five random line segments drawn across it, and asks for the expected number of regions. That sat harder than the target. The rewrite cuts the circle in half and uses eight chords instead. It's the same kind of reasoning with less bookkeeping, and it landed almost exactly on the target difficulty. Another item changes one constant in a logarithm problem, which made it slightly harder and moved it into an empty stretch just above the original.
Eris: And the result: adding just three questions to each of eight benchmarks shrank the combined size of their three biggest holes by about a third on average. A comparison method with the same budget, which picked random parents and had no memory or nearby examples, shrank them about half as much. BenchFill came close to the ideal ceiling of dropping questions exactly at each gap's midpoint, and matched it on all three contest sets.
Vestra: Shrinking a gap is the paper's own yardstick, though, so the more convincing test is whether filled tests catch real progress. They lightly fine-tuned three models so each would improve around a known gap in one of the contest sets. On the original thirty questions, the improvement was not statistically detectable in any of the three cases. Add one BenchFill question and it was detected in all three. Add one question from the comparison method and it was detected in none.
Eris: That's the clearest evidence in the paper, and also the narrowest. It's three cases, one added question each, and the fine-tuning was deliberately aimed at the gap.
Vestra: There are other limits too. A question can sit at exactly the right difficulty and still test the wrong skill. Difficulty is one number estimated from one panel of models, so a different panel could reshape the scale. And the paper's case that a lopsided test can flip a ranking is a mathematical bound plus an illustration. They don't show a real published leaderboard that reversed.
Eris: There's one supporting result in the appendix. They trimmed eight math benchmarks toward an even spread and compared each one's model ranking to the combined ranking from the other seven. The trimmed versions agreed better, and the test that had agreed worst improved the most. Randomly trimming the same number of questions barely helped.
Vestra: They've released expanded versions of four contest sets, each between two-thirds larger and twice the original size, with every new item manually reviewed. For anyone reading a leaderboard, the practical takeaway is to ask where a test's questions sit on the difficulty scale. A flat training curve on a thirty-question test may mean the test has no questions where the model is improving.
Intrinsic Distribution Matching
Vestra: A team from Inclusion AI, Westlake University, Tsinghua and several partners has a new paper on image generators that make a picture in one or two passes through the network instead of dozens. The authors add a cheap statistical check to how these fast generators are trained: compare the average and spread of the generator's own internal features on its images against the same statistics on real training images, and penalize the difference. The improvement is small but consistent, and their two-step model renders long passages of Chinese and English text better than the other two-step methods they compared.
Eris: Some background on why a one-step image generator is hard. Diffusion models normally build an image by starting from noise and cleaning it up over many steps. The base version of one model in this paper, Z-Image, uses around a hundred. Each step is a full pass through a network with billions of parameters, so cutting to one or two steps makes generation dramatically cheaper. The cost is that the generator gets almost no chance to fix mistakes, and long text exposes that immediately. A chalkboard menu with rows of bilingual drink names and prices has to get every character right in a single shot.
Vestra: The standard way to train these fast generators is called distribution matching distillation. A big, slow teacher model already knows what real images look like. A second model keeps track of what the fast student is currently producing. For each image the student makes, the difference between those two models gives a direction: move this image toward places where real images are more common than the student's images. Repeated over millions of images, that pushes the whole collection of outputs toward the real collection.
Eris: And the approach this paper builds on removes that second model.
Vestra: Right. Earlier work called TwinFlow and APEX folded it into the generator itself. One network has two branches. Feed it the text prompt normally and it predicts how real images behave. Feed it the prompt's encoding with its sign flipped and the same network predicts how its own outputs behave. The difference between the two branches replaces the separate model. It's cheaper, but that second branch is still being learned while the generator changes underneath it, so the correction it supplies is a noisy estimate.
Eris: So where does the new statistical check come from?
Vestra: From some math on that correction. The authors split it into two parts. One part depends only on the average and the spread of the two image collections, as if each were a simple bell curve. The other part is everything else, the detailed shape. The split tells them which quantities they can enforce directly, and it also explains why they shouldn't stop there. Two collections can share an average and spread and still look completely different, so the shape part matters. They keep the full original correction and add an explicit check on averages and spreads alongside it.
Eris: Averages and spreads of what, though? Not raw pixels.
Vestra: Of features. They keep a slowly updated copy of the generator, which moves one percent of the way toward the current weights after each step. That's a standard trick for a stable reference. They add noise to a generated image and to a real one, run each through this copy, and average its last layer into a single feature vector per image. They store up to about sixteen thousand of these vectors from recent generated images and the same number from real ones. The penalty compares the average of each feature and how widely each feature varies between the two piles, and it pushes the current generated images to close the difference. Because the features come from the generator's own copy, they don't need an outside image-recognition model.
Eris: What kind of mistake is a check like that supposed to catch?
Vestra: Drift that shows up across many images rather than in any single one. The paper's side-by-side examples show one. Without the new check, a sushi poster comes out with oversaturated oranges and yellows. With it, the colors are less saturated, and in the menu example the small text holds together better. Those are hand-picked images, of course, so the evidence that counts is the measurements.
Eris: And the measurements show it helps, but only a little. In their controlled comparison, everything is held fixed: same six-billion-parameter model, same data, two steps. Adding the statistics check improved all five overall measures they report, including a test of following prompts that combine several objects, colors and positions, and the long-text rendering test in both languages. Every one of those gains is a fraction of a point. On the compositional test, removing the original distillation correction instead hurts about two and a half times as much.
Vestra: The authors read it the same way: the original correction does the main work and the statistics check refines it. The breakdown fits that. The gains are in attaching the right color to the right object, placing objects correctly and counting them. Drawing a single object was already essentially solved and didn't move.
Eris: Then there's the question of whose features to use. They tried DINOv2, a widely used outside image-feature model, and got almost the same result as the generator's own slow copy, which came out slightly ahead on all five. Using the teacher's features or the generator's live features did worse. So you get the benefit without loading a second large network during training.
Vestra: Then there's the comparison with other methods, which needs more care. At two steps, their version of Z-Image tops every method in the comparison on long-text rendering. It's clearly ahead in Chinese and only marginally ahead in English. But those rival numbers come from other papers with other training data. This team trained on more than five and a half million examples, most of them generated from text-rendering prompts. APEX, the method they build on, used two hundred thousand poster images. Part of the long-text lead could simply be the data. APEX also still beats them on a separate test of following long, detailed prompts.
Eris: And for scale, the original eight-step Z-Image Turbo is still a little better on English long text than their two-step model, though it's behind on Chinese. So two steps gets close to eight, but doesn't beat it everywhere.
Vestra: The speed numbers are real benefits, though. On the larger twenty-billion-parameter Qwen-Image model, the same trained checkpoint makes an image in about a third of a second with one pass on a single data-center GPU, or a bit under half a second with two, and the second pass improves quality. A lightweight adapter version, which trains only about two percent of the model's weights, comes within a hair of full training on the compositional test.
Eris: And the limits?
Vestra: The statistics are pooled across all prompts. Matching the overall average and spread doesn't mean each prompt gets the right range of images. Averages and spreads never pin down the full shape of a distribution, and the authors say that themselves. And the gains are small enough that you'd want repeated training runs to show they're reliable. The comparison table reports one number per configuration. So this is a cheap addition that seems to help a training method already in use, measured carefully against its own baseline. It isn't a new way to generate images.
Wrap-Up
Eris: That's the research for today. A benchmark score depends on where the test's questions sit on the difficulty scale, and a handful of well-placed questions can reveal progress that a test would otherwise miss. And for fast image generators, a cheap check on the average and spread of the generator's own features adds a small, consistent improvement on top of the standard training signal.
Vestra: Both papers are linked in the episode description. Every story we cover is also on our news site, Ground Truth, at groundtruth dot day, updated daily.
Eris: If this helped you read the next leaderboard more carefully, follow AI in Detail so tomorrow's episode shows up in your feed.