News · 2026-07-31

Asking a model to check its own work lost every comparison against just sampling more answers

A single-author study published July 30 re-ran the comparison everyone cites and nobody controls properly: seven methods that make a language model critique, rewrite, debate or select among its own answers, measured against the simplest possible baseline at identical token cost. Across 36 paired comparisons, no method was reliably better than just sampling several answers and taking the most common one. Ten were reliably worse - and every one of those ten involved the model inspecting its own output.

Key facts

• Across all 36 comparisons, no method reliably beat repeated sampling at equal token cost; 10 were reliably worse.
• All 18 comparisons involving self-inspection were negative.
• Published July 30, 2026, by Iliya Mirzaei; code, prompts, all generations and verification scripts released.
• Primary source: Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B.

The methodological point is the whole result. Methods that make a model "think about its thinking" - plan, critique, revise, reflect on mistakes, debate copies of itself - all share one property: they generate a great deal more text than answering once. And generating more text raises accuracy on its own, independent of whether the reflection contains any insight. So a paper reporting that self-refinement beats a single chain of thought has demonstrated almost nothing, because it spent five times the tokens to get there. This is the central confound in the test-time compute literature, and it has been named before without being measured carefully.

This study measures it. Seven methods, open models at 1.5B, 3B and 7B parameters, two mathematics benchmarks, 150 questions each. Every generated token is counted, including tokens spent on critiques, reflections, debate turns and checking. Each method is then compared against repeated sampling at that method's own measured cost - if reflection burned 4,000 tokens, the baseline gets 4,000 tokens of independent attempts. All 36 comparisons are paired by question, with bootstrap confidence intervals and correction for testing many hypotheses at once. That is a designed experiment rather than a leaderboard.

The most interesting finding is that the two kinds of self-inspection behave differently as models grow. Choosing gets better: taking Best-of-N's eight samples and simply counting the most common answer beats letting the model pick its favourite by 8.0 and 11.3 points at 1.5B, but only 2.0 and 1.3 points at 7B - by then indistinguishable from zero. Rewriting does not recover: Self-Refine and a forced Reflexion remain 3.6 to 10.1 points below the baseline even at 7B. Small models are bad judges of their own work and get less bad with scale. Small models that rewrite their work make it worse, and keep doing so.

Buried in the results is the sharpest observation in the paper. Reflexion, implemented as published, never triggered its own retry on the smallest model: "It judged itself correct every time and silently became a single chain of thought." A method built entirely on the model recognising its own errors, running on a model that never recognised one, quietly degenerated into the baseline it was supposed to beat - while still being reported as a distinct technique.

The practical translation for anyone building with these models: if a task is verifiable and repeatable, the default should be several independent attempts plus a deterministic vote, not a reflection loop. Reflection is worth adding when it receives genuinely new evidence - test output, tool results, retrieval, or a separately trained verifier - rather than when it only receives the model's own previous words. That distinction also shows up in practitioner work this month, such as Henry Pan's harness-training experiments, where the gains came from fixing what the agent could see and do rather than from asking it to think harder.

The limits are real and stated. This covers small open models on mathematics, not frontier models, coding agents, external feedback or trained verifiers. Some long answers hit output limits, and the strict correction for multiple comparisons weakens several individual claims. What survives is not "reflection never works" - it is that the burden of proof has moved, and any method that spends extra tokens now has to beat spending those tokens the boring way.

Primary source, verified: read the paper → (arXiv 2607.28576)

Ground Truth. (2026, July 31). Asking a model to check its own work lost every comparison against just sampling more answers. Ground Truth. https://groundtruth.day/news/self-reflection-lost-all-36-comparisons-against-simply-sampling-more.html

@misc{groundtruth:self-reflection-lost-all-36-comparisons-against-simply-sampling-more,
  title  = {Asking a model to check its own work lost every comparison against just sampling more answers},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/self-reflection-lost-all-36-comparisons-against-simply-sampling-more.html}
}

Topics: reasoning · test-time-compute · evaluation · prompting · research · negative-result