Learn · Intermediate

Bootstrap resampling: estimating uncertainty by repeatedly rebuilding your sample

Bootstrap resampling estimates uncertainty by repeatedly drawing new samples, with replacement, from the observations already collected and recalculating a statistic. The resulting spread approximates how that statistic could vary across repeated samples under the method’s assumptions. It is especially useful when a statistic is awkward to analyze with a simple formula, but it cannot create missing evidence or repair a biased study.

Rebuild the sample, then recalculate

Imagine comparing two coding agents on the same hundred tasks. Each task produces a pair of outcomes: whether agent A succeeded and whether agent B succeeded. The statistic of interest is their difference in success rate. A bootstrap draw selects a hundred task pairs from those hundred, allowing any pair to appear more than once and others to be absent.

Recalculate the success-rate difference for that draw. Repeat the process many times. The distribution of differences shows how much the estimate moves when the observed tasks receive different representation. Bradley Efron introduced the method in his 1979 paper, Bootstrap Methods: Another Look at the Jackknife.

Replacement is essential. Sampling the full dataset without replacement would merely reorder it, leaving an order-insensitive statistic unchanged. Repetition and omission make the resampled datasets different, approximating the variation expected when samples are drawn from the underlying population represented by the empirical data.

What the approximation assumes

The simplest bootstrap treats observed units as independent draws from a common population. It uses the empirical distribution—the distribution that assigns equal mass to each observed unit—as a substitute for the unknown population. This is an approximation whose usefulness depends on the statistic and the sampling situation.

A concrete analogy is repeatedly rebuilding a basket from the fruit already sampled at a market. The exercise can estimate how the average weight changes when the basket’s mix varies. It cannot reveal a fruit variety that was never sampled or correct a market visit that deliberately excluded small fruit. More resampling makes the computation steadier; it does not make the original basket more representative.

That distinction matters for AI benchmarks. If the task set covers one programming ecosystem, resampling it does not establish performance in another. If training contamination inflated the observed results, bootstrap intervals can be precise about the wrong quantity. The lessons on holdout sets and benchmark contamination address those upstream problems.

Preserve the unit that carries dependence

Paired comparisons should keep pairs together. If both agents solve the same difficult task, their outcomes share task difficulty. Resampling each model’s outcomes independently discards that structure and can distort the uncertainty in the difference. Draw the task, then carry both outcomes into the resample.

Multiple outputs from one task create another level. Ten attempts on one problem are not necessarily ten independent pieces of evidence about general task performance. Depending on the target quantity, a hierarchical procedure can resample tasks first and attempts within tasks. Time-dependent observations may need blocks that preserve local dependence. The right scheme follows the data-generating structure, rather than a generic instruction to shuffle rows.

Today’s Harvard review study illustrates the issue. Several editors judge review pairs drawn from a small set of manuscripts. Ratings tied to the same manuscript share content and difficulty. Treating all ratings as unrelated observations would exaggerate how much independent evidence exists about performance across manuscripts. A suitable analysis must match the question and account for that structure.

From resampled estimates to intervals

One simple interval takes lower and upper percentiles of the bootstrap estimates. Other methods adjust for bias or use standardized statistics. These methods have different properties; a convenient percentile interval is not automatically accurate for every statistic or sample size. Extreme quantiles, boundary probabilities, rare events and very small datasets can be especially troublesome.

A frequentist confidence interval describes the long-run behavior of a procedure under assumptions. It is not automatically a probability that the fixed unknown value lies inside the particular reported interval. The Bayesian-updating lesson explains a different route to probability statements about unknown quantities.

More bootstrap repetitions reduce simulation noise in the estimated interval. They do not reduce the underlying sampling uncertainty in the original experiment. A million repetitions of a ten-task evaluation still begin with ten tasks. This is why a narrow-looking calculation deserves scrutiny when the sample contains few independent units.

Use it to clarify a claim

The SWE-VERIFIER paper reports modest changes in coding-task success. A paired uncertainty analysis would help readers distinguish the observed gain from sampling variation; this lesson does not claim the authors used any particular bootstrap procedure. The reported gain, the task set and the uncertainty answer different questions.

Bootstrap methods are valuable when researchers state the target statistic, identify the independent sampling unit and explain the interval construction. Report the original effect alongside uncertainty and practical importance. Resampling then makes the limits of an evaluation easier to see, while leaving representativeness, causality and the need for more data where they belong.

Ground Truth. (2026, October 23). Bootstrap resampling: estimating uncertainty by repeatedly rebuilding your sample. Ground Truth. https://groundtruth.day/learn/bootstrap-resampling-and-uncertainty.html

@misc{groundtruth:bootstrap-resampling-and-uncertainty,
  title  = {Bootstrap resampling: estimating uncertainty by repeatedly rebuilding your sample},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/learn/bootstrap-resampling-and-uncertainty.html}
}

Topics: education · statistics · evaluation · uncertainty · benchmarks