News · 2026-07-18

Kaggle Names Winners of DeepMind's AGI Benchmark Hackathon, and They're About Knowing What You Don't Know

Kaggle has announced the winners of Google DeepMind's Measuring Progress Toward AGI hackathon, and the four grand-prize benchmarks are narrow, technical, and specific: they test whether a model knows when it's wrong, when to stay silent, and when it can learn a new rule on the fly. None of them try to score general intelligence directly.

Key facts

• Kaggle's winners announcement says the Measuring Progress Toward AGI Hackathon drew submissions from more than 1,000 teams across five cognitive tracks, and awarded four grand prizes.
• The four grand-prize benchmarks: MEDLEY-BENCH (does a model recognize it's wrong and hold firm under social pressure), LearningBench (can a model learn a new system's rules within one conversation), GAUGE (does a model know when to answer versus abstain), and Metaproteus (can a model predict its own likely responses).
• The hackathon was built on DeepMind's March 17 framework, which defines 10 cognitive abilities and targeted the five with the biggest evaluation gaps: learning, metacognition, attention, executive functions, and social cognition.
• The launch drew sustained pushback, including a 214-comment Hacker News thread arguing Google was outsourcing AGI evaluation to Kaggle contestants — the controversy we covered at launch.

Back in March, DeepMind pitched a different way to track AGI progress: instead of one leaderboard number, measure a model against a cognitive profile — ten separate abilities, each scored against how actual humans perform on the same tasks. The plan was to run a three-step protocol: test models on broad cognitive tasks, collect baseline scores from a demographically representative sample of human adults, then map where each model's ability level actually falls against that human distribution, ability by ability. To get there faster, DeepMind turned to Kaggle and threw the hardest five abilities open as a public hackathon, betting that outside researchers could design better tests for gaps in learning, metacognition, attention, executive function, and social cognition than an internal team alone.

That bet is what drew the backlash we covered at the time: a popular framing on Hacker News was that a trillion-dollar lab was crowdsourcing its core evaluation science from unpaid contestants, dressed up as a contest. Now that the results are in, the four grand-prize winners suggest the exercise produced something narrower and, arguably, more useful than the "AGI" branding implied. Every one of them is really about a model's relationship to its own uncertainty. GAUGE tests whether a model can tell the difference between a question it should answer and one it should decline — the same failure mode covered in our lesson on hallucination, where a model states a wrong answer with total confidence instead of admitting it doesn't know. MEDLEY-BENCH pushes on a related but distinct problem: does the model hold its ground when it's actually right, even when a user pushes back and insists it's wrong? That's a test of social resistance to pressure, not raw knowledge. Metaproteus goes a layer further, asking a model to predict what it itself would say — a kind of self-model check that has no analogue in standard multiple-choice benchmarks. And LearningBench measures something closer to an old-fashioned intelligence test: can the model figure out a brand-new system's rules from context alone, within a single conversation, rather than relying on anything memorized during training.

Think of the everyday analogy DeepMind's framework implies: a student who has memorized a thousand facts isn't the same as a student who knows which of those facts they're shaky on, and who says "I'm not sure" instead of guessing on a test. Most AI benchmarks today reward the confident guesser, because standard scoring only checks the final answer, not whether the model should have answered at all. These four benchmarks specifically target that blind spot, which is exactly the territory covered in how AI is benchmarked — the gap between a leaderboard score and whether a system actually knows the edges of its own competence.

The honest caveat is that this is still a company's own contest naming its own winners, published as a LinkedIn feed post rather than a formal paper or dataset release, and there's no newer DeepMind follow-up directly answering the March criticism about crowdsourcing its evaluation work. Whether MEDLEY-BENCH, LearningBench, GAUGE, and Metaproteus become benchmarks the rest of the field actually adopts — the way certain reasoning and coding tests did — depends on whether DeepMind or an independent group publishes the full datasets and methodology, not just a winners list. For now, the useful signal is narrower than "AGI progress measured": four specific tools for catching whether a model knows what it doesn't know, built by outside teams responding to a real gap DeepMind identified in March.

Primary source, verified: read the paper →

Ground Truth. (2026, July 18). Kaggle Names Winners of DeepMind's AGI Benchmark Hackathon, and They're About Knowing What You Don't Know. Ground Truth. https://groundtruth.day/news/kaggle-agi-hackathon-winners-announced.html

@misc{groundtruth:kaggle-agi-hackathon-winners-announced,
  title  = {Kaggle Names Winners of DeepMind's AGI Benchmark Hackathon, and They're About Knowing What You Don't Know},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/kaggle-agi-hackathon-winners-announced.html}
}

Topics: agi · benchmarks · deepmind · kaggle · evaluation · hallucination