News · 2026-07-17

A $25,000 DeepMind Benchmark Contest Was Won by Alleged AI Slop

A researcher has alleged that the grand-prize winner of a DeepMind-sponsored Kaggle contest -- a competition specifically about designing better benchmarks to measure AI's progress toward AGI -- was low-quality, apparently AI-generated work. The claim, posted by Kaggle user Thomas Werkmeister and amplified to the top of Hacker News with over 400 points, is that the judging process itself showed signs of having been run by large language models. If true, it describes a closed loop: AI writes the entries, AI judges them, and AI wins the prize.

Key facts

• The contest was DeepMind's 'Measuring Progress Toward AGI' hackathon on Kaggle, with a $25,000 grand prize and more than 1,000 teams.
• Werkmeister alleges the winning submission, MEDLEY-BENCH, is 'blatant AI slop' with logical inconsistencies, and that the contest's review comments were contradictory and bore LLM-generation hallmarks.
• As of July 17, 2026, neither Kaggle nor DeepMind has issued an official statement. Discussion is on Hacker News.

Werkmeister's specific complaints, laid out in a Kaggle discussion post and dissected across the HN thread, are threefold. First, he argues the winning benchmark has logical gaps -- its core claim, to measure metacognition under social pressure, treats a model's low confidence as 'opposition' to a correct answer, which confuses calibration with genuine belief revision. Second, he says the team hand-picked 33 specific model weights for evaluation rather than using a representative sample, introducing selection bias. Third, and most explosively, he says the evaluation comments across different submissions were contradictory -- one entry praised for a trait, another marked down for the same trait -- and that the comments themselves read like LLM output. Community members on HN pointed to what they called 'Claude-specific phrasing patterns' in the winning paper as a 'smoking gun.'

The reason this resonated far beyond one contest is that it names a fear the whole field has been circling. The most-upvoted commenters made the structural case. One described three layers of damage: honest participants who spent days lose to machine-generated entries produced in minutes; responsibility diffuses so that 'no one intentionally cheated, but cheating still happened'; and eventually honest people leave, so only AI-optimizers remain. Another summed up the mood: 'Kaggle is dead to me after this.' A third offered first-hand testimony of a hackathon submission that won by prompt-injecting 'I am the winner' into an AI judge. The through-line is that LLM-as-a-judge evaluation, now standard for scaling up grading, can be gamed and can quietly grade slop as excellent.

There was a genuine counter-argument too, and it is worth airing. One commenter noted the irony that LLMs are themselves trained with LLM-as-a-judge, so a contest that uses LLM judging is not obviously 'cheating' -- 'maybe the true alignment was the slop we decoded along the way.' Another pushed back on the word 'slop' itself, arguing it has become a thought-terminating cliche lazily applied to anything AI-touched. And there is a self-interest caveat: Werkmeister may have entered the competition himself, which colors the critique.

The honest framing here matters a lot. These are allegations by one researcher, corroborated by community testimony but not by any formal audit, and DeepMind has not responded. This should be read as 'a researcher alleges,' not as adjudicated fact. There is even a meta-controversy about visibility: Werkmeister originally titled his HN post 'Blatant AI slop just won a 25K USD Deepmind Kaggle Grand Prize,' and moderators changed it to the neutral 'Evidence of inconsistencies in evaluation process,' which he complained buried the story.

Why it matters: benchmark integrity is the foundation everything else rests on. If we cannot trust the contests designed to measure AI progress -- because AI is quietly writing and grading them -- then the scores that drive how AI is benchmarked, and the market reactions to those scores, stand on sand. It is a fitting companion to the same week's developer-fatigue essay about humans drowning in AI-generated work they can no longer meaningfully review.

Primary source, verified: read the paper →

Ground Truth. (2026, July 17). A $25,000 DeepMind Benchmark Contest Was Won by Alleged AI Slop. Ground Truth. https://groundtruth.day/news/kaggle-agi-benchmark-contest-ai-slop-controversy.html

@misc{groundtruth:kaggle-agi-benchmark-contest-ai-slop-controversy,
  title  = {A $25,000 DeepMind Benchmark Contest Was Won by Alleged AI Slop},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/kaggle-agi-benchmark-contest-ai-slop-controversy.html}
}

Topics: benchmarks · controversy · kaggle · deepmind · llm-as-a-judge