methodology
Cross-validation: how you find out whether a model learned anything or just memorised the answers Lesson
A model's score on the data it trained on tells you nothing about whether it will work, because memorising is easier than learning. Cross-validation and holdout sets solve this by measuring the model only on examples it has never seen, and the discipline of keeping a final test set untouched is what separates a real result from a self-flattering one.
OpenAI says a leading coding benchmark can no longer tell the best models apart News
OpenAI published an analysis concluding that SWE-Bench Pro, a widely-cited coding benchmark, has hit a roughly 70% noise ceiling where higher scores may reflect quirks rather than real skill, and retracted its recommendation to use the benchmark to rank frontier models.
Your AI judge might be reliable — and still be wrong News
The largest audit of AI language model judges to date — 21 judges, over half a million grading decisions — finds that standard reliability metrics are inflated by roughly a third, that the same judge can score differently on different benchmarks, and that high consistency and severe bias can coexist in the same system.