methodology
Everything on Ground Truth tagged “methodology” — 2 items.
OpenAI says a leading coding benchmark can no longer tell the best models apart News
OpenAI published an analysis concluding that SWE-Bench Pro, a widely-cited coding benchmark, has hit a roughly 70% noise ceiling where higher scores may reflect quirks rather than real skill, and retracted its recommendation to use the benchmark to rank frontier models.
Your AI judge might be reliable — and still be wrong News
The largest audit of AI language model judges to date — 21 judges, over half a million grading decisions — finds that standard reliability metrics are inflated by roughly a third, that the same judge can score differently on different benchmarks, and that high consistency and severe bias can coexist in the same system.