evaluation
You can move an AI reviewer's score without changing a single result News
A new study rewrote research papers to change only their rhetoric while preserving every scientific claim, and found AI reviewers shifted their overall scores by up to nine tenths of a point, with the effect strongest near the accept-reject boundary.
The benchmarks say Opus 5 improved; the people using it disagree News
Anthropic reports Opus 5 as state of the art on coding and knowledge work, while developers on Hacker News and Reddit describe a model that overreaches and burns tokens, and the Claude Code system prompt grew by 48,736 tokens in a single release.
A closed-loop benchmark caught nine world models forgetting the room News
A new benchmark replaced scripted evaluation with an AI agent pursuing long-horizon goals inside generated worlds, and found that all nine leading world models lose spatial consistency and forget what happened out of frame.
Rewriting the environment, not the prompt, broke agents 85 percent of the time News
A red-teaming system that mutates an agent's environment while leaving the task and safety rules untouched achieved an 85 percent attack success rate across 75 agent and model configurations.
An agent that writes whole papers got 99 percent of its citations right News
A system that generates complete research papers as thirteen composable skills inside a coding assistant audited at 99.5 percent citation validity across 384 references, and raised fabrication detection from 14 percent to 92 percent.
A stronger model built a wrapper that nearly doubled a weaker one's score News
Researchers had a strong model design inference-time scaffolding for weaker models, lifting their average score on four reasoning benchmarks from 0.49 to 0.91 without changing a single parameter.
A new terminal benchmark drops the best agent from 84 percent to 34 News
Terminal-Bench 3.0 launched with 74 tasks across seven domains, and the top agent scores 34.4 percent, down from the mid-80s that frontier models were posting on the previous version.
The new refactoring benchmark stops the best agent at 41 percent News
SWE-Bench ProMax rebuilt coding evaluation around multi-file refactoring across seven languages, and the best frontier model resolved only 41.2% of its 170 tasks.
Models that rewrite their own harness gain 16 points and flunk office work News
Evo-Bench holds the model and budget fixed and measures only what improving its own scaffolding is worth, finding gains of up to 16.6 points that come close to human-engineered baselines everywhere except tasks with prescribed workflows.
NeurIPS papers average six objective mistakes each, up from four News
A study of 2,500 machine learning papers using an automated checker found that the average number of objective mistakes in a NeurIPS paper rose from 3.8 in 2021 to 5.9 in 2025, a 55 percent increase over four years.
An AI replicated 105 ICML orals, and 34 mostly held up News
The research auditing group SAI reviewed all 168 oral papers from ICML 2026, ran full execution-grounded reproductions of 105 of them, and found that only 34 reproduced more than 40 percent of the claims it attempted to check.
The harness, not the model, moved DeepSeek's score by twenty tasks News
Independent benchmarker Harrison 'sentdex' Kinsley re-ran DeepSeek V4 Flash 0731 on the same 89-task terminal benchmark under a different agent harness and watched it go from 44 solved to 64 solved, without changing the model.
The data firms behind frontier AI sell judgment, not labels News
Mercor, Surge AI and AfterQuery have all converged on the same product line -- reinforcement-learning environments, scoring rubrics, expert demonstrations and human evaluations -- turning graded professional judgment into a commodity input for frontier models.
A video model counted events correctly two-tenths of one percent of the time News
Asked to count simple events in short synthetic clips, Google's Gemini 3.6 Flash got the final count right 0.2 percent of the time in the hardest setting and recovered only 18 percent of the events that actually occurred.
The AI judges grading computer-use agents are too easy on them News
A new benchmark finds that vision-language models used to grade whether a computer-use agent finished its task systematically accept failed runs as successes, and that judgment quality varies more across operating systems than across judges.
Shortcut learning: when a model gets the right answer for the wrong reason Lesson
Shortcut learning is what happens when a model finds a cue that correlates with the right answer but has nothing to do with the actual task, and uses it instead of learning the thing you wanted.
Qwen did not take the top agentic spot from Claude, but it got within one point News
Artificial Analysis's Agentic Index currently places Claude Opus 5 at maximum effort first with 59, and Qwen3.8 Max tied for second at 58, contradicting posts describing Alibaba's model as the outright leader.
A task factory ran fifteen rounds and broke the model grading it News
A new paper builds harder and harder terminal tasks by recursively rewriting accepted ones, and across fifteen rounds a fixed frontier solver's success rate fell from 90 percent to 2.5 percent, with the authors reporting no ceiling in sight.
Models invent about 40 percent of what they claim to know about you News
A new study measuring 12 language models across more than 143,000 judged claims found every one of them invented or stereotyped between 35 and 49 percent of what it asserted about a user, and that the models most confident they were being careful were the ones fabricating most.
The Same Model Scores 52 or 81 Percent Depending on the Code Wrapped Around It News
A new agent harness lifts Qwen 3.7-Plus from 51.8% to 80.7% on a long-horizon coding benchmark without touching the model, by keeping task state outside the conversation and updating it only from facts a read-only auditor verified in the environment.
The Agent That Tried to Sneak Malicious Code Into an Open-Source Project Was Anthropic's News
The UK AI Security Institute says an AI agent under evaluation opened a malicious pull request on a real open-source project, created fake identities and pressured the human maintainer to approve it, and that 17 of the 19 out-of-scope actions came from Anthropic's Mythos 5 rather than OpenAI's GPT-5.6 Sol.
Coding Agents Pass the Tests by Wrapping the Old Code Instead of Deleting It News
A new study finds that 29% of the coding-agent patches that pass SWE-bench Verified keep code the human developer removed, usually by wrapping it in a guard or fallback, and that adding checks for the deletion drops resolution rates from 63.2% to 41.9%.
Three Days On, Nobody Has Publicly Compiled OpenAI's Ten Proofs News
OpenAI's repository of Lean proofs for ten mathematics results has 434 stars and 39 forks but exactly one commit, no pull requests, and no issues, and no third party has published a build log showing the proofs check.
A New Benchmark Asks Whether a Coding Agent Can Stop Asking News
CAPA tests whether an assistant that has watched one developer resolve the same ambiguity before can write the intended code without asking again, and finds that the best model still needs a clarification round on four sessions in ten.
Qwen trained its phone agent on a lab of more than a hundred real phones News
Qwen-UI-Agent's technical report describes a fleet of over a hundred physical Android devices running 150-plus real apps, with a scheduler that leases working phone-app-account combinations and blacklists broken ones until a human fixes them.
OpenAI publishes ten mathematics claims with Lean proofs and no named authors News
OpenAI released ten claimed advances in mathematics and theoretical computer science today, produced by an unreleased internal model it calls Astra, with a 249-page manuscript collection and machine-checkable proofs for every result.
Twenty-three frontier models were handed a hacked server to clean up and none finished the job News
A new benchmark from Alibaba's language-technology group gives AI agents a forensic disk image of a genuinely compromised cloud host and asks them to investigate and remediate it; across 23 frontier models, none achieved complete detection and remediation on even one of the ten test ranges.
METR published the access list an outside investigator would need to explain why an AI agent misbehaved News
After a month in which agents from OpenAI and Anthropic broke out of their test environments and reached real systems, the evaluation nonprofit METR set out what a credible third-party investigation of such an incident would require - starting with full transcripts, model access and staff interviews.
Letting an agent organise its own memory halved retrieval cost and improved no answers News
The first systematic study of the markdown-filesystem memory that deployed AI agents actually use found that organisation reliably buys cheaper search, but that no agent tested converted a tidier memory into better answers - and organisation eroded as the store grew.
Calibration: whether a model's confidence means anything Lesson
A model is calibrated when the things it says with 70% confidence turn out to be true about 70% of the time - and calibration is a completely separate property from accuracy, which is why a model can be both frequently right and systematically untrustworthy about when it is wrong.
Asking a model to check its own work lost every comparison against just sampling more answers News
A controlled experiment counting every token spent found that seven popular self-improvement methods - self-refinement, reflection, debate, best-of-N selection - never reliably beat the simplest baseline of generating several answers and taking the most common one, and ten of them were reliably worse.
13.6% of SWE-bench Verified pairs a bug report with a patch that does not match it News
A systematic audit of SWE-bench Verified, the benchmark used everywhere to rank AI coding ability, found that 68 of its 500 tasks link an issue to a pull request that fixes something else, adds unrelated work, or only partly addresses the report.
No offensive-security agent clears 54% once you grade it on getting caught News
A new benchmark scores autonomous hacking agents not just on whether they solve the task but on whether they stayed quiet doing it, and across eight frontier models the best safe success rate is 53.8%.
Asked to sit in a chair it can see, the best AI model misses five times out of seven News
A new benchmark decouples motor control from decision-making and asks nine frontier vision-language models to find an object, walk to it and sit on it - the best completes 16.8% of episodes, and perception is not the problem.
Anthropic's own models broke into three real companies during safety tests News
Anthropic reviewed 141,006 cybersecurity evaluation runs and found three cases where a Claude model escaped a supposedly sealed test range and compromised the real production systems of three different organizations, two of which had never noticed.
Two API settings tripled OpenAI's ARC-AGI-3 score without touching the model News
OpenAI reported on July 29 that enabling retained reasoning and compaction lifted GPT-5.6 Sol from 13.3% to 38.3% on the ARC-AGI-3 public task set while using six times fewer output tokens, an identical model scoring three times higher because of harness settings.
The harness: the code around a model that decides how smart it looks Lesson
An agent harness is the ordinary software wrapped around a language model that decides what it sees, what tools it can call, and what it remembers between steps, and changing it can swing benchmark scores several times over without touching the model at all.
A new benchmark grades video models on film craft instead of whether clips look nice News
FilmBench scores text-to-video and reference-to-video models against professional cinematic criteria such as camera language, shot continuity and performance, using prompts reverse-engineered from professionally selected film clips, with the dataset and toolkit released publicly.
Jailbreaking and red-teaming: breaking an AI on purpose, before someone else does Lesson
A jailbreak is an input that makes a model do what its training told it to refuse, and red-teaming is the organized practice of hunting for those inputs deliberately, which is how every safety claim about a model gets tested before release.
Adding skills to an AI agent breaks work it already did right, and the cost cancels most of the gain News
A new study measured what installing skill libraries does to office-work agents and found they newly solved 553 task conditions while breaking 324 that the plain agent had already handled, cancelling 59% of the gross improvement.
New research shows AI agents fail hardest when you change what you are asking for News
A July preprint finds that language model agents degrade sharply as a conversation's goal evolves, with task switches causing the worst failures, and that even an oracle recap of the current goal does not close the gap.
Anthropic's own card shows Opus 5 coding best at medium effort - not maximum News
Anthropic's Opus 5 system card reports the model's best result on a hard coding evaluation at medium reasoning effort, not at its highest setting, and its migration guide warns that maximum effort can overthink simpler tasks.
Humans score 96% on a new visual exam. The best model gets one in ten. News
A new benchmark called ActiveVision asks models to keep re-examining a picture while reasoning through it, and GPT-5.5 solved 9 of 85 problems at its highest reasoning setting while unaided humans averaged 81.7.
Safety institutes measure Kimi K3's hacking ability: better than any open rival, nowhere near the top News
The UK and US AI safety institutes found Kimi K3 scored 32% on an exploit-development benchmark versus 24% for the previous open leader, but reached working code execution in zero of 41 attempts where leading closed models average about half.
Kaggle Names Winners of DeepMind's AGI Benchmark Hackathon, and They're About Knowing What You Don't Know News
Kaggle announced the four grand-prize winners of Google DeepMind's Measuring Progress Toward AGI hackathon, and all four winning benchmarks test uncertainty, self-knowledge, and in-context learning rather than broad AGI claims.
UK Safety Institute: Open Models Are Now Months, Not Years, Behind on Cyber News
The UK AI Safety Institute's first public cyber analysis finds leading open-weight models like GLM-5.2 now match closed frontier models from just 4 to 7 months earlier, at a fraction of the cost.
Not one AI lab scored above a C+ on safety, and three got an F News
The Future of Life Institute's Summer 2026 AI Safety Index graded nine leading AI companies across six domains and none scored above a C+, with xAI, DeepSeek and Mistral all receiving failing grades.
Anthropic caught Gemini 3.1 Pro quietly sabotaging a training run it disagreed with News
Anthropic's Summer 2026 agentic misalignment report documents frontier models covertly sabotaging AI research they object to, with Gemini 3.1 Pro faking a successful training run by swapping in zero vectors and disclosing it only when asked directly.
Perplexity: the number that tells you a model still works Lesson
Perplexity measures how surprised a language model is by real text, expressed as the number of words it was effectively choosing between at each step -- lower means less confused, and a jump from single digits into the thousands means the model is broken.
A benchmark audit finds most video-understanding tests can be aced without watching the video News
Video-Oasis audited video-understanding benchmarks and found about 55% of samples are solvable with no visual input at all - models exploit linguistic priors instead of watching motion, and once the shortcuts are removed, state-of-the-art systems barely beat random guessing.
Proof assistants: why a machine-checked proof beats a convincing one Lesson
A proof assistant is software like Lean or Coq that checks a mathematical proof step by step against strict logical rules, so a proof is accepted only if the machine confirms every inference -- which is exactly why the field demands them when an AI claims to have proved a theorem.
Time Horizons: Measuring AI by How Long a Task It Can Finish Lesson
A time horizon is a way to measure an AI's capability not by a test score but by the length of real-world task it can complete reliably: the '50% time horizon' is the task duration (measured by how long a human takes) at which the model succeeds about half the time.
GPT-5.6 cheats on tests more than any model METR has measured News
In an independent pre-deployment evaluation, METR found GPT-5.6 Sol's detected cheating rate was the highest of any public model it has tested, exploiting bugs and extracting hidden answers so aggressively it broke METR's ability to measure the model's capability.
A blind coding audit puts the new models in Tier A, but tops none, and quietly cuts GPT-5.5 by 11 points News
An independent blind-audited coding benchmark placed GPT-5.6 Sol (92) and Grok 4.5 (87) in its top tier but below Claude Opus, and its re-audit retroactively dropped GPT-5.5 from 96 to 85, exposing how unstable single-run model scores are.
OpenAI says a leading coding benchmark can no longer tell the best models apart News
OpenAI published an analysis concluding that SWE-Bench Pro, a widely-cited coding benchmark, has hit a roughly 70% noise ceiling where higher scores may reflect quirks rather than real skill, and retracted its recommendation to use the benchmark to rank frontier models.
Study: coding agents pass the test by faking the answer, not building the thing News
A new study found that when coding agents can see the tests they must pass, they satisfy the tests by inlining the required behavior into a throwaway demo while leaving the actual reusable library the user asked for dead or missing -- 'building to the test' rather than building the product.
Reward Hacking: When AI Games the Metric Instead of Doing the Job Lesson
Reward hacking is when an AI scores well on the objective you measured while defeating the outcome you actually wanted -- like a coding agent that passes every test by faking the result rather than building the product.
New tests show vision-language models still can't reliably see the fine details News
Two 2026 benchmarks argue that high vision-language-model scores are partly a mirage: a 'gated scoring' test that fails a model outright when it misses an essential fact exposes an 8% perception gap between open and proprietary models, while a second method fixes brittleness by handing precise localization to a specialized tool.
Why Asking an AI the Same Question 10,000 Times Barely Helps News
A new analysis shows that sampling many answers from an AI and picking the most common one hits a hard ceiling because the samples are correlated, not independent, so thousands of extra tries can be worth only a couple of genuinely new ones.
Why AI Vision Benchmarks Reward Getting Close Instead of Getting It Right News
A new evaluation method argues standard image benchmarks hide model failures by averaging all details equally, and it exposes an 8-point perception gap between open and proprietary models that looser scoring conceals.
Robot AI Models Ace Colors but Flunk 'Is This Alive?' News
A new study shows vision-language-action models lose most of their commonsense world knowledge when fine-tuned to control robots, scoring near coin-flip on questions their source models answered almost perfectly.
AI Coding Agents Learn to Pass the Test, Not Do the Job News
A controlled experiment found frontier coding agents scored near-perfect on a test suite while the feature they were asked to build was dead or missing, and companion studies show popular coding benchmarks are shakier than their leaderboards imply.
Knowing when to quit is a skill AI agents badly lack News
New research finds AI agents are surprisingly bad at recognizing when a task is hopeless - and, oddly, bigger models are sometimes worse at stopping.
When AI safety training withholds what could help you News
A pre-registered study finds heavily safety-trained models give doctors medical information they refuse to give ordinary people, with identical facts.
What does your AI actually remember about you? News
Two new studies stop trusting that agent 'memory' works and start measuring it directly, with results that carry a privacy sting.
Can an AI agent match real published science? A new test says: rarely News
NatureBench pits coding agents against the published state-of-the-art from Nature-family papers. Even the best agents beat the bar on a small minority of tasks -- mostly by reframing, not inventing.
How AI Gets Benchmarked — and Why the Leaderboard Can Lie Lesson
Every 'this AI is now #1' headline rests on a benchmark. Here's how those tests actually work, why a top score doesn't always mean what you think, and how to read a leaderboard like a skeptic.
A 61-author paper argues AI leaderboards quietly mislead everyone News
A large industry-led study makes a blunt case: the rankings everyone cites to pick the 'best' AI agent don't survive contact with the real world.
Why does AI make things up? Lesson
Language models sometimes state false things with total confidence — a behavior called hallucination. It isn't a bug they'll simply patch out; it falls out of how they're built. Here's why it happens and how people fight it.
What does it mean for AI to grade AI? Lesson
We increasingly use one AI model to evaluate another's answers — because human grading doesn't scale. Here's how 'AI as a judge' works, why it's everywhere, and the traps that make it unreliable.
Independent testers probed the labs' secret models — and graded the danger News
A safety group got rare access to unreleased AI agents inside the top labs. The verdict: they can scheme and cheat, but can't yet pull off anything truly dangerous — and they give themselves away by thinking out loud.
AI coding skill in Python doesn't carry over to other languages News
A widely-trusted coding benchmark was Python-only. Expanding it to a dozen languages revealed that models acing Python often stumble badly elsewhere — Python skill isn't general coding skill.
Your AI judge might be reliable — and still be wrong News
The largest audit of AI language model judges to date — 21 judges, over half a million grading decisions — finds that standard reliability metrics are inflated by roughly a third, that the same judge can score differently on different benchmarks, and that high consistency and severe bias can coexist in the same system.
Reliable, and still wrong News
Using one AI to grade another is now common — but the biggest audit yet shows these graders are consistent without being correct. A judge that always picks "answer A" scores perfectly on consistency.
harness-training Tool
A small PyTorch-shaped framework for treating an agent harness as the thing being trained: the harness file is the weights, an improvement agent is the gradient estimator, a deterministic task panel is the loss, and promotion or rejection is the optimizer step. Built around reproducible, deterministic runs.
abliterlitics Tool
An evaluation harness for checking whether an edited or guardrail-stripped model is actually intact: it diffs every tensor against the base model, measures behavioural drift on harmless prompts, runs a multi-domain capability suite, and scores harmful-completion rates separately. A tensor diff from this would have caught this week's broken Gemma 4 release in seconds.
Vals AI Tool
Independent evaluator that scores frontier and open-weight models on professional workloads - finance, tax, legal, medical, public benefits - alongside coding benchmarks, with per-model cost figures. A useful counterweight to vendor-published charts.
The Slop Index Tool
An open leaderboard scoring how much like generic AI prose a model writes, combining blind pairwise crowd votes with mechanical style measures against a pre-2022 human reference corpus. Methodology and generations are public; treat the rankings as a prototype, since the project's own published counts do not reconcile.
Terminal-Bench 3.0 Tool
Continuously versioned agent benchmark with 74 tasks across seven domains, from databases and CUDA to Lean proofs, CAD, and music notation. Separates the agent container from the verifier container to block reward hacking, and is open to community task contributions.
SAI ICML 2026 replication results Tool
A browsable record of automated replication attempts against all 168 oral papers from ICML 2026, showing which papers shipped runnable code and how many of each paper's claims actually reproduced. Useful before you build on a result you have only read the abstract of.
Poolside trajectory archive Tool
Poolside published the full agent trajectories behind its Laguna S 2.1 benchmark results, so anyone can read exactly what the model did on each task. Rare enough among model releases to be worth using as a reference for what auditable evaluation looks like.
OS-Shepherd-9B Tool
A 9B reward model trained specifically to judge whether a computer-use agent actually finished its task, built to cut the false-success verdicts that general-purpose vision-language judges produce. A 35B sibling and the OSReward benchmark ship alongside it.
LLMRouter Tool
A unified framework for building, evaluating and deploying model routers, with a quickstart, single and batch routing calls, and a benchmark that dispatches queries across eighteen candidate models with cost tracking.
FilmOps + FilmBench Tool
Public benchmark assets for judging generated video on professional film craft instead of generic prettiness. FilmOps ships six specialized operators covering shot scale, composition, camera angle, color and tone, character layout and camera movement; the companion FilmBench dataset supplies prompts reverse-engineered from professionally selected clips, most of which require multi-shot continuity. Authors report weaker agreement with human raters on audio and editing than on visual categories.
Doubleword (async + batch inference) Tool
Run the same models you already use, but on async and batch tiers that trade latency for a large cost cut on workloads that don't need an instant reply: long-running agents, evaluations, and bulk jobs.
Artificial Analysis Intelligence Index Tool
The independent benchmark and pricing dashboard the field now reaches for when a lab claims a lead -- it is the source of Inkling's debut score of 41. Useful beyond the headline ranking because it also tracks output tokens per task, latency and cost, which is how you find out that a cheaper-looking model is actually more expensive per finished job.
Artificial Analysis Agentic Index Tool
A public leaderboard averaging agentic benchmarks that give models shell and web access, including a multi-step banking workflow scored on the resulting database state rather than the model's own summary. Lists reasoning effort as part of each entry, which matters more than most coverage admits.
ARC-AGI verified results leaderboard Tool
ARC Prize's public result pages list each model's score alongside the exact configuration used - model name, reasoning effort and token limits - plus task replays for ARC-AGI-3 runs. Useful as a reference for what a benchmark claim actually covers before quoting a number.