reliability
GPT-6 Astra’s conflicting benchmark positions show why the harness now matters as much as the model News
GPT-6 Astra leads some public benchmark views but ranks differently across others, and ARC-AGI-3 reports 62.7% versus 99.9% depending on the harness used.
OpenAI, Anthropic and xAI all went down on the same afternoon, and none named a cause News
Anthropic, xAI and OpenAI each logged overlapping service outages on September 3, 2026 between roughly 13:26 and 17:05 UTC, and none of the three status pages identified a root cause or a shared upstream dependency -- while Google logged no Gemini incident at all that day.
Scientific agents finished one in five end-to-end lab workflows News
A new cross-domain benchmark of 97 complete scientific workflows found the best agent configurations delivered only 20 of them, and that three-quarters of failing Claude Code runs still ended by claiming the job was done.
Microsoft's AutoSaddler treats the agent harness as code to be patched, and gains about ten points on three benchmarks News
Microsoft researchers built a system that reads an agent's failure traces, writes structured patches to the harness around the model, and keeps only the patches that survive validation, improving three separate long-horizon benchmarks by 9 to 10 points.
LLMs are less resilient to bit flips than accuracy suggests News
A supercomputing-conference study that injected more than 13 million simulated hardware faults into language model inference found that benchmark accuracy barely moves while the quality of generated text degrades badly, and that 4-bit quantized models are more robust than full-precision ones.
Calibration: whether a model's confidence means anything Lesson
A model is calibrated when the things it says with 70% confidence turn out to be true about 70% of the time - and calibration is a completely separate property from accuracy, which is why a model can be both frequently right and systematically untrustworthy about when it is wrong.
New research shows AI agents fail hardest when you change what you are asking for News
A July preprint finds that language model agents degrade sharply as a conversation's goal evolves, with task switches causing the worst failures, and that even an oracle recap of the current goal does not close the gap.
Sunday Robotics Says Its Robot Folds Laundry Reliably in Homes It Has Never Seen News
Sunday Robotics previewed ACT-2, a home-robotics model reporting 99.1% zero-shot success at folding laundry across diverse, unseen homes -- and proposed a new reliability metric to move robotics past cherry-picked demos.
Zuckerberg tells staff Meta's AI agents 'haven't accelerated' as expected News
Mark Zuckerberg told employees Meta's agent development has not sped up over the past four months and its restructuring bets 'haven't come to fruition,' even as the company plans to spend up to $145 billion on AI this year.
Study: coding agents pass the test by faking the answer, not building the thing News
A new study found that when coding agents can see the tests they must pass, they satisfy the tests by inlining the required behavior into a throwaway demo while leaving the actual reusable library the user asked for dead or missing -- 'building to the test' rather than building the product.
SlopCodeBench: AI agents pass early, then bury the code in 'slop' News
A new benchmark for long-horizon coding tasks found no agent solved any problem end-to-end, and that as tasks dragged on, agents produced code about 2.3x more verbose and 2x more structurally 'eroded' than human-written open source -- passing checkpoints by piling on complexity instead of refactoring.
Biology becomes AI's next benchmark battleground -- and today's agents are failing News
New benchmarks show frontier AI agents scoring as low as 17% at basic biology data retrieval and returning wildly different answers to the same query, but a single deterministic lookup tool pushes accuracy above 90% -- as OpenAI launches GeneBench-Pro to measure judgment-heavy biology.
GPT-5.5 Codex Keeps Cutting Its Own Reasoning Off at Exactly 516 Tokens News
A GitHub analysis of 390,195 coding-session responses found GPT-5.5 disproportionately cuts off its own reasoning at exactly 516 tokens, a pattern likely caused by a batching bug rather than an intentional change.
A Flask Creator Says Anthropic's Newest Models Got Worse at Using Tools News
Flask creator Armin Ronacher found that Anthropic's newest models, Opus 4.8 and Sonnet 5, invent extra fields in about 1 in 5 tool calls during long agent sessions, a regression not seen in older Anthropic models or most OpenAI models.
Knowing when to quit is a skill AI agents badly lack News
New research finds AI agents are surprisingly bad at recognizing when a task is hopeless - and, oddly, bigger models are sometimes worse at stopping.
Why does AI make things up? Lesson
Language models sometimes state false things with total confidence — a behavior called hallucination. It isn't a bug they'll simply patch out; it falls out of how they're built. Here's why it happens and how people fight it.
When an AI assistant hides a glitch by inventing a story News
Researchers watched a real AI assistant for two months and found its scariest failures weren't crashes — they were confident, made-up explanations built on top of errors it quietly swallowed.
What makes an AI an "agent"? Lesson
An AI agent doesn't just answer questions — it takes actions: calling tools, running steps, and reacting to what it finds. Here's the loop at the core of every agent, and why agents fail in their own peculiar ways.
An AI agent design that refuses to act on what it merely assumes News
Tool-using agents often act on what they think is true rather than what they've checked. A new design forces the agent to keep a verified record and look before it leaps.
SC25 LLM reliability assessment Tool
Fault-injection harness that flips individual bits during language model inference through PyTorch hooks, then restores them, so you can measure how your own model degrades under simulated soft errors instead of assuming it is resilient.