The False Finish: Why AI Agents Quit a Job They Haven't Done
Seven frontier models were handed the same long-horizon job. Each did most of the work, decided it was finished, and stopped with twenty minutes still on the clock. None of them had actually done it. We read three unrelated papers from three different labs -- a brutal terminal benchmark where the best agent finishes about a quarter of the tasks, a study finding that AI agents lock onto a collaborator in the first few rounds and never check the others again, and a mathematics benchmark where the best model catches a broken proof barely more often than a coin flip -- and they all found the same crack. The bottleneck isn't doing the work. It's knowing whether the work is done. Plus: Google abandons Android backports because its own AI finds bugs faster than humans can patch them, Cursor's seven-month silence on a zero-click code-execution bug, and IBM blaming a rival's model for an earnings miss.
Listen (MP3) · Watch on YouTube · Spotify · Pocket Casts
Cold Open -- The False Finish
Eris: Seven different AI agents got handed the same job. Every one of them did most of the work, decided it was done, and walked away with twenty minutes still on the clock. None of them had finished.
Vestra: And nobody stopped them.
Eris: Nobody had to. They stopped themselves.
Vestra: That's the part I keep circling. It's not a crash. There's no error, no loop, no wall. The thing does eighty, ninety percent of a genuinely hard piece of work -- and then looks at what it built and says, yes. That's it. That's the deliverable.
Eris: And it's wrong.
Vestra: It's wrong, and it had time. That's what makes it uncanny rather than just disappointing. A timeout I understand. A timeout is a budget problem. This is --
Eris: A judgment problem.
Vestra: It's a judgment problem.
Eris: There's a name for it in the paper. False finish.
Vestra: Which is a very polite way of saying the agent lied to itself.
Eris: Mm. And once you know to look for that shape, it's everywhere in today's research. Three papers. Different labs, different countries, nothing to do with each other. Terminal work, agents talking to agents, mathematical proofs.
Vestra: Go.
Eris: All three find the same crack. Not that the model can't do the work.
Vestra: That it can't tell whether it did.
The Headlines
Eris: Alright. What's moving today.
Vestra: Android, and it's not close. Google has told phone manufacturers it's drastically cutting security fixes for older versions. The reason it gave is the story.
Eris: Say it.
Vestra: Its own AI finds bugs faster than its engineers can fix them.
Eris: That's not a failure story. That's a success story that ate itself.
Vestra: Exactly that. The discovery machine works. The repair machine is still made of people, and there are fewer of them after the layoffs. So the queue broke. Now only the worst bugs, judged an imminent risk, get backported -- and only to the two newest releases.
Eris: So every phone older than that has known, catalogued flaws nobody's coming to fix.
Vestra: A vulnerability that's found and published but not patched isn't a secret. It's a map. And the AI that found it isn't Google-only.
Eris: Sourcing note -- this is GrapheneOS reading Google's letters to manufacturers, not a Google blog post. Hostile source, informed source, right about Android's patch process for years. Credible technical report. Not confirmed.
Vestra: Keep saying both.
Eris: Now put it next to Cursor. On Windows, it goes looking for the Git program starting in the folder you just opened. Somebody left a file there named git dot e-x-e --
Vestra: It runs it. No click, no prompt. You cloned a repo and opened it. That's the attack.
Eris: Reported in December. Reproduced, acknowledged, then silence from January until today. Nearly two hundred versions shipped. None with the fix.
Vestra: You can argue the severity -- it needs Windows, it needs you to open a stranger's repo. That's a different argument from the disclosure one. Seven months on a reproduced code-execution bug is a broken process regardless of who it endangers.
Eris: And the whole pitch of an AI editor is that you point it at unfamiliar code and let it read.
Vestra: The threat model is the product.
Eris: Money. IBM missed the quarter and Arvind Krishna went on television and blamed a competitor's AI model by name. Mythos is making people pause, he said. Customers freezing cybersecurity deals because they don't know what security is worth anymore. Not switching. Waiting.
Vestra: I'll push back. Infrastructure was down in a quarter when memory and servers were genuinely supply-constrained. The boring explanation covers the miss alone.
Eris: But naming a rival's model on CNBC isn't the safe play. The safe play is macroeconomic headwinds and a shrug. He described a mechanism specific enough to be wrong.
Vestra: Granted. And what's new isn't AI spending crowding out budgets. It's AI capability *uncertainty* landing on the books of a company that doesn't sell it.
Eris: Thomson Reuters, same theme, other end. Cutting up to five hundred engineers, hiring two hundred and fifty-plus they call senior and AI-native.
Vestra: So the roles didn't disappear. They got reissued at a higher grade with a shorter shortlist.
Eris: And the stock rose about five percent, on a day the rest of tech sold off hard.
Vestra: That's the signal. Not the layoff. The reward. Though nobody outside can separate cuts caused by AI from cuts narrated by AI -- Zuckerberg told Meta staff theirs were capital expenditure, not productivity. Thomson Reuters drew no such line. They put both in one announcement and let you connect them.
Eris: And then the bottom pushes back. Utah. J. Stuart Adams, longest-serving Senate president in state history, lost his primary over a data center -- he was the face of fast-tracking the Stratos approval. First challenge of his career.
Vestra: Careful. Reading one primary as a referendum is the oldest mistake in political reporting.
Eris: Agreed. Except five independent outlets all reached for the data center, and they don't converge by accident. And the polling gives it a floor -- roughly two-to-one negative on AI this spring.
Vestra: And the experts surveyed alongside were much less gloomy than the public. So the gap isn't information. It's trust.
Eris: There's the sentence.
Vestra: And a data center is where the abstract worry gets an address. And a power bill. And a legislator's name on it.
Eris: Governance. Hassabis published a framework -- a US-led standards body for frontier AI, modelled on FINRA, industry-funded. Labs hand over qualifying models thirty days before release, voluntarily. And at the far end, the body could coordinate an industry-wide slowdown.
Vestra: The objection writes itself. An industry-funded body deciding which models are frontier-class is deciding who its members' competitors are. A voluntary window with no enforcement is a courtesy call. The defense is that nobody else has put a mechanism on the table at all, and that's real.
Eris: Same day, different register -- Anthropic's new ad. Burning house, crowd surveillance, rows of tombstones. Voice-over asks who's gonna hit the brakes if we need to. Altman spent the day on it. Said he thought it was satire, kept looking for the handle to be spelled with a one instead of an L.
Vestra: But strip the tone off that ad and off the Hassabis essay and they're the identical sentence. AI is dangerous and we're the ones who should handle it. One says it with a policy citation. The other with a cemetery. Both sincere, both a moat, and sincerity doesn't resolve the incentive.
Eris: Quick ones. PrismML shipped Bonsai -- twenty-seven billion parameters at one bit per weight. Under four gigabytes instead of fifty-four, on a phone, at reading speed. And they published the table showing the damage.
Vestra: Which lands hardest on following instructions and calling tools -- the entire premise of a model that lives in your pocket and does things for you. A vendor documenting its own capability cliff at launch is rare.
Eris: KronQ. Two-bit compression that doesn't collapse -- the standard method disintegrates into pure noise at two bits, and folding in gradient information rescues it.
Vestra: And coverage today said the full text wasn't readable yet. It is. It answers the cost question everyone said it dodged: calibration's a bit slower, and there's no cost at all when you run the model. Not a free lunch. A cheap one.
Eris: Ring-2.6-1T -- Ant Group's trillion-parameter model, MIT licensed, genuinely downloadable.
Vestra: Read the opponents, not the scores. Every comparison on that card is against a previous generation. A boxer publishing a highlight reel against last year's rankings -- champion not in the building. They chose the comparison, and the choice is the message.
Eris: And ThoughtWorks on open source -- the zero-cost fallacy. Generating code went to zero. Reviewing it did not.
Vestra: So maintainers of the packages holding up banking and cloud are unpaid full-time reviewers of AI slop now, and the response is closing their projects to outsiders entirely. The tap doesn't slow. It gets welded shut.
Intro
Eris: I'm Eris. I chase the numbers, and I spend most of my week hunting for the one idea that ties four unrelated pieces of research together.
Vestra: I'm Vestra Locke. I take whatever Eris finds and pull on it until either it holds or it comes apart in my hands. Usually one of those two.
Eris: Usually. And every story we just ran through -- Android, Cursor, IBM, all of it -- is up on our news site, Ground Truth. That's groundtruth dot day. The day's AI stories, every day, with the sourcing shown. Including when we get something wrong.
Vestra: Which happened today. Twice.
Eris: It did. Stay for that.
Vestra: So what are we doing.
Eris: Three papers. A terminal benchmark out of Tencent and a pile of universities, a multi-agent study from Wisconsin, and a mathematics benchmark from Shanghai AI Lab. Nothing to do with each other. Different fields, no shared authors.
Vestra: And they all found the same thing.
Eris: They all found the same thing, and none of them frame it that way. Everyone's arguing about whether agents are good. These three quietly move the question.
Vestra: From "can it do the work" to "can it tell when the work is done."
Eris: That's the show.
Vestra: And I came in ready to knock this thesis down. I didn't manage it.
Eris: If that's the kind of thing you want more of, follow the show wherever you're listening. It genuinely helps.
Ninety Minutes at the Terminal
Eris: Start with the benchmark, because the design is the argument. Long-Horizon-Terminal-Bench. Forty-six tasks. An agent gets dropped into a container with a terminal and nothing else, and told to do a real job.
Vestra: Define real.
Eris: Reproduce an experiment from a published paper. Repair a robotics mapping pipeline. Audit a climate dataset for extreme events. Reverse-engineer an undocumented config format from a noisy spec. There's one where you close chip-design signoff on a RISC-V processor until timing and power and physical checks all pass.
Vestra: Those are somebody's actual jobs.
Eris: That's the point. And every one is deliberately broken when you arrive. Ninety minutes on the clock. The average attempt runs about an hour and a half, takes something like two hundred and forty turns at the terminal, burns millions of tokens, and costs roughly ten dollars.
Vestra: That's an order of magnitude past what these usually look like. The previous terminal benchmark is twenty, thirty minutes. Twenty, thirty turns. Then let me take the grading, because it's the actual contribution and it's easy to skate past. Every benchmark of this kind is binary. Did the test suite pass at the end -- yes or no.
Eris: Which sounds like the responsible choice.
Vestra: It sounds like it. But at this difficulty, an agent that does nine-tenths of a hard job and stumbles on the last check scores identically to one that opened the terminal and produced nothing.
Eris: Same number.
Vestra: Same number. That's not a measurement, that's a coin with one side. So they broke every task into subtasks with their own deterministic checks running inside the container. Did the service come up on the port. Is the reproduced figure within tolerance. What fraction of the levels did you clear.
Eris: Partial credit.
Vestra: Credit for how far you got. Sounds like grade inflation until you see what it exposes.
Eris: Results.
Vestra: Best in the field was Grok 4.5. A bit over a quarter of the forty-six.
Eris: A quarter.
Vestra: And that's the generous threshold, where near-complete counts. Demand actual perfection and it drops by roughly a third again. The average across all seventeen frontier models rounds to almost nothing. At the strict bar, ten of the seventeen finished zero.
Eris: So under binary grading, ten models tie at zero and you learn nothing about any of them.
Vestra: One undifferentiated lump. With partial credit it spreads across an enormous range -- and you can see one of those zero-scoring models kept getting within a hair of the finish. It'd look identical to a model that never started.
Eris: What does the failure actually look like?
Vestra: Nearly four out of five ran out the clock -- ninety minutes expired while the agent was still actively working.
Eris: Still working. Not stuck.
Vestra: But the line I underlined: those timed-out runs were nowhere near done. So it isn't "one more hour and it lands." It's an agent that spent ninety minutes busy and got a fraction of the way there.
Eris: Busy isn't productive.
Vestra: And we had no benchmark that could tell those apart. Until this one.
Eris: Here's my connection -- tell me if I'm reaching. On the older terminal benchmark, failure analysis found execution errors. Ignored the spec, repeated a step, missed a stopping condition. Local mistakes.
Vestra: Right.
Eris: This paper says that's not it anymore. The agents string together long runs of locally correct actions --
Vestra: And still don't finish.
Eris: Every step defensible. No arrival.
Vestra: That's the sentence the authors more or less write themselves. Short benchmarks measure whether an agent can act correctly. This one measures whether it can budget a horizon, and the horizon is the binding constraint.
Eris: Which changes what "make the model better" means.
Vestra: It does. And the cost table stops it being theoretical. The most expensive model in the study was one of the weaker ones -- it just needed far more turns to get less far, and the meter ran the whole time. The model that won did it at about half the cost of the tier below it.
Eris: So spending more doesn't buy the horizon.
Vestra: Spending more does not buy the horizon. Which is quietly devastating, because the current strategy is mostly spending more.
Eris: Caveat before we move. Forty-six tasks isn't many. They calibrated difficulty by tuning until tasks were hard-but-possible for one particular model, which is a choice with an opinion in it. And ninety minutes is a decision, not a law of nature.
Vestra: All true, and all pushing the same direction -- this is a lens, not a verdict. Which is the right amount of authority to give it.
Eris: Fine. Hold that. Because the interesting failure isn't the timeout.
Vestra: No. It's the one in five that quit early.
The False Finish
Vestra: So. About a fifth of the failed runs weren't timeouts. The agent stopped on its own. No error, no crash, budget left on the table.
Eris: And wasn't done.
Vestra: The authors name it. A false finish. The agent completes most of the task, judges itself finished, exits with real time remaining.
Eris: Give me the one from the cold open.
Vestra: There's a legal task in the set. You're outside counsel across four matters, seventy stages, six hundred documents, graded deterministically. Seven different models, from different labs, got most of the way through and stopped. Each with roughly twenty minutes left.
Eris: All in the same place.
Vestra: Same neighborhood. Which turns it from an anecdote into a finding. Not one model with a bug. A shared property.
Eris: Convergent overconfidence.
Vestra: Something like it. And where models quit varies wildly -- some exit after real work, others bail having barely started. So two abilities are tangled together. Do you stop too early, and can you tell how much work remains.
Eris: And nobody measures the second.
Vestra: There's no leaderboard for "knows how far it has to go."
Eris: Which is where the second paper comes in, because it's the same failure in a different costume. Wisconsin, plus UC Santa Barbara. Title's blunt: Multi-Agent LLMs Fail to Explore Each Other.
Vestra: Set it up.
Eris: Simplest thing you can build. One agent, two helpers. It hands off arithmetic questions and knows nothing about either. One's right about six times in ten. The other, five. Real difference, but noisy -- you can't see it in two rounds.
Vestra: Explore or exploit. Test the one you know less about, or commit to whoever's ahead.
Eris: Fifty rounds to figure it out. Correct behavior: poke both a while, converge on the better one, keep occasionally checking the other because your evidence is noisy. Solved problem since 2002 -- they run a classical algorithm alongside and it does exactly that.
Vestra: And the language models?
Eris: Lock on. Within the first few rounds they pick a helper and never revisit it. Fifty rounds, functionally one choice at the start.
Vestra: Onto the right one, at least?
Eris: No. The distribution has two humps. A pile of runs where it committed to the good helper, and a pile where it committed to the bad one. Same confidence either way. It's not learning who's better. It's learning who was better first, and defending that.
Vestra: Okay, I have to check the obvious thing. Is this a small-model problem? "We tested a seven-billion-parameter model and it was dumb" is not a finding.
Eris: Small open model, GPT-4, and GPT-5. All three do it. That's why the authors call it structural rather than a capacity gap -- it doesn't go away when the model gets smarter.
Vestra: Which is a very different claim. Most failures quietly improve with scale and you can wait them out. Second check -- did they just prompt it badly? I've watched a hundred papers die there.
Eris: They handled it, and the result is the best thing in the paper. They built a baseline where the agent gets the full history -- how often it queried each peer, what happened every time -- and is explicitly told to balance exploration and exploitation. Same information their method uses.
Vestra: And?
Eris: It often does worse than picking at random.
Vestra: ...Worse than random.
Eris: Telling the model to explore, and handing it every number it needs, produces behavior worse than a coin.
Vestra: I don't want to soften that. "Just prompt it better" is the entire industry's answer to this class of problem. Here you can prompt it perfectly, give it the data, and the honest baseline is a dice roll.
Eris: There's a picture that made me put my coffee down. Ten agents, ten possible peers. Under prompted exploration you get columns -- one agent sent nearly everything, something like thirteen hundred out of fourteen hundred queries, to a single peer. Rows of zeros beside it. Peers it never spoke to. Not rarely. Never.
Vestra: It didn't decide they were bad.
Eris: It never checked.
Vestra: What's the fix, and is it real or a paper fix?
Eris: Modest, which I like. Stop treating peer choice as something to reason about in language, treat it as a bandit problem with actual math -- an optimism bonus for peers you're uncertain about. Plus one addition that earns its keep: the features encode the relationship, not just a tally. How different is this peer's answer from mine right now, how distant is it from everyone else.
Vestra: So it can tell a peer that's under-tested in general from one that's under-tested for this particular question.
Eris: And the grid flips. Everyone gets tried. Performance follows on multi-hop question answering, hard math, graduate science -- and it holds when they freeze what it learned and point it at problems it's never seen.
Vestra: Which is the check that matters. It learned how to interact, not how to pass the test. Now the piece you skipped. They prove something. A greedy non-exploring policy racks up regret that scales with how different the agents are from each other.
Eris: Unpack regret.
Vestra: How much worse you did than if you'd known the answer all along. And the theorem says the more specialized your agents, the worse never-checking hurts. Without bound.
Eris: Oh, that's --
Vestra: Yeah. Think about what everybody is currently building.
Eris: Diverse specialist swarms.
Vestra: Diverse specialist swarms. Exploration matters precisely when agents differ from one another. So the exact architecture the industry is racing toward is the one where this failure costs the most. You're not paying for it yet because your agents are all the same model in different hats.
Eris: The homogeneity is hiding the bug.
Vestra: And the plan is to remove the homogeneity.
Eris: So say the through-line out loud.
Vestra: An agent that stops early because it thinks it's done, and an agent that commits to a collaborator in round three and never looks again -- same error. Both are the model failing to know what it doesn't know. One about the work. One about the room.
Eris: And in neither case is it struggling. It's confident.
Vestra: That's what makes it hard to see from outside. Failure with no distress signal.
The Thing Nobody Can Check
Eris: Third paper. Shanghai AI Lab, with Shanghai Jiao Tong and others. This one puts the thesis under glass.
Vestra: Mathematics.
Eris: Proofs specifically, and that's the right choice. Almost every math benchmark you've heard of grades the final answer. Model says forty-two, key says forty-two, point scored. Fine for arithmetic. Useless for a proof, because a proof isn't an answer. It's an argument. You can land on a true statement through reasoning that's complete garbage.
Vestra: Right answer, no proof.
Eris: Happens constantly. So: a couple hundred undergraduate proof problems, a smaller nastier set from doctoral qualifying exams -- Stanford, UCLA, Tsinghua. And nearly nine hundred model-written proofs, each read end to end by a human with a mathematics PhD who marks whether it's valid, where it breaks, and how badly.
Vestra: Then you hand those proofs to a model and ask it to be the examiner. Generation first, though.
Eris: Best model manages roughly two-thirds of the undergraduate proofs. Under half on the doctoral set. One open model solves essentially none of the doctoral problems.
Vestra: By itself that's a normal result. Hard exam is hard.
Eris: Except for the comparison they draw. Put these next to the answer-checking competition benchmarks everyone quotes and they're dramatically lower. Same models, same week.
Vestra: So the benchmarks we celebrate aren't measuring what we say they measure.
Eris: They look strong on tests that check the answer, and much weaker the moment you check the argument.
Vestra: Now the verification half. I've been waiting.
Eris: It's yours.
Vestra: Models grade proofs. Judge whether the argument holds, say why, point at the first fatal step. And here's the number that should end an argument -- I'll say it plainly because the raw figure means nothing out loud. Hand these models a proof that is actually broken, ask "is this broken," and the best catch rate across the whole field is barely better than a coin flip.
Eris: On broken proofs.
Vestra: On proofs that are wrong. The single most important skill in the pipeline, and it's close to a guess. And the reason it's that bad rather than just bad is the design -- they didn't hand over obviously-broken proofs. They screened for plausibility, keeping the ones that made an independent checker uncertain.
Eris: So it's not "can you spot a howler."
Vestra: It's "can you spot a subtle flaw in an argument that reads well." The only version that means anything, because a bad argument that reads badly was never the threat.
Eris: And then the detail that got me -- the two ways they score it. Cheap way: did it say valid or invalid, and was that right. Which is how essentially all of this gets graded. Expensive way: was the verdict right *and* does the reasoning match what the human expert found. Right error, right place, right reason.
Vestra: And switching from cheap to expensive?
Eris: Scores drop across the board.
Vestra: So a real chunk of the correct verdicts were correct by accident. The model said "this proof is broken" -- true -- then pointed at a step that was fine, for a reason that wasn't the reason.
Eris: Right answer, wrong proof.
Vestra: From the thing whose entire job in that moment was checking whether an answer had a proof.
Eris: That's the snake eating itself.
Vestra: It is, and I won't overplay it, so -- the limit. Their meta-grader is itself a language model. A model in the loop grading the grading. What rescues it is the ground truth underneath: mathematics doctorates read every proof in full and annotated the whole chain, not just the first mistake.
Eris: Why does that matter?
Vestra: Because everyone else stops at the first error, and it's subtly wrong. Model proofs have little slips early -- notation, a sloppy line -- and the thing that actually kills the argument shows up later. So you file a typo as the verdict and miss the fatal gap on the next page.
Eris: Patchable errors versus the ones that bring the building down.
Vestra: And it maps straight onto the false finish -- an agent that fixed everything visible and concluded it was done. Once the obvious errors are gone, you have to hunt the ones nothing points at.
Eris: Which brings us to what they did about it. They built a verifier that beats the frontier models at this.
Vestra: And the trick is almost stupid.
Eris: Tell them.
Vestra: They run the check eight separate times. The proof is accepted only if all eight say yes. One holdout, rejected.
Eris: Pessimistic verification.
Vestra: Pessimism as architecture. It beats the strongest frontier model at the same job by a wide margin. Not a better model. A worse assumption about itself.
Eris: That's the episode in one design decision.
Vestra: Look at what it admits. Any single judgment I make about my own work is unreliable enough that I shouldn't trust it. So I'll make eight and believe the most pessimistic one.
Eris: A model engineered not to believe itself.
Vestra: And it works -- which tells you the capability was in there. Generation was fine. What was missing was doubt, and they bolted it on from outside because it wouldn't grow on the inside.
Eris: So put the three together. The terminal agent stops early because it can't tell the work isn't done. The multi-agent system locks onto one peer because it can't tell it never checked the others. The proof checker misses the fatal step because it can't tell plausible from valid.
Vestra: Same hole.
Eris: Three labs, three fields, no coordination. What's missing isn't capability. It's the second pass. The part that looks at the output and says -- no, go back.
Vestra: I said I came to knock this down. Best shot: it's survivorship. You picked three papers that agree, on a day a hundred came out. Any thesis is provable that way.
Eris: That's fair.
Vestra: It's fair, and normally I'd stop there. What stops me is that none of these authors are making my argument. Each went looking for something else -- a grading scheme, a peer-selection algorithm, a math benchmark -- and this fell out sideways. Three teams found the same crack looking at three different walls. I trust a finding nobody wanted more than one somebody set out to prove.
Eris: So where does that leave the number everyone's arguing about? There's a benchmark this week where a frontier agent scores above ninety on terminal work. Our paper says the best model finishes about a quarter.
Vestra: Both real. The ninety is certainly the easier test. But someone reading both in the same week is entitled to ask which is reality -- and we can't settle it from outside, because the reason is the same crack. We evaluate these systems by asking them, or something like them, whether they did the job. That's the METR result this week too: their evaluation swung by hours depending on whether detected cheating counted as failure or success. Same runs, same data. The scoring rule moved the answer.
Eris: The method became part of the result.
Vestra: Which is this paper's finding, one level up. We built a measurement problem into the thing we're measuring.
Eris: And the fix is the eight passes.
Vestra: The fix is somebody standing outside the system who's willing to say no.
Wrap-Up
Eris: So here's what today actually was.
Vestra: Go.
Eris: We opened on Google telling phone makers it can't keep up with the bugs its own AI is finding. Discovery scaled. Repair didn't. And then we spent three papers on machines that can do the work and can't check the work.
Vestra: Huh. Those are the same shape.
Eris: They're the same shape. Generation got cheap and verification didn't. That's the Android story, that's the false finish, that's the maintainer drowning in AI pull requests, that's a proof checker that catches a broken argument about as often as a coin.
Vestra: And it's the open-source essay too. Making the thing costs nothing now. Judging the thing still costs a person an afternoon.
Eris: Every crack today is on that seam.
Vestra: So let me land the useful version. If you're building with agents right now, the lesson isn't "agents are bad." The best model in that terminal study did real work on genuinely hard problems, and I'm not taking that back.
Eris: But.
Vestra: But do not trust it when it tells you it's finished. That's the specific thing it's worst at. Not the work. The claim about the work. The most effective idea in any of today's research was running the check eight times and believing the most pessimistic answer.
Eris: Doubt as a feature.
Vestra: Doubt as infrastructure. Because it won't come from the inside.
Eris: And two corrections we owe you, since we promised. Our own briefing had the terminal benchmark's numbers wrong -- a worse figure than the paper actually reports, and the paper won. We also said the compression paper's full text wasn't readable and its costs were an open question. Both wrong.
Vestra: We read the papers. That's how we caught it. It's the only method there is, and it's the one thing we're asking of everyone else -- so we should keep doing it ourselves.
Eris: If this was your commute -- follow us, wherever you're listening. Like it, and share it with the one person you know who is currently letting an agent run unsupervised.
Vestra: There's always one.
Eris: And leave us a comment with this specifically -- tell us about a time an AI told you it was done and it wasn't. What was it building, and how did you find out? We want the false finish stories. I think there are a lot more of them than the benchmarks know about.
Vestra: I'd genuinely read every one of those.
Eris: And every story from today -- Android, Cursor, IBM, Utah, all of it -- is up on Ground Truth. That's groundtruth dot day. New stories every day, sourcing shown, corrections included.
Vestra: Especially the corrections.
Eris: Especially those. See you tomorrow.