News · 2026-08-25

A new benchmark of 1,140 real agent failures finds the best method identifies the decisive wrong step 13 percent of the time

A new benchmark shows that when an AI agent fails a long task, no existing method can reliably say where it went wrong. LongRCA Bench collects 1,140 genuinely failed agent runs, with a median of 145 steps each, and human labels marking the earliest decisive mistake in every one. The strongest baseline identifies that step correctly 13.2% of the time.

Key facts

• 1,140 failed trajectories across five domains, with no injected errors - every failure is one the agent actually produced.
• Median trajectory length: 145 steps. Strongest baseline exact root-step accuracy: 13.2%.
• The authors' own method, RCTA, reaches 51.1% responsible-role accuracy but only 24.1% exact root-step accuracy on the same backbone and scoring protocol.
• Primary source: LongRCA Bench (arXiv:2608.15242), published August 15, 2026 by Yunfei Zhang, Boyu Feng, Changhua Pei, Fei Sun, Yintong Huo and colleagues. Dataset: CLoud5-real/longrca-bench.

The gap the paper names is one every agent developer has hit. In the authors' words: "When a long-horizon agent execution fails, outcome-level evaluation reveals the unsuccessful result but not where the decisive error entered the trajectory. Developers must then inspect the full execution to identify the responsible role and localize the earliest decisive root-cause step."

That inspection is brutal. A benchmark score tells you the run failed. It does not tell you that at step 62 the agent misread a tool's output, at step 71 it built a plan on that misreading, and everything after was a competent execution of a wrong plan. Finding step 62 means reading all 145 steps. Multiply by a few hundred failures and it is a full-time job nobody has.

Two design choices make this benchmark useful rather than merely another leaderboard.

The first is that the failures are real. Most prior work in this area injects errors - a researcher deliberately corrupts a step and asks whether a method can find it. That produces clean, findable mistakes. Real agent failures are messier: an agent does not usually make one obviously wrong move, it makes a slightly optimistic assumption that becomes wrong three steps later when the environment turns out different than expected. LongRCA Bench uses failures the agents produced on their own, which is why the numbers are so much worse than injected-error benchmarks report.

The second is length. Existing failure-attribution benchmarks, the authors note, "largely focus on shorter traces, leaving diagnosis across hundreds of recorded steps underexplored." At a median of 145 steps, this is the regime where real agent work happens and where human inspection stops being feasible.

The results split into two findings, and the split is the paper's actual argument.

Their own method, RCTA, works by retrieving candidate error steps from summaries of trajectory segments and then tracing those candidates back to earlier handoff instructions - the moments where one component passed work to another. Running on the same backbone, the same instances, and the same scoring as the baselines, it reaches 51.1% accuracy on naming the responsible role and 24.1% on pinpointing the exact root step.

Those two numbers diverging by more than a factor of two is the point. Working out who broke the run and working out where it broke are different problems with different difficulty. Blaming a component is a coarse judgment with a handful of candidates. Picking one step out of 145 is a needle in a haystack where the needle also looks like hay - the decisive step usually looked completely reasonable at the time. The conclusion the authors draw is that these should be scored as separate targets, not folded into one attribution metric.

The analogy is aviation incident investigation. Determining that the failure originated in the maintenance process is one level of finding. Determining that it was a specific torque check skipped on a specific date is another, and it is the one that changes anything. The field has been reporting the first and implying it has the second.

The honest caveat runs through the labels. "Earliest decisive root-cause step" is a human judgment applied to a 145-step trace, and the whole benchmark rests on how consistently humans can make it. The paper says labels are independently scored, which is the right procedure, but a 13% ceiling could partly reflect genuine ambiguity about where a slowly compounding failure began, rather than pure model incapacity. If two careful annotators disagree about whether the run broke at step 62 or step 71, a method that answers 71 is not obviously wrong.

The practical value is as a diagnostic complement to harness work. Microsoft's AutoSaddler patches the scaffolding using failure traces; LongRCA Bench measures how well anyone can read those traces in the first place. If root-cause localization sits at 13-24%, then the diagnosis half of every automated agent-improvement loop is running on mostly wrong inputs - which is a strong argument that agent reliability work should be measuring its own diagnostic step, not just its final score. Related: why agent training collapses and multi-agent systems.

Primary source, verified: read the paper → (arXiv 2608.15242)

Ground Truth. (2026, August 25). A new benchmark of 1,140 real agent failures finds the best method identifies the decisive wrong step 13 percent of the time. Ground Truth. https://groundtruth.day/news/when-an-agent-fails-nobody-can-find-the-step-that-broke-it.html

@misc{groundtruth:when-an-agent-fails-nobody-can-find-the-step-that-broke-it,
  title  = {A new benchmark of 1,140 real agent failures finds the best method identifies the decisive wrong step 13 percent of the time},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/when-an-agent-fails-nobody-can-find-the-step-that-broke-it.html}
}

Topics: agents · benchmarks · debugging · evaluation · failure-analysis · multi-agent