News · 2026-10-17

A customer-delivery benchmark stopped the best agent at 14 completed tasks out of 95

FDE-Gym's strongest tested agent configuration resolved 14 of 95 simulated customer-delivery tasks in its realistic setting, according to a new research paper. The benchmark measures business outcomes after agents gather requirements, modify systems and interact with stakeholders. Its results show a gap between producing a plausible deliverable and achieving the customer's intended improvement within the tested environments.

Key facts

• Dongxu Yang and colleagues submitted the paper October 15, 2026.
• The benchmark contains 95 tasks adapted from 20 scenarios across 18 sectors.
• The best configuration averages 0.56 and resolves 14 tasks under the paper's scoring rule.
• The primary source is FDE-Gym: The Last Mile of AI Delivery, with full methods and results.

A customer rarely hands a contractor a perfect specification. A request to improve a workflow may omit constraints, hide conflicting needs or describe a proposed solution rather than the actual business problem. Much of delivery consists of discovering those differences, choosing an appropriate intervention and checking whether it helps after implementation.

FDE-Gym tries to represent that work. Industry experts adapted delivery projects into production-like systems with stakeholders an agent can question. The environments include held-out data and customer-specific measures of success. The agent must gather facts, work out what matters and alter the system, rather than simply answer a question whose entire specification is visible at the start.

The paper's title, “The Last Mile of AI Delivery,” captures the focus. An agent can produce code that runs and still fail to deliver value. A familiar analogy is a restaurant ordering system: adding a requested dashboard does not help if the real delay is a broken handoff between kitchen and pickup. Understanding the request, identifying the bottleneck and making the change are separate achievements.

The benchmark compares business uplift against an unchanged system and an expert reference solution. It also penalizes stakeholder interactions, including messages, trial runs and waiting. That structure gives information gathering a cost. Asking every possible question or testing indefinitely is not automatically an effective strategy, just as repeatedly interrupting a customer consumes real delivery time.

A task counts as resolved only when its score reaches at least 0.8 without an incident. The best configuration's mean score of 0.56 therefore does not mean 56% of customers received a complete solution. Average score and count of resolved tasks describe different things. Fourteen of 95 resolutions is the clearer anchor for the strict completion outcome.

The authors compare 13 model configurations from seven providers. In the realistic setting, five configurations resolve no tasks. Supplying all relevant constraints improves leading models' scores substantially. This establishes an information-discovery burden within the benchmark, while leaving room for a second failure after discovery: implementing the right solution once the facts are available.

That second gap is especially revealing. On the 23 tasks where the best configuration obtains every key stakeholder fact, it reaches at least 0.8 normalized business uplift on only 10. Gathering the right information does not automatically produce sound operational judgment. The agent may still misunderstand the system, choose a poor intervention or implement it incorrectly.

The findings pair naturally with Mercor's benchmark coverage map. Mercor asks which kinds of work appear in a selected collection of tests. FDE-Gym asks what happens when a benchmark includes customer discovery and business outcomes. The studies do not validate each other's numerical conclusions, and FDE-Gym cannot establish a national share of work that agents can or cannot perform.

The project site and code repository provide the benchmark's accompanying materials. This is evaluation infrastructure, rather than a generally useful consumer tool or a newly released model checkpoint. There is no new downloadable model footprint to report for the benchmark itself.

The strongest caveat is realism. Stakeholder responses are scripted, and scenarios are adapted, de-identified or fictionalized. Real customers change their minds, disagree internally and supply incomplete or contradictory accounts in ways that are hard to reproduce. The paper acknowledges that simulated stakeholders can differ from people. DeepLethe's role developing and operating the platform is also a disclosed competing interest.

Nor does an author-run benchmark establish an outside replication. The dossier verifies the paper's full text and reported outcomes, but no independent assessment demonstrating the same results. Ground Truth's benchmarking lesson explains why test construction and scoring choices shape what a result can support.

The useful conclusion is specific: in these simulated delivery environments, strong agents often fail after the problem extends beyond a well-specified ticket. Evaluation should track business outcomes, discovery costs and incidents alongside whether code runs. The last mile is measurable, and it remains a substantial source of failure in this test.

Primary source, verified: read the paper → (arXiv 2610.22350)

Ground Truth. (2026, October 17). A customer-delivery benchmark stopped the best agent at 14 completed tasks out of 95. Ground Truth. https://groundtruth.day/news/fde-gym-customer-delivery-benchmark.html

@misc{groundtruth:fde-gym-customer-delivery-benchmark,
  title  = {A customer-delivery benchmark stopped the best agent at 14 completed tasks out of 95},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/news/fde-gym-customer-delivery-benchmark.html}
}

Topics: research · evaluation · agents · enterprise