News · 2026-11-17
Runta says most priced inference spending in its coding-agent test went to non-passing runs
Runta reports that 64.5% of priced inference spending in its coding-agent benchmark went to runs that did not pass. The finding comes from a specific collection of terminal tasks and model configurations, so it does not measure a typical software team’s production waste.
Key facts
- FrontierHarness Eval v1.1 covers 27,280 trials across 22 harness configurations, five models, 62 tasks and four attempts.
- Runta attributes 52.7 percentage points of priced cost to failures and 11.8 to timeouts.
- Timeouts were graded and could pass or fail; infrastructure errors were rerun.
- The Runta report is the primary source for its methods and cost accounting.
A coding agent combines a model with a surrounding harness: instructions, tools, a runtime, and a way to decide when to stop. A capable model may perform differently under different harnesses, and a long run can consume substantial tokens even if its final work does not meet the task’s tests. Runta compares these combinations on terminal tasks with time limits ranging from 15 minutes to eight hours.
The cost figure uses Baseten list prices and provider-reported token counts. Runta says 208 trials had no price, and the report does not supply enough row-level outcome and pricing data to independently recalculate the overall breakdown. Its 64.5% figure is therefore the authors’ accounting for priced runs, not an audited universal rate. A failure means the grader did not award a pass; it does not necessarily mean the agent crashed or produced nothing useful.
The authors say that among their five best-value model and harness pairings, 68% to 82% of failed runs ended with a final answer instead of reaching the time limit. Their trace analysis characterizes these as confident answers, but the study does not establish a calibrated measure of confidence. The distinction matters: a polished final message can be wrong, but its mere presence does not reveal whether the model was certain.
Consider a contractor who completes a long repair and submits an invoice. The invoice describes work done; it does not prove the appliance now works. A check controlled independently of the person doing the repair provides a stronger basis for deciding whether the job is finished. In software, that might be an owner-maintained test, a known historical failure, or a comparison against a trusted implementation. An agent can run the check, but a test that only repeats the agent’s mistaken assumptions may miss the same defect.
A small Stet study shows another problem with single-number rankings: a low-cost system can appear best on cost per success by solving easy tasks while failing harder ones. That study covers one repository and one attempt per configuration, so it illustrates metric sensitivity rather than establishing a general leaderboard. The existing benchmarking lesson explains why workload composition changes the conclusion.
The four attempts in the study are repeated experimental trials, not necessarily four retries by a user in a real workplace. Each trial started in a new runtime, and the models were served by one provider. Those choices make the comparison easier to describe but exclude important aspects of real development, including a developer changing the instructions halfway through, reviewing a partial patch, or using different prices and cache arrangements. Runta says model choice explained most of the variation in its analysis; with a model fixed, the harness still changed cost materially. Neither result tells a team which setup will win on its own backlog without a representative test.
Runta suggests additional checks before an agent terminates might help, but did not test the savings from that intervention. In Hacker News discussion, readers questioned whether binary scoring captures useful partial output and how much human steering changes results. Those are reasonable limits: production systems have different tasks, prices, review practices and retry rules. The benchmark’s strongest contribution is to make cost and pass criteria visible together, while leaving the transfer to any particular team to be measured.
Runta calls some of the final responses “confident,” but its account treats that as a reading of traces rather than a calibrated probability. That qualification matters when comparing what the agent says to the verifier’s separate result.
Key questions
Does 64.5% mean that most production AI coding spending is wasted?
Were all timed-out trials automatically failures?
What counts as a pass?
Cite this
APA
Ground Truth. (2026, November 17). Runta says most priced inference spending in its coding-agent test went to non-passing runs. Ground Truth. https://groundtruth.day/news/runta-coding-agent-inference-cost-failed-trials.html
BibTeX
@misc{groundtruth:runta-coding-agent-inference-cost-failed-trials,
title = {Runta says most priced inference spending in its coding-agent test went to non-passing runs},
author = {{Ground Truth}},
year = {2026},
month = {nov},
url = {https://groundtruth.day/news/runta-coding-agent-inference-cost-failed-trials.html}
}
Comments are replies to this story on Bluesky — reply with any Bluesky account to join in.