News · 2026-10-17

Mercor says its agent-benchmark map leaves most American work hours uncovered

Mercor estimates that about 90% of American working hours fall on tasks with little or no coverage in its selected index of public agent benchmarks. Its new benchmark map measures where evaluation questions exist, rather than what artificial intelligence can automate. The result challenges broad economic readings of leaderboard progress while leaving substantial uncertainty around the exact percentage.

Key facts

• Mercor's initial index contains 29 benchmarks and marks tasks covered at five or more mapped prompts.
• The map covers 924 of 18,797 listed occupational tasks at that threshold.
• The work circulated on Hacker News October 16; the research page does not establish its publication date.
• The primary record is Mercor's benchmark map and methodology.

The missing distinction is between a map of exams and a map of ability. If a school has many mathematics tests and few music tests, its test catalog cannot establish that students are incapable of music. Conversely, a music test existing does not show that its students perform well. Mercor's project applies that distinction to the expanding collection of evaluations for software agents.

Mercor calls the project “A benchmark map of the US economy.” It maps benchmark prompts to tasks in O*NET, a standardized catalog of occupations and their activities. Its selection includes software, office, professional and computer-use tests. It also includes Mercor's own benchmark family. The list is a curated starting set, with restrictions intended to limit overrepresented job families and exclude tests considered saturated.

The mapping method uses a language model to generate and select plausible occupation-task associations for each prompt. One prompt may connect to several tasks. A task becomes covered in the headline accounting only after at least five prompts map to it. Four prompts and five prompts therefore fall on opposite sides of a hard boundary, even if their practical measurement quality is similar.

Human checking was narrower than an unrestricted claim of expert validation would suggest. Mercor reports expert review of 1,013 prompts from three of its own benchmark collections. Reviewers accepted the model's first-choice mapping in about 77% of judgments. The disclosed methodology does not establish the same human audit across the rest of the index. That limitation concerns how confidently prompts are assigned, separately from whether their underlying benchmarks are useful.

Turning task counts into working hours requires another layer. Mercor relies on task-time estimates from Hatgis-Kessell and colleagues' Estimating time spent on work tasks. The researchers combine worker-reported task frequency, labor-survey information where available, and model judgments comparing task durations. Their full paper explains the ranking and optimization steps. These are modeled estimates of task shares, rather than direct observation of every occupation's workday.

There is also an unresolved reproducibility question. Mercor's task catalog and the linked time-share paper use different O*NET versions and occupation coverage. The pages reviewed in the dossier do not disclose the full crosswalk, final national aggregation or treatment of occupations without complete time estimates. That gap does not establish that the result is wrong. It identifies information another researcher would need to reproduce the exact headline.

Mercor acknowledges that its incomplete benchmark list biases estimated coverage downward and that the size of the bias is difficult to bound. Excluding easy, saturated tests can also make measured coverage look narrower. Meanwhile, even five mapped prompts do not guarantee realistic stakeholder behavior, robust scoring or meaningful economic outcomes. Coverage and evaluation quality are different dimensions, as Ground Truth's benchmarking lesson explains.

The Hacker News discussion focused on those boundaries. Commenters challenged the fit of occupational taxonomies to modern specialist work, the use of hours rather than economic value, and the leap from missing tests to missing capability. Those objections are meaningful counterarguments to expansive interpretations, rather than proof that the map has no value.

Mercor's incentives also belong in the analysis. The company sells expert data and evaluation services, and its own research materials promote those offerings. Wider gaps in evaluation are commercially relevant to that business. The appropriate response is to demand disclosed assumptions and independent replication, rather than treat the conflict as either irrelevant or automatically disqualifying.

The actionable finding survives those caveats: current leaderboards sample a concentrated set of measurable activities. Organizations deciding whether agents can deliver useful work need representative tasks from their own operations. A high public score cannot establish coverage of an occupation, and an uncovered occupation cannot establish immunity from automation.

Primary source, verified: read the paper →

Ground Truth. (2026, October 17). Mercor says its agent-benchmark map leaves most American work hours uncovered. Ground Truth. https://groundtruth.day/news/mercor-map-finds-agent-benchmark-coverage-gap.html

@misc{groundtruth:mercor-map-finds-agent-benchmark-coverage-gap,
  title  = {Mercor says its agent-benchmark map leaves most American work hours uncovered},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/news/mercor-map-finds-agent-benchmark-coverage-gap.html}
}

Topics: evaluation · benchmarks · labor · agents