Glonce

Mercor's Benchmark Map: about 90% of US work hours lack AI benchmark coverage

researchThe Glonce editorsHacker News🔥80Glonce rating80Impact68Novelty62Relevance95Surprise60Players40Credibility70

Mercor has published a Benchmark Map, an interactive chart that measures how much of American work existing AI benchmarks actually touch. Each tile is one of the 923 occupations in the US Department of Labor's ONET taxonomy, grouped into 22 job families and shaded by the share of that occupation's tasks represented in Mercor's initial list of 29 benchmarks. ONET lists 18,797 tasks across all occupations, and one benchmark prompt can map to several of them.

The headline finding: about 90% of American working hours are spent on tasks with little or no coverage from existing benchmarks. Only 208 of the 923 occupations have at least one task with at least 5 benchmark prompts, and 6 of the 22 job families have no such occupation. Mercor says this makes it very hard to know how good AI models are at the work Americans actually do.

Method. Following work by Wang et al., an LLM judge tags each benchmark prompt with as many ONET tasks as it thinks fit. A task counts as covered on the map if the judge has mapped it to at least 5 prompts from the 29 benchmarks. Mercor checked the LLM tagging against human expert tags on its own APEX benchmarks, which are a subset of the 29. To turn task coverage into a share of working hours, it combined the coverage estimates with Hatgis-Kessell et al.'s LLM-derived estimates of the working time share of each ONET task. The 29 benchmarks were chosen for quality and economic relevance; the full list and criteria sit in a separate Benchmark Index tab, and a Methodology tab has the detail.

Where coverage falls. It is not random. Benchmarks concentrate on tasks most exposed to AI automation, as measured by Eloundou et al.: tasks rated "exposed" are about 6x more likely to appear on benchmarks than tasks rated "unexposed". That is why benchmarks are full of tasks in software engineering and financial analysis but sparse on truck driving and gas plant operation. Even after controlling for AI exposure, tasks from occupations with larger wage bills are significantly overrepresented among benchmark prompts. After controlling for occupation and exposure, the tasks that O*NET experts score most important (on a 1 to 5 scale) are about 6 to 10 times more likely to be on benchmarks than those scored least important. One example: the benchmarks heavily cover AI-exposed lawyers' tasks but barely cover any tasks of similarly exposed legal secretaries or court clerks. Mercor reads this as the AI community focusing on the largest labor markets first, leaving the long tail of smaller, middle-income knowledge work jobs undercovered. It calls this a reasonable use of limited resources, but expects significant blind spots unless benchmarks grow in number and diversity.

From coverage to occupation scores. Mercor argues that scoring models by occupation matters for enterprises choosing models, policymakers forecasting AI's economic impact, and people thinking about careers. A simple average over an occupation's benchmark prompts would overweight frequently benchmarked tasks, so it tried weighting so each ONET task counts equally. As a first test it used APEX for four occupations: investment banking, law, accounting and software engineering. Weighted and unweighted scores came out generally quite similar, so weighting changed very little there. But less than a third of the ONET tasks relevant to those four occupations have enough APEX coverage to be included at all. Mercor says it is hopeful that weighting over a much larger set of benchmarks will give high-quality occupation-specific scores.

Caveats from the authors. The results are provisional because the 29 benchmarks are not a complete or perfectly representative sample, and any incomplete list biases coverage estimates down by an amount that is hard to bound. Capping benchmarks per job family, done to maximize breadth, biases down coverage in the most-benchmarked families such as software engineering. Choosing unsaturated benchmarks risks understating coverage on tasks that are easier for LLMs; including saturated ones would risk overstating it, and Mercor thinks that bias would be larger, though neither option is ideal. Some selection criteria, like economic realism, are subjective. Further obstacles: very few benchmark items per task, so one or two low-quality items can badly skew a score; no explicit way yet to weight by item quality; real-world AI use is mostly human-assisted, and measuring how much a model augments people has only early work and no scalable method; and benchmarks suit digital knowledge work best. Citing Dingel and Neiman, Mercor notes only 37% of US jobs could feasibly be done entirely from home, so about 63% involve some physical-world component that is hard to benchmark today. With robotics advancing fast, it expects a significant need for physical task benchmarks.

Key facts

  • Mercor's Benchmark Map estimates that about 90% of American working hours are spent on tasks with little or no coverage from existing AI benchmarks; the authors call this provisional.
  • The map covers 923 O*NET occupations in 22 job families, scored against an initial list of 29 benchmarks; only 208 occupations have a task with at least 5 benchmark prompts, and 6 job families have none.
  • Coverage follows AI exposure and money: tasks rated exposed are about 6x more likely to be benchmarked than unexposed ones, and high-wage-bill occupations are overrepresented even after controlling for exposure.
  • In a test on Mercor's APEX benchmarks for investment banking, law, accounting and software engineering, O*NET weighting changed scores very little, but less than a third of relevant tasks had enough coverage to be included.
  • The authors say an incomplete benchmark list biases coverage estimates down by an amount that is hard to bound, and that physical work is largely outside what benchmarks measure.

Why it matters

Benchmarks are how the field judges AI progress, yet Mercor's map suggests they describe only a thin slice of paid work. Its estimate is that about 90% of American working hours fall on tasks with little or no benchmark coverage, which makes it hard to say how good models are at what people actually do. The pattern is also skewed: software engineering and financial analysis are crowded, while truck driving, gas plant operation and the work of legal secretaries or court clerks are sparse. Mercor argues that occupation-level capability scores matter for enterprises choosing models, policymakers forecasting AI's economic effects, and individuals planning careers.

Who it affects

Benchmark builders get a guide to where the gaps are, especially the long tail of smaller, middle-income knowledge work jobs. Enterprises picking models, policymakers and workers looking for evidence about AI in their occupation are the readers Mercor names. People in lightly covered jobs, such as legal secretaries, court clerks, truck drivers and gas plant operators, are the ones for whom current benchmark results say least. Mercor also invites others working on these problems to get in touch.

How to use it

The Benchmark Map is an interactive page: each tile is an occupation, shaded by benchmark coverage, and clicking a job family opens it. A Benchmark Index tab lists the 29 benchmarks and the selection criteria, and a Methodology tab gives the detailed account. Practically, the map is a way to check how much benchmark evidence exists for a given occupation before relying on headline model scores for it, and to see which areas a new benchmark would fill. Mercor describes the resource as continuously updated and says it will update results as the benchmark list expands.

How solid is it

Treat the 90% as a first-pass estimate, not a measurement. It rests on a hand-picked list of 29 benchmarks that Mercor itself says is not complete or perfectly representative; it also depends on an LLM judge tagging prompts to tasks and on LLM-derived estimates of time spent per task. The tagging was validated against human expert tags only on Mercor's own APEX benchmarks, a subset of the list. Mercor states plainly that an incomplete list biases coverage down and that the bias is hard to bound, and that its capping of benchmarks per job family further lowers estimates for heavily benchmarked areas. The sizeable gap between 208 covered occupations and 923 total is a count of what falls in this particular dataset.

Risks and caveats

Low coverage is not the same as poor AI performance: the map shows where measurement is missing, and Mercor says only that it is hard to know how good models are on those tasks. Benchmark choice cuts both ways: excluding saturated benchmarks may understate coverage on tasks LLMs find easy, while including them may overstate it. Some selection criteria are subjective. In the APEX test, weighting by O*NET task changed little, and the authors admit that the test used fewer than a third of the relevant tasks. Scoring by occupation faces further obstacles: few items per task, no way yet to weight for item quality, no scalable method for measuring how much a model augments humans, and little room for physical work, since about 63% of US jobs involve some physical-world component according to Dingel and Neiman's estimate.

“This makes it very hard to know how good AI models are at the work Americans actually do.”

— Mercor, Benchmark Map