OR-License benchmark tests LLM agents on 31 real operations research jobs
A paper on Hugging Face asks whether LLM agents can deliver expert-level operations research (OR) solutions straight from original client materials, without an expert first rewriting the problem. The authors argue that existing OR benchmarks usually start from self-contained descriptions or predefined optimization tasks, so they underrepresent the step of turning raw business material into a professionally validated solution.
To test that step, they introduce OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios. The instances come from paid OR engagements totaling USD 3.68 million. After de-identification, the benchmark keeps the original requirements, input data, deliverable formats and information boundaries.
To study how agents gather information, the paper adds ORSim, a controlled client simulator. It discloses business facts but does not supply OR expertise. Evaluation is hidden: it checks whether a delivered solution is feasible and compares its objective quality with the original expert solutions, which clients had accepted before the benchmark was built. The headline metric, Deployment-Adjusted Quality (DAQ), averages expert-relative quality across instances and assigns zero to any infeasible delivery. The expert reference scores 1.
The team ran 11 frontier LLMs in coding-agent harnesses. Several exceed the expert reference in conditional objective quality, but none matches its DAQ. The strongest primary Claude Code configuration reaches 88.39% strict feasibility and a DAQ of 0.87 against the expert's 1.
Two further findings concern process. Client information acquisition goes beyond clarification into solution development and validation, and some later stages show higher confirmed issue-resolution proportions. Trajectory analysis identifies formulation and implementation failures, and task-equalized research-chain completion rates range from 24% to 45% across models on evaluable tasks. The authors conclude that these results motivate sustained information-gap resolution, faithful problem formulation, and evidence-linked self-improvement for reliable long-horizon agents.
Key facts
- OR-License is a benchmark of 31 instances across 10 industrial scenarios, drawn from paid operations research engagements totaling USD 3.68 million.
- ORSim, a client simulator, discloses business facts without supplying OR expertise, so agents must pull information out of a simulated client.
- Across 11 frontier LLMs in coding-agent harnesses, several exceed the expert reference in conditional objective quality, but none matches its DAQ.
- The strongest primary Claude Code configuration reaches 88.39% strict feasibility and a DAQ of 0.87, against the expert reference of 1.
- Task-equalized research-chain completion rates range from 24% to 45% across models on evaluable tasks.
Why it matters
Most OR benchmarks hand a model a clean, predefined optimization problem. OR-License starts from the messy original client materials of real, paid engagements, which the authors say existing benchmarks underrepresent. That makes the gap it reports more telling: some models match or beat the expert on objective quality when they produce a feasible answer, yet none reaches the expert's DAQ of 1, because DAQ gives zero to infeasible deliveries. The best Claude Code configuration is at 0.87.
Who it affects
The work is aimed at people building and judging LLM agents for long-horizon, professional tasks, and at anyone weighing whether such agents could take on operations research work now done by experts. The authors frame their conclusions around reliable long-horizon agents in general, not just OR.
How to use it
As described, the benchmark gives an agent original client materials and lets it ask the ORSim client simulator for business facts. Its solution is then scored by hidden checks for feasibility and objective quality against the accepted expert solution. No release date, code or dataset availability is stated in the text, so whether others can run it is not yet clear.
How solid is it
This is a preprint-style abstract on Hugging Face, so the numbers are the authors' own and have not been independently checked here. The design has real strengths: instances come from paid engagements, and the reference solutions were accepted by clients before the benchmark was built. Limits are visible too: 31 instances is a small set. The abstract names no authors or institutions, does not name the other 10 LLMs, does not say which Claude model sits behind the Claude Code configuration, and does not say which models beat the expert reference in conditional objective quality or by how much.
Risks and caveats
The headline 0.87 is one configuration's DAQ, and DAQ values for the other models are not given, so the spread across the 11 models is unknown. "Exceeding the expert" applies only to conditional objective quality, not to overall quality. The 24% to 45% completion rates cover only evaluable tasks. The 10 industrial scenarios and the clients are not identified in the abstract, and the instances were de-identified, which may affect how far results carry over to other settings.
“These findings motivate sustained information-gap resolution, faithful problem formulation, and evidence-linked self-improvement for reliable long-horizon agents.”
— OR-License paper abstract