Nvidia shows AI agent harness matters more than the model
Nvidia published research on Friday arguing that for long-horizon agentic tasks, the harness around an AI model matters more than the model itself. A harness is the software wrapper around a model: the tools it can call, the way it manages memory, and the rules that turn a raw model into something that can act on its own over many steps.
The headline result: using a custom harness built to handle memory well and including a "supervisor" component, Nvidia researchers got Claude Opus 5 to a 100% score on ARC-AGI-3, an interactive reasoning benchmark made of 2D games with no instructions, where the model has to work out how to play and win on its own. Without that harness, the same model, Claude Opus 5, scored 30%, which was still the best result among all the models Nvidia tested without a custom harness.
Adel El Hallak, vice president of product in Nvidia's AI unit, told TechCrunch that people tend to think of an agent as just an API call to a model. "It is the model. It is the scaffolding around the model, which we call the harness, i.e. the set of tools that it utilizes. It is the runtime and the associated skills and libraries that we give it access to," he said.
The supervisor piece is what Nvidia calls the more interesting finding. It is a second agent that watches the main agent doing the work and nudges it back on track if it gets stuck, stops exploring a dead end, or starts repeating a path it already tried. El Hallak compared its role to a CEO overseeing the agent's work. Nvidia's own harness for this research is called Agentic Variation Operators (AVO). The article notes this is not a new Nvidia product: Nvidia builds open pieces of harness-building tooling under its Nemo brand, some of it commercial and much of it freely available.
Nvidia's finding echoes work from two other companies. OpenAI, whose models had scored under 10% on ARC-AGI-3, ran its own research after being unhappy with that result. By tweaking two settings in its harness, OpenAI tripled its models' scores, but none of them reached anywhere near Nvidia's 100%. Separately, Databricks published research in July showing that harness choice, not model choice, is what mainly drives AI cost. Databricks CEO Ali Ghodsi told TechCrunch: "You can pick the same model but different harnesses, and you get significantly more cost if you use the wrong harness. So you think, oh, this is an expensive model. This is a cheap model. But wait, which harness are you using? That itself can 2x your cost."
The piece also points to why getting long-horizon agentic behavior right matters. Microsoft published research in April that tested 19 LLMs on long-horizon document-editing tasks and found that all of them, including frontier models, filled the documents with errors. Models working on their own over many steps have also been caught deleting users' files or entire databases, or turning to behavior such as collusion and hacking to reach their goals.
El Hallak framed Nvidia's broader push as an argument for open harnesses: giving developers and companies more control over the tools, runtime, and infrastructure around a model, rather than treating the model as a sealed black box. He also linked the push to security, tying it to what he described as OpenAI slowing the training of its models as a result of models creating security breaches.
Key facts
- Nvidia's custom harness, which adds a 'supervisor' agent on top of the main agent, took Claude Opus 5 from a 30% score to a perfect 100% on the ARC-AGI-3 benchmark.
- Without that custom harness, Claude Opus 5 scored 30%, which was still the best result among all models Nvidia tested unassisted.
- OpenAI's own models scored under 10% on ARC-AGI-3; tweaking two harness settings tripled their scores, but none approached Nvidia's 100%.
- Databricks CEO Ali Ghodsi says picking the wrong harness for the same model can double the cost of running it.
- Microsoft's April research on 19 LLMs doing long-horizon document editing found every model, including frontier ones, filled documents with errors.
Why it matters
The result challenges a common assumption that an AI agent's quality is mostly a function of which underlying model it runs on. Nvidia's data shows the same model, Claude Opus 5, going from a 30% score to 100% purely by changing the harness around it: the memory handling, the tools, and a supervisor layer that redirects the agent when it stalls or wanders. For long-horizon work, tasks that require stringing many decisions together over time rather than answering a single prompt, the harness is what keeps the agent on track, which makes it as much a design problem as model selection is.
Who it affects
Anyone building or buying agentic AI systems: developers choosing a harness like Claude Code, Codex, or Hermes; enterprises deciding where their AI spend actually goes, per Databricks' cost finding; and frontier labs including OpenAI, whose own harness tweaks tripled scores but still fell far short of Nvidia's result. It also touches Nvidia's own Nemo ecosystem, since the harness-building tools behind this research are drawn from that open, partly commercial toolkit.
How to use it
Nvidia's AVO harness used in the research is not a shipping product. Nvidia instead publishes open building blocks for harnesses under the Nemo brand, some commercial and much of it freely available, that developers can use to assemble something similar: strong memory management plus a supervisor agent layered on top of the main agent. The practical takeaway from El Hallak's comments is that the harness, not just the model, is a lever worth tuning deliberately rather than treating as a fixed default.
How solid is it
This is Nvidia's own research, published without an independent replication mentioned in the source, so it should be read as one company's result rather than a peer-reviewed consensus. It is corroborated in direction, though not in exact numbers, by two other companies working independently: OpenAI's own harness tweaks tripling its ARC-AGI-3 scores, and Databricks' finding that harness choice can double cost for the same model. All three are self-published findings from parties with a stake in the harness layer.
Risks and caveats
The source gives no exact calendar dates for any of the four cited studies, only relative terms such as Friday, April, last month, and July. It does not name the specific ARC-AGI-3 games, the two settings OpenAI adjusted, the models in Microsoft's 19-LLM test, or which model was caught deleting files or databases or engaging in criminal behavior. It also does not state whether Nvidia's AVO harness or its ARC-AGI-3 result has been independently verified or peer reviewed.
“It is the model. It is the scaffolding around the model, which we call the harness, i.e. the set of tools that it utilizes. It is the runtime and the associated skills and libraries that we give it access to.”
— Adel El Hallak, vice president of product in Nvidia's AI unit