Hybrid Computer Use (HCU) lets agents mix code, accessibility and GUI

A paper titled "Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents" takes aim at how computer-use agents interact with machines. The authors start from an observation: agents now handle tasks from everyday desktop activities to long-horizon professional workflows, but interfaces built around screenshots and mouse-and-keyboard actions can impose a human-interaction prior. That means the agent has to interpret the screen again and again, while the computer's semantic and programmatic capabilities stay underexposed.

Their answer is to expose those capabilities and let the model choose and combine them. They introduce Hybrid Computer Use (HCU), a unified interface for general agents that combines three complementary channels. The first is programmatic interaction through code and command-line tools. The second is semantic interaction through accessibility (AX). The third is visual interaction through graphical user interfaces (GUI).

Several design pieces are described. HCU supports efficient AX interaction and guards against stale-state actions when the agent switches between channels. A supplementary HCU skill guides agents in selecting and coordinating channels so they use the unified interface effectively. The AX tools return compact observations, execute actions in batches and report their effects as AX diffs, so the model needs fewer round trips.

On the result side, the authors say that on OSWorld 2.0, HCU achieves the highest Partial and Binary scores among the compared systems. From an analysis of agent trajectories, they find that all evaluated models improve with HCU, yet allocate work across the channels differently. They also state that the models consistently outperform compared baselines in both efficiency and task success. Their conclusion is that unified computer-use interfaces, which allow flexible channel selection and combination, are valuable.

Key facts

Hybrid Computer Use (HCU) is a unified interface for general computer-use agents that combines three channels: programmatic (code and command-line tools), semantic (accessibility, AX) and visual (GUI).
The authors argue that screenshot and mouse-and-keyboard interfaces impose a human-interaction prior, forcing repeated screen interpretation while leaving semantic and programmatic capabilities underexposed.
The AX tools return compact observations, execute actions in batches and report effects as AX diffs, so the model needs fewer round trips; HCU also guards against stale-state actions when switching channels.
On OSWorld 2.0, HCU achieves the highest Partial and Binary scores among the compared systems, according to the authors.
Trajectory analysis shows all evaluated models improve with HCU, though they allocate work across the channels differently.

Why it matters

Most computer-use agents work the way a person does: look at a screenshot, move the mouse, press keys. The authors argue this wastes what the machine can already offer, namely code, command-line tools and accessibility information. HCU is a proposal to stop treating the screen as the only way in and to let the model decide which channel suits each step. If the reported results hold, the lesson is about interface design as much as about model strength: the same models improve when given a richer interface.

Who it affects

Teams building general computer-use agents, from everyday desktop assistants to agents meant for long-horizon professional workflows, are the audience the paper addresses. It also concerns anyone evaluating such agents on OSWorld 2.0, since the paper reports results on that benchmark.

How to use it

The paper describes a design rather than a product. Its practical ideas are to expose code and command-line tools, accessibility and GUI side by side, to give the accessibility tools compact observations, batched actions and AX diffs, and to add a supplementary HCU skill that guides the agent in selecting and coordinating channels. It also describes guarding against stale-state actions when the agent switches channels.

How solid is it

This is a preprint listed on Hugging Face Papers, and the claims are the authors' own. They report the highest Partial and Binary scores among the compared systems on OSWorld 2.0, and say all evaluated models improve with HCU. The abstract gives no numeric Partial or Binary scores for HCU or for any baseline. The abstract does not name the evaluated models or the compared baseline systems. The abstract gives no figure for the reduction in round trips, steps or time. The size of the gains cannot be judged from the text alone.

Risks and caveats

The claim of the highest scores holds only "among the compared systems", so it depends on which baselines were chosen. The abstract does not say whether code or weights are released, and it does not say which operating systems the AX tools support. The authors also note that models allocate work across channels differently, so the benefit may vary by model, and the supplementary skill that guides channel choice is part of the setup that produced the results.

“We argue for exposing these capabilities and letting models choose and combine them.”

— Abstract of the Hybrid Computer Use paper