Glonce

SoftwareGym synthesizes 182 GUI apps to train computer-use agents

researchThe Glonce editorsHugging Face🔥50Glonce rating50Impact55Novelty60Relevance100Surprise30Players15Credibility80
SoftwareGym synthesizes 182 GUI apps to train computer-use agents

SoftwareGym is an automated pipeline for synthesizing GUI software environments, built to train computer-use agents (CUAs). The paper starts from a problem it sees in existing options. Native application environments take substantial construction and integration effort, which limits application coverage and raises the risk of application-specific overfitting. Browser-hosted environments, in turn, offer limited interaction depth and workflow complexity.

The pipeline aims for diverse interfaces, executable functionality and runtime-verified behavior. It generates structurally diverse designs, realizes them as runnable applications, and then applies GUI evaluation with iterative repair to make the apps functionally reliable.

Using SoftwareGym, the authors synthesized 182 applications across twelve categories. They report a mean GUI task pass rate of 91.5% for these applications. They also built SoftwareGym-Bench, a 120-task evaluation suite that spans the same twelve categories, and collected trajectories from the synthesized environments to train a CUA.

The resulting model improves Pass@3 on OSWorld from 41.39% to 56.97%, a gain of about 15.6 percentage points. The authors describe OSWorld as strictly out-of-domain relative to the training environments. The same model achieves 40.0% on SoftwareGym-Bench. The authors conclude that the results demonstrate the potential of synthesized software environments as scalable and transferable supervision for CUA training.

Key facts

  • SoftwareGym is an automated pipeline that designs, builds and repairs runnable GUI applications for computer-use agent training.
  • The authors synthesized 182 applications across twelve categories, with a mean GUI task pass rate of 91.5%.
  • SoftwareGym-Bench is a 120-task evaluation suite spanning the same twelve categories.
  • A model trained on trajectories from the synthesized environments improves OSWorld Pass@3 from 41.39% to 56.97%, and OSWorld is described as strictly out-of-domain.
  • The same model scores 40.0% on SoftwareGym-Bench; the metric is not specified in the source.

Why it matters

Training computer-use agents needs many realistic software environments, and the paper argues the current ways of getting them are limiting. Native apps are costly to build and integrate, and browser-hosted ones are shallow. SoftwareGym proposes generating the apps instead, with runtime checks and repair so they actually work. The headline result is a jump in OSWorld Pass@3 from 41.39% to 56.97% on a benchmark the authors call strictly out-of-domain, which suggests the training signal carries over to apps the model did not see.

Who it affects

Researchers and teams that train or evaluate computer-use agents are the direct audience, since environment supply is the bottleneck the paper targets. Anyone benchmarking CUAs on OSWorld may also take note of the reported gain.

How to use it

The abstract describes the method rather than a product. The recipe is to generate structurally diverse application designs, realize them as runnable apps, run GUI evaluation against them, repair what fails, and then collect agent trajectories from the working environments for training. The source mentions no code, data or model release, so there is nothing to download or run from the abstract alone.

How solid is it

The numbers come from the paper's own abstract and are internally consistent: 182 applications, twelve categories, a 120-task benchmark, and a Pass@3 move from 41.39% to 56.97% on OSWorld. Several details that would help judge the result are not given: the base model is not named, the metric behind the 40.0% on SoftwareGym-Bench is not stated, and no baseline or other-model comparison on SoftwareGym-Bench is given. The training method, number of trajectories and compute cost are not stated, and no comparison with other environment-synthesis methods or other agents on OSWorld is given. The source does not say whether the 91.5% pass rate covers all 182 applications or a subset.

Risks and caveats

The authors frame the outcome as showing the potential of synthesized environments, not a settled result. The 91.5% figure is a mean, and the twelve categories are not listed, so it is hard to tell how varied the apps really are. The benchmark SoftwareGym-Bench was built by the same authors from the same twelve categories as the training environments, so the 40.0% score there is not an independent test. The OSWorld gain is the stronger evidence, but without the base model and training details it is hard to say how much of it comes from the synthesized environments alone.

“These results demonstrate the potential of synthesized software environments as scalable and transferable supervision for CUA training.”

— SoftwareGym paper abstract