Glonce

EvoSkill-GUI lets agents rewrite their own GUI skills without retraining

researchThe Glonce editorsHugging Face🔥75Glonce rating75Impact68Novelty78Relevance95Surprise62Players35Credibility85
EvoSkill-GUI lets agents rewrite their own GUI skills without retraining

Researchers propose EvoSkill-GUI, a training-free framework that lets GUI agents update their own skills while they run, instead of relying on skills fixed before deployment. The paper's premise is that GUI agents handle long, multi-step tasks on interfaces that keep changing underneath them: pop-ups appear, pages load late, widgets move, and a plan drawn up in advance breaks. Existing agent-skill frameworks package reusable procedural knowledge to help with this, but the authors argue those skills are still treated as static artifacts written once before the agent is deployed, not as knowledge that keeps improving through use. Their proposed fix is to make each skill a structured, multi-file package holding retrieval metadata, executable plans, backup ways to locate on-screen elements, rules for recovering from failure, accessibility utilities, and records of past failure cases. EvoSkill-GUI runs a reflect-revise-reuse loop: the executor makes instant revisions mid-rollout, a separate critic component analyzes failed attempts under strict isolation from the executor, and the executor then edits the specific skill files through a restricted tool interface, rather than an open-ended one. The authors tested the framework across three GUI benchmarks that span both mobile and desktop platforms, MobileWorld, AndroidWorld, and OSWorld. They report that EvoSkill-GUI consistently improved multiple base models with no additional training, with maximum accuracy gains of +16.2% on MobileWorld, +6.0% on AndroidWorld, and +10.5% on OSWorld. They also report that the skill libraries the framework builds up keep helping on related tasks afterward, rather than needing to be rebuilt from scratch for each new task. The paper's code is published on GitHub under the ZJU-REAL organization.

Key facts

  • EvoSkill-GUI is a training-free framework that revises GUI agent skills from execution feedback at deployment time, rather than treating skills as fixed before deployment.
  • Each skill is a multi-file package: retrieval metadata, executable plans, backup element-localization, failure-recovery rules, accessibility utilities, and past failure cases.
  • It works through a reflect-revise-reuse loop: the executor revises mid-rollout, an isolated critic diagnoses failed trajectories, and the executor edits skill files through a restricted tool interface.
  • Reported maximum accuracy gains: +16.2% on MobileWorld, +6.0% on AndroidWorld, and +10.5% on OSWorld, across multiple base models with no extra training.
  • Evolved skill libraries keep benefiting related tasks afterward instead of being rebuilt from scratch; code is public on GitHub.

Why it matters

Most GUI-agent skill systems bake procedures in once, before the agent ever runs, so they cannot adapt when a real interface behaves differently than expected. EvoSkill-GUI's contribution is treating a skill as something that keeps changing after deployment, revised from the agent's own failures rather than from a fresh round of training. That is a meaningful shift for a field where retraining an agent every time an interface changes is expensive and slow.

Who it affects

The paper does not name the specific base models it improved, but the approach targets anyone building or operating GUI agents that navigate mobile or desktop interfaces over long, multi-step tasks, since that is exactly the setting where fixed plans break down.

How to use it

EvoSkill-GUI is training-free: it works as a layer on top of an existing agent rather than requiring the base model to be retrained. The authors have published their code on GitHub (ZJU-REAL/EvoSkill-GUI), so the framework can in principle be tried against an existing GUI agent setup without a separate training run.

How solid is it

The results come from the paper's own experiments across three established GUI benchmarks, MobileWorld, AndroidWorld, and OSWorld, spanning mobile and desktop platforms, with reported gains against multiple base models. The abstract does not give baseline accuracy figures alongside the gains, and does not name the base models tested, which limits how far the reported percentages can be checked or generalized from the abstract alone.

Risks and caveats

The abstract does not specify what 'strict information isolation' between the executor and the critic actually consists of beyond the phrase itself, nor does it give dates for when the benchmark runs were conducted. No baseline numbers are given, only the size of the improvement, so the absolute accuracy level the agents reach is not stated here.

“What GUI agents need is not better static skills, but skills that can be revised from execution feedback at deployment time, without additional training.”

— the paper's authors