Glonce

AnTrap benchmark finds GUI agents fail under runtime anomalies

researchThe Glonce editorsHugging Face🔥74Glonce rating74Impact70Novelty75Relevance95Surprise55Players45Credibility80
AnTrap benchmark finds GUI agents fail under runtime anomalies

Researchers introduce AnTrap, a benchmark built to test whether GUI agents deployed on Android devices can cope with dynamic runtime anomalies, the kind of disruptions that show up in real deployments: unexpected pop-ups, misused actions and other unplanned interruptions to a task in progress. The authors note that existing benchmarks do not systematically test agents against this kind of disruption, which is what AnTrap is built to fix. The benchmark organizes real-world anomalies into a taxonomy of four layers, State, Thinking, Action and Round, broken down further into ten fine-grained subcategories. A construction pipeline injects these anomalies into agent execution trajectories while preserving whether the underlying task can still be completed, so a model's failure can be attributed to the anomaly rather than to an unsolvable task. Testing 16 leading GUI models on this benchmark, the authors report universal vulnerability to dynamic anomalies: even the strongest models suffer significant performance degradation once anomalies are introduced, though the paper does not give the specific numeric drop for any individual model. To probe how fixable this is, the authors also ran GRPO reinforcement learning training in both the original and the anomaly-injected environments. This let them separate anomalies that models can learn to handle from ones that reflect a deeper reasoning limitation. The result: single-step traps at the state and action layers are largely addressable through adversarial reinforcement learning, meaning a model can be trained to cope with a pop-up or a single misfired action. Deep contextual traps, the paper's example is a state deadlock, are different. These expose limitations that adversarial training inside trap-containing environments alone cannot resolve, meaning better training data is not enough to fix them.

Key facts

  • AnTrap is a new benchmark that injects dynamic runtime anomalies into Android GUI agent execution trajectories to test robustness.
  • Its taxonomy covers four layers, State, Thinking, Action and Round, split into ten fine-grained anomaly subcategories.
  • 16 leading GUI models were evaluated and all showed universal vulnerability, with significant performance degradation even in the strongest models.
  • GRPO reinforcement learning training in both clean and adversarial environments was used to separate learnable failures from deeper reasoning limitations.
  • Single-step traps at the state and action layers are largely fixable through adversarial RL training, but deep contextual traps like state deadlock are not.

Why it matters

GUI agents are increasingly expected to operate real Android apps unsupervised, and the paper's authors argue that existing benchmarks lack systematic evaluation of agent robustness against runtime anomalies, meaning the field has largely been testing agents against clean, cooperative environments rather than the messy ones they will actually face. AnTrap closes that gap by deliberately engineering anomalies that a real device can throw at an agent mid task, and its headline finding, universal vulnerability across all 16 models tested, is a direct measurement of how far current agents are from being reliable outside the lab.

Who it affects

The benchmark speaks most directly to teams building or evaluating Android GUI agents, since it gives them a way to measure robustness rather than only task success. It also matters to anyone building products on top of such agents, since it quantifies a category of failure that would otherwise only surface after deployment, when an agent encounters an anomaly it was never tested against.

How to use it

AnTrap is a benchmark and evaluation pipeline rather than a product: its use is to evaluate a GUI agent's robustness by running it through anomaly-injected trajectories across the four taxonomy layers. The paper also demonstrates a mitigation path, GRPO training conducted in adversarial environments, which teams can apply to harden agents against the single-step traps the paper identifies as learnable, though it does not resolve the deeper reasoning-bottlenecked category.

How solid is it

The claims rest on a systematic construction pipeline and testing across 16 leading GUI models, giving the robustness findings reasonably broad coverage across current systems. That said, the source text does not give per-model numeric degradation figures, nor does it name the authors' affiliations, a publication venue, or a date, so the result should be read as reported rather than independently checked against those details.

Risks and caveats

The paper does not specify how much any individual model's performance dropped, only that the degradation was significant and universal, so the exact severity per model is not verifiable from the available text. It also does not describe the technical detail of how the anomaly-injection pipeline preserves task solvability. Most notably, the finding that deep contextual traps like state deadlock resist adversarial training altogether points to a category of GUI agent failure that current training methods, on the evidence here, cannot fix by adding more adversarial examples alone.