Glonce

ECHOES trains GUI agents to replace screenshot history with text

researchThe Glonce editorsarXiv🔥57Glonce rating57Impact45Novelty60Relevance95Surprise25Players10Credibility70

The paper starts from a trade-off in long-horizon GUI agents. These agents use visual history to keep track of the state of a task, but keeping past screenshots raises visual-token usage and inference costs. Replacing them with textual summaries is cheaper, but the authors say such summaries can discard critical state information and hurt reliability over long tasks.

Their answer is ECHOES, which uses training-time privileged information to guide how the history is compressed and to strengthen long-horizon reasoning. The agent works in a Compression-Think-Action (CTA) format: at each step it recursively updates a textual history state, reasons, and then acts.

The training has two parts. The same model, when given privileged multi-image inputs, acts as a teacher. It gives dense, token-level on-policy distillation on CTA rollouts that the student model generates itself. On top of that sits a Perception-Memory Fidelity (PMF) Gap-guided OPD-RL framework, which dynamically balances privileged distillation against task-reward optimization. The teacher is updated periodically, which the authors say supports continued improvement.

The paper also introduces ECHOESBMK, a fine-grained diagnostic benchmark built from real GUI agent failure trajectories. It is meant to evaluate visual perception and cross-modal state retention together.

The authors report experiments on four GUI benchmarks plus ECHOESBMK and say they show substantial gains in task performance and memory fidelity. At deployment ECHOES does not keep historical screenshots; the authors say it still reaches competitive performance with a favorable performance-versus-inference-cost trade-off, and shows positive zero-shot transfer to unseen visual planning tasks.

Key facts

  • ECHOES lets a GUI agent carry a recursively updated textual history state instead of retaining past screenshots at deployment.
  • The agent uses a Compression-Think-Action (CTA) format: compress history, reason, then act.
  • The same model with privileged multi-image inputs acts as teacher, giving token-level on-policy distillation on the student's own rollouts; a PMF Gap-guided OPD-RL framework balances distillation against task reward.
  • The paper adds ECHOESBMK, a diagnostic benchmark drawn from real GUI agent failure trajectories, and tests on four GUI benchmarks plus ECHOESBMK.
  • The authors claim substantial gains in task performance and memory fidelity, competitive performance with a favorable cost trade-off, and positive zero-shot transfer to unseen visual planning tasks.

Why it matters

Agents that operate graphical interfaces over many steps face a real cost problem: every retained screenshot adds visual tokens and inference expense. The obvious fix, summarizing history in text, risks losing state the agent needs later. ECHOES targets that gap by training the compression itself, using information available only at training time (multi-image inputs given to the teacher) to teach the student what its textual history must preserve.

Who it affects

Researchers and engineers building long-horizon GUI agents, especially those who pay for visual-token usage when keeping screenshot history. The ECHOESBMK benchmark is also aimed at anyone who wants to diagnose failures in visual perception and cross-modal state retention.

How to use it

This is a research method, not a product. The practical idea for builders is the pattern: keep a compact textual history state that is updated at every step, and use a teacher with access to the full multi-image history during training only. No code or model release is mentioned in the source.

How solid is it

This is an arXiv paper, and everything above comes from its abstract. The authors report experiments on four GUI benchmarks and ECHOESBMK, a benchmark they introduce themselves. No numeric results (accuracy, success rates, token savings, cost figures) are given in the abstract. The four GUI benchmarks are not named. The abstract says 'competitive performance', not that ECHOES beats all baselines, and does not say what it is compared against.

Risks and caveats

The headline claims of 'substantial gains' and a 'favorable trade-off' cannot be checked from the abstract alone, since it gives no figures and no named baselines. No base model, model size or training dataset is named. The authors themselves note that textual summaries can lose critical state information, which is the failure mode this method is meant to reduce rather than a problem shown to be eliminated. The zero-shot transfer result is described as positive, with no size given.