Glonce

iPrefBench tests whether iOS agents know when to infer and when to ask

researchThe Glonce editorsHugging Face🔥57Glonce rating57Impact45Novelty55Relevance95Surprise30Players10Credibility75
iPrefBench tests whether iOS agents know when to infer and when to ask

The paper presents iPrefBench, a benchmark for personal mobile-use agents. Its starting point is that computer-use agents can now execute fully specified tasks across apps, but real user requests are often underspecified. To act correctly for a particular user, an agent has to infer some missing information from the user's activity history and ask for other details that only the user can provide. It should do this without making unsupported assumptions and without burdening the user with unnecessary questions.

The benchmark runs in an executable iOS environment with an interactive simulated user. It is grounded in a coherent persona with 3,051 past activity events and comprises 130 underspecified user tasks across 26 apps. By construction, each omitted user-dependent requirement is either history-inferable (the agent can find it in the persona's past activity) or ask-only (only the user can supply it).

iPrefBench scores agents on two things: whether they complete tasks correctly for the user (personalized task completion), and whether they seek user input when necessary (user-agent interaction).

The authors evaluated 7 frontier agents and report a substantial gap between execution and personalization. Agents reach high action completion scores, up to 91.6%, but full task success is low, up to 47.7%. The authors attribute the low success to personalization lagging behind. Agents especially struggle with ask-only information: they fail to request enough of the necessary information. The authors conclude that knowing what to infer and when to ask is a real challenge for personalized agents, and they hope iPrefBench can serve as a testbed for building agents that know their user and work with them.

Key facts

  • iPrefBench has 130 underspecified user tasks across 26 apps, run in an executable iOS environment with an interactive simulated user.
  • The benchmark is grounded in one coherent persona with 3,051 past activity events; each missing requirement is either history-inferable or ask-only.
  • Seven frontier agents were evaluated: action completion reaches up to 91.6%, but full task success only up to 47.7%.
  • Agents especially struggle with ask-only information, failing to request enough of what they need from the user.
  • The authors frame the open problem as knowing what to infer and when to ask.

Why it matters

Most agent benchmarks test whether an agent can carry out a fully specified task. iPrefBench tests a different step: a vague request where the agent must work out what is missing and where to get it. The reported gap, up to 91.6% action completion against up to 47.7% full task success, suggests that executing taps and steps is no longer the main obstacle for these agents. Personalization is.

Who it affects

Researchers and teams building personal assistants and mobile-use agents are the direct audience, since the benchmark is offered as a testbed for agents that know their user. Users of such assistants are affected indirectly: the failure the paper highlights is an agent that guesses or skips a question when only the user could have answered it.

How to use it

The abstract describes iPrefBench as a testbed: an agent is run on the 130 tasks in the executable iOS environment, talks to the simulated user, and is scored on personalized task completion and on user-agent interaction. Developers can use the split between history-inferable and ask-only requirements to see whether an agent fails at finding information in the history or at recognizing that it must ask.

How solid is it

The source is the paper's abstract, so the numbers are the authors' own headline results. Both figures are 'up to' values for the best case among the 7 agents, not averages. The setup is concrete: 130 tasks, 26 apps, 3,051 persona events, and requirements that are inferable or ask-only by construction. The abstract does not name the agents, so the reader cannot tell which agent produced which score.

Risks and caveats

The abstract does not name the 7 agents or say whether the 91.6% and 47.7% figures come from the same agent. It gives no separate numbers for ask-only versus history-inferable requirements, although it says ask-only is where agents struggle most. It also does not say how the persona was built or whether it is based on real users. The benchmark rests on one persona and a simulated user, so results may not carry over to other people.