SPIEval benchmark finds mobile AI assistants top out at 57% accuracy
Researchers introduced SPIEval, a human-curated benchmark for testing how well large language models work as mobile assistants when the information they need to complete a task is scattered across several apps rather than sitting in one place. The benchmark is built around five cognitive capabilities: reasoning, disambiguation, integration, preference inference, and multi-intent decomposition. It comprises 250 tasks spanning 4,335 personal records distributed across 10 apps, and supports multi-turn interaction through 21 tools, so a model has to actually search and retrieve information rather than answer from a single screen. The authors describe the benchmark as covering diverse scenarios and challenging tasks, with scattered information, controllable environments, and verifiable outcomes. They evaluated nine representative LLMs on SPIEval. The best performer, GPT-5.5 (xhigh), reached only 57.3% accuracy; the weakest model in the test managed just 16.4%. Digging into why models fail, the authors found that 79% of failures trace back to inaccurate information localization: models often settle on an answer that looks plausible instead of continuing to retrieve and verify the correct information across apps. They also found that fewer than 2% of retrieval actions used advanced search methods, with substantial variation in search efficiency from one model to the next. The authors say these findings expose fundamental limitations of current LLM-based mobile assistants and should motivate further research in the area. Data and code for SPIEval are published at huggingface.co/datasets/Junjie-Ye/SPIEval.
Key facts
- SPIEval has 250 tasks covering 4,335 personal records across 10 apps and 21 tools, built around five cognitive capabilities.
- Nine LLMs were evaluated; the best, GPT-5.5 (xhigh), scored 57.3% accuracy, and the weakest scored 16.4%.
- 79% of failures stem from inaccurate information localization: models commit to a plausible but wrong answer instead of continuing to verify.
- Fewer than 2% of retrieval actions used advanced search methods, with search efficiency varying widely between models.
- Data and code are released publicly at huggingface.co/datasets/Junjie-Ye/SPIEval.
Why it matters
LLM-based mobile assistants are increasingly expected to pull together personal information that lives across many apps, from messages to calendars to notes, to complete a single instruction. SPIEval is built specifically to test that scattered-information case rather than single-app lookups, and it shows the gap is large: even the strongest model tested clears barely more than half its tasks. The dominant failure mode it identifies, models settling for a plausible but wrong answer instead of continuing to search, points at a specific weakness in how current assistants handle uncertainty during retrieval.
Who it affects
The benchmark is aimed at researchers and developers building or evaluating LLM-based agents and mobile assistants that need to work across multiple apps and data sources, as well as teams deciding which model to deploy for personal-assistant style products.
How to use it
SPIEval's data and evaluation code are published openly at huggingface.co/datasets/Junjie-Ye/SPIEval, so a team can run its own models against the 250 tasks and compare results against the nine models already evaluated, including the top-scoring GPT-5.5 (xhigh).
How solid is it
The benchmark is human-curated and grounded in five defined cognitive capabilities, with a sizeable task set: 250 tasks over 4,335 personal records spread across 10 apps and 21 tools, and it supports multi-turn interaction so tasks require genuine retrieval rather than single-step lookup. Nine LLMs were evaluated on it, with a clear spread of results from 16.4% to 57.3% accuracy, and the source backs its key failure-mode claim with a specific figure, 79% of failures tied to information localization.
Risks and caveats
The source does not name eight of the nine evaluated models, only the top performer, GPT-5.5 (xhigh); the weakest-performing model is left unidentified. It does not specify what counts as an 'advanced search method' or which models used them, does not break results down by the five cognitive capabilities, and gives no author affiliation or publication date.