InteractiveBench tests 12 MLLMs as egocentric AI assistants
The authors of a new preprint ask whether today's multimodal large language models (MLLMs) can really help people with physical tasks. The setting is egocentric AI assistants on wearable devices: they see what the user sees and give instructions for physical work when needed. Testing such assistance is hard, the authors say, because it usually requires real people, and real interaction data is expensive and rare.
To get around that, they propose InteractiveWorld, which they call the first verifiable environment for closed-loop interaction between a human and an AI assistant. Unlike existing robot-centric simulators, it lets the assistant guide a human agent who observes the scene, understands instructions and acts independently. The loop works like this: the assistant instructs, the human acts, and the environment executes the actions and updates what the human sees next.
On top of the environment the authors build InteractiveBench, which they describe as the first dynamic benchmark for egocentric AI assistants. It has 120 multi-turn tasks across six categories. In each task the assistant guides the human in real time, and every instruction shapes the outcome. Outcomes are scored by verifiable, state-based rewards.
The authors ran experiments with 12 frontier MLLMs and conclude that they are still far from reliable assistants. Even the strongest model, which the abstract names as Claude Fable 5.1, achieves at most 20% on long-horizon Guidance tasks and at most 25% on long-horizon Planning tasks. The authors present the work as a verifiable platform and a rigorous evaluation set, meant as a foundation for research on interactive MLLM assistants that truly help people in the physical world.
Key facts
- InteractiveWorld is described as the first verifiable environment for closed-loop interaction between a human and an AI assistant: the assistant instructs, the human acts, the environment updates what the human sees.
- InteractiveBench has 120 multi-turn tasks across six categories, scored by verifiable, state-based rewards.
- Twelve frontier MLLMs were tested, and the authors conclude they are still far from reliable assistants.
- The strongest model, named in the abstract as Claude Fable 5.1, scores at most 20% on long-horizon Guidance tasks and at most 25% on long-horizon Planning tasks.
Why it matters
Wearable assistants that see what the user sees and give instructions for physical work are a stated goal of MLLM development. Testing them has been hard because it usually needs real people, and real interaction data is expensive and rare. The authors position InteractiveWorld as a way to test this without people in the loop, by replacing the user with a human agent inside a simulated scene. They contrast it with robot-centric simulators, where the assistant controls a robot rather than guiding a person who acts on their own.
Who it affects
Researchers building and evaluating egocentric and wearable MLLM assistants are the direct audience: the authors offer the environment and benchmark as a platform for that research. Developers of the 12 tested frontier models get a measured picture of where long-horizon guidance and planning fall short.
How to use it
The benchmark is designed for evaluation. An assistant guides a human agent through a multi-turn task in real time, and the result is judged by state-based rewards. No release of code, data or a public leaderboard is mentioned in the source, so readers cannot yet tell how to run it.
How solid is it
This is a preprint abstract, and the claims are the authors' own, including the "first" claims for the environment and for the benchmark. The headline result is stated as "at most" 20% and 25% for the strongest model on two long-horizon task types. No scores are given for the other 11 models, and the six task categories are not listed beyond Guidance and Planning being mentioned. The abstract names no authors or institutions. The model name Claude Fable 5.1 is reported here exactly as the abstract gives it.
Risks and caveats
The abstract does not say how the human agent is implemented, for example whether it is itself a model, which matters for how far results transfer to real people. It also does not say whether the 20% and 25% figures are exact scores or ceilings across runs. With only 120 tasks across six categories, per-category results may rest on a small number of tasks, though the source gives no breakdown.
“they are still far from reliable assistants”
— Abstract of the InteractiveWorld paper, on 12 frontier MLLMs