Fuse framework finds LLMs stumble at inferring hidden motives
The authors introduce Fuse, a multi-agent simulation framework built to evaluate how well LLM assistants reason about social situations when they only hear about them secondhand, through a user's own account of what happened. In everyday use, someone describes a conflict or an ambiguous interaction to an AI assistant and expects it to work out what the other person really wanted, but testing that skill has been hard for two reasons: the assistant has to reconstruct the situation entirely from one person's subjective narrative, and there is usually no objective ground truth for something as slippery as another person's true intentions.
Fuse solves the ground truth problem by construction. In each simulated scenario, a target agent is given a hidden motive and interacts with other agents, one of which plays the role of a user. That user agent then consults the assistant under evaluation and asks it to infer the target agent's real motive, so the correct answer is always known because the researchers set it themselves.
To confirm the simulated scenarios read like genuine social situations rather than artificial ones, the authors ran a human study collecting 24k annotations and used it to validate the simulation's faithfulness. They then applied Fuse to 12 different LLMs and used it to isolate four separate weaknesses: user mediation compounds the inherent difficulty of social reasoning; models show systematic sensitivity to biased user framing, so a slanted retelling skews their conclusions; some models need more detail than a human would to reach the same correct inference; and letting a conversation run longer, even though it gives the assistant room to ask clarifying questions, does not reliably improve its guesses. The authors have open-sourced Fuse along with a dataset of 21k examples.
Key facts
- Fuse is a multi-agent simulation where a target agent has a hidden motive and a user-representing agent consults the evaluated assistant to infer it, giving verifiable ground truth by construction.
- Simulation faithfulness was validated through a human study collecting 24k annotations.
- Fuse was applied to 12 LLMs; user mediation compounds the difficulty of social reasoning, and models show systematic sensitivity to biased user framing.
- Models can need more detail than humans do to reach a correct prediction, and longer conversations do not always improve performance despite allowing clarifying questions.
- Fuse and a 21k-example dataset have been open-sourced.
Why it matters
People increasingly turn to AI assistants for advice on social and interpersonal situations, and the value of that advice depends on whether the assistant can accurately read between the lines of what the user tells it, since it never observes the actual interaction, only one side's account of it. Fuse gives researchers a way to measure that skill against a verifiable ground truth, rather than relying on unfalsifiable judgments about what really happened between people.
Who it affects
Builders of AI assistants and chatbots that offer social or relationship advice, and researchers studying social reasoning and theory-of-mind in LLMs, gain a benchmark where ground truth exists by construction rather than by guesswork. Everyday users who rely on an assistant to interpret an ambiguous message or someone else's behavior are the people whose real-world experience the paper's findings speak to.
How to use it
Fuse and its 21k-example dataset are open-sourced, so teams evaluating or fine-tuning an assistant's social reasoning can run their own models through the same simulation setup the authors used to test the 12 LLMs.
How solid is it
The setup is explicitly a simulation of social interactions rather than a study of real ones, but the authors address that objection directly: a 24k-annotation human study is used to confirm the simulated scenarios are faithful to genuine social situations. Applying the framework across 12 different LLMs, and isolating four distinct factors rather than reporting a single headline score, points to a systematic pattern rather than an artifact of one model or one metric.
Risks and caveats
The text names no authors, institutions, or specific per-model scores, so it is not possible to say from it alone which of the 12 LLMs performed best or worst, or how large the gaps between them are. No timeframe is given for when the human study or the model evaluations took place, and no detail is provided on how the 24k annotations or the 21k dataset examples were collected or split, which limits how much can be said about the strength or representativeness of the underlying data.