Explainable AI should predict when people actually want to know
A new position paper argues that human-centered explainable AI (HCXAI) has a blind spot: it optimizes for making explanations available and technically sound, but rarely asks whether a person actually wants one at a given moment. Drawing on Sharot and Sunstein's framework of information-seeking motives, the authors propose that people decide whether to engage with an explanation by weighing three kinds of expected utility: instrumental (will it help me act better), hedonic (will it make me feel better), and cognitive (will it improve my understanding). Each of these estimates, the paper argues, is distorted by well-documented cognitive biases: illusion of control, automation bias, unrealistic optimism, impact bias, overconfidence, and confirmation bias. Those distortions push people toward one of two failure modes. Excessive information-seeking fragments attention across explanations without actually improving the decisions being made. Insufficient information-seeking leaves real risks and misunderstandings unexamined because the person never asked for the explanation that would have surfaced them. The paper argues this problem is sharper for agentic AI systems than for single-output models, since an explanation there has to do more than clarify one answer: it needs to help a person anticipate cascading actions the system might take, assess the risk of those actions, and judge when to step in and intervene. The authors' overall call is a reframing of the HCXAI design goal: instead of asking how to make explanations available, designers should ask when and why a given user would actually seek one, and build systems around that answer.
Key facts
- The framework rests on three expected-utility types that drive whether someone seeks an explanation: instrumental, hedonic, and cognitive.
- Six named cognitive biases distort each utility estimate: illusion of control, automation bias, unrealistic optimism, impact bias, overconfidence, and confirmation bias.
- The biases produce two opposite failure modes: over-seeking that fragments attention without improving decisions, and under-seeking that leaves risks and misunderstandings unexamined.
- Agentic AI systems raise the stakes because explanations there must support anticipating cascading actions, assessing risk, and deciding when to intervene, not just understanding one output.
- The framework builds on Sharot and Sunstein's existing psychology of information-seeking motives rather than introducing a new empirical study.
Why it matters
Explainable AI research has largely treated the problem as one of supply: build an explanation that is accurate and available, and the design job is done. This paper argues that framing misses the actual bottleneck, which is demand. Whether an explanation helps depends on whether the person facing it wants to engage with it at that moment, and that want is shaped by predictable psychological biases the field has not been designing around.
Who it affects
The argument is aimed at people who design and study human-centered explainable AI systems. It also names agentic AI systems specifically as the setting where the gap matters most, since users of those systems have to decide not just whether they understand a single output but whether to let a chain of automated actions continue or step in and intervene.
How to use it
As a position paper, it offers a design lens rather than a shippable tool: when building an explanation interface, evaluate it against the three utility types (does it help the user act, feel better, or understand more) and check which of the six named biases might be suppressing or inflating the user's felt need for it. The paper's own call to action is to shift design practice from making explanations available toward making them sought.
How solid is it
This is a position paper: it proposes and argues for a framework by synthesizing an existing psychology model (Sharot and Sunstein's account of information-seeking motives) and applying it to explainable AI. It does not report new experiments, user studies, or empirical tests of the framework, and it does not name specific AI systems or products as worked examples.
Risks and caveats
The two failure modes described, explanation overload that fragments attention and explanation gaps that leave risks unexamined, are the paper's own diagnosis of what goes wrong when this psychology is ignored, not problems it claims to have solved. Because the framework has not yet been tested empirically, how well it predicts real user behavior in deployed HCXAI or agentic systems remains open.