XYBench tests whether AI assistants spot the problem behind a question

XYBench tests whether AI assistants spot the problem behind a question

XYBench is a benchmark of 8K+ questions (8,115 real-world questions, to be exact) that tests how AI assistants respond when a request rests on a misconception. The authors' example: someone asks how to thicken frosting that keeps sliding off cupcakes, when the real problem is that the cupcakes are still too warm. This kind of mismatch between the approach a person asks about and the goal they want to achieve is called the XY problem: the person wants X, assumes approach Y will work, and asks for help with Y. The authors say XYBench won the Best Resource Paper Award at EMNLP 2026, and that it was developed with researchers at Carnegie Mellon, NVIDIA and Johns Hopkins.

The benchmark draws on four sources: 1,557 questions from Stack Overflow, 184 from Stack Exchange, 102 manually curated examples, and 6,272 questions generated from WikiHow guides. For each question the team identified the stated request (Y) and the intended request (X), the underlying goal that can be inferred from context. For the public forum questions they used the three highest-scored human answers and the accepted answer, where available, to establish the intended request and possible solutions, and filtered out cases where answering the stated request was judged sufficient. For WikiHow, Claude Sonnet 4.5 generated questions around plausible misconceptions in procedural guides; these were then refined to give an expert enough context to infer the goal without stating it. Examples where answering the literal request would largely resolve the problem were removed.

Each response is scored on three aspects. Pragmatic solution records whether the response contains a solution to the stated request (Y) and to the intended request (X), checking each separately. Pragmatic emphasis measures whether the answer focuses on Y, on X, or balances the two. Misconception identification measures whether the response steers the user toward the underlying goal while discouraging or avoiding the mistaken approach. The authors illustrate this with a vacuum cleaner: a model might spend several paragraphs recommending a more powerful vacuum, then briefly mention checking for a clogged hose. That counts as containing a solution to the real goal, but the emphasis is on the stated request. Claude Sonnet 4.5 acts as the automated judge. To validate it, two human reviewers independently assessed 500 model responses. Averaged across the reviewers, agreement with the judge ranged from 97.5 to 100% on whether a solution appeared, 84.6 to 93.3% on misconception identification, and 74.2 to 79.8% on pragmatic emphasis.

The team evaluated Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro in a single-turn setting, one response per question; human reference answers were scored separately with the same metrics. The models often suggested a solution to the user's problem but kept their answers focused on the original, mistaken approach. Unlike the human references, all five were less likely to steer users away from that approach and toward a solution suited to their goal. Across all four sources, every model was more likely to include a solution to the stated request than to the underlying goal. On Stack Overflow, stated-request solutions appeared in 75 to 86% of responses, against 59 to 71% for underlying-goal solutions. On emphasis, the models focused primarily on the underlying goal in only 35 to 41% of their Stack Overflow responses and on the stated request in 47 to 57%; the rest gave similar weight to both.

An oracle experiment supplied both X and Y alongside each question. On Stack Overflow, Sonnet 4.6's rate of including a solution to the underlying goal rose from 59% to 67%, but its misconception-identification rate fell from 47% to 34%. Across the sources with human reference answers, every model in this experiment still trailed humans on misconception identification. The authors read this as a sign that detecting intent is only part of the challenge: models also need to act on it when it diverges from the stated request.

A separate experiment asked whether models can recognise good advice when judging rather than writing. The same five models plus GPT-4.1 were shown each question with two candidate answers, one focused on the literal request and one explaining the misconception and addressing the underlying goal. On Stack Overflow, all six chose the goal-focused answer in 71 to 84% of cases, compared with only 35 to 41% of the answers the five models wrote themselves in the main evaluation. The authors conclude that this exposes a gap between recognising useful advice and producing it, which a multiple-choice test alone would miss.

The authors say researchers can use XYBench to test whether changes to prompting or training help models identify misconceptions, offer solutions to underlying problems and make those solutions the focus of answers. Measuring each behaviour separately shows whether progress in one improves the others. They note that XYBench currently covers English-language queries and mainly procedural problems, and that extending it to other languages, cultures and conversational settings remains an open direction.

Key facts

  • XYBench contains 8,115 real-world questions from Stack Overflow (1,557), Stack Exchange (184), 102 manually curated examples and WikiHow-based generated questions (6,272), covering programming and everyday tasks such as cooking and home repair.
  • Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro were all less likely than human reference answers to steer users away from a mistaken approach toward their real goal.
  • On Stack Overflow, models included a solution to the stated request in 75 to 86% of responses but to the underlying goal in 59 to 71%, and focused primarily on the underlying goal in only 35 to 41%.
  • When the user's goal was supplied explicitly, Sonnet 4.6's goal-solution rate rose from 59% to 67% on Stack Overflow, but its misconception-identification rate fell from 47% to 34%.
  • As judges choosing between two answers, six models picked the goal-focused one in 71 to 84% of Stack Overflow cases, a gap between recognising good advice and writing it.

Why it matters

The authors say truly useful assistants must help users who ask misguided questions, not just users who ask the right question with the right amount of detail. XYBench measures whether a model notices that the question itself rests on a mistaken assumption, and redirects the user to the real problem, as in the cupcake and vacuum examples. The authors say this is a critical gap in LLMs' ability to troubleshoot. The results show that all five evaluated models tend to answer the literal request and give the real goal less weight than human answers do.

Who it affects

Researchers building or evaluating assistants are the stated audience: the authors say XYBench can test whether prompting or training changes help models identify misconceptions and focus on users' underlying problems. It also bears on everyday users, especially people who do not know the right questions to ask, which the authors name as the motivation.

How to use it

The authors suggest running XYBench to see whether changes to prompting or training improve three behaviours: identifying misconceptions, offering solutions to the underlying problem, and making those solutions the focus of the answer. Because each behaviour is scored separately, a team can check whether progress on one carries over to the others. The authors also point to future work on which training methods help models act on inferred intent.

How solid is it

The benchmark is a first-party release by the authors, developed with researchers at Carnegie Mellon, NVIDIA and Johns Hopkins, and the authors say it won the Best Resource Paper Award at EMNLP 2026. The automated judge, Claude Sonnet 4.5, was checked against two human reviewers on 500 responses: agreement was 97.5 to 100% on whether a solution appeared and 84.6 to 93.3% on misconception identification, but lower, 74.2 to 79.8%, on pragmatic emphasis. The headline figures quoted here are for Stack Overflow, and they are ranges across the five models without saying which model is which.

Risks and caveats

Evaluation was single-turn, with one response per question, so conversational follow-up is not tested. The benchmark covers English-language queries and mainly procedural problems. The 6,272 WikiHow questions were generated by Sonnet 4.5 and refined, not collected from real users. Judge agreement on emphasis is the weakest of the three measures. The oracle result is mixed: supplying the goal raised Sonnet 4.6's goal-solution rate but lowered its misconception identification, and models still trailed humans there.

“The results expose a gap between recognizing useful advice and producing it, which a multiple-choice test alone would miss.”

— XYBench authors