Glonce

GPT-5.4 underestimates test item difficulty, study finds

researchThe Glonce editorsarXiv🔥65Glonce rating65Impact55Novelty60Relevance95Surprise45Players35Credibility80

A new study examines whether large language models can predict item difficulty, the standard measure of how hard a single test question is, using items drawn from a large-scale Reading and Writing test. The researchers tried multiple LLMs under various prompting strategies and parameter settings, and compared their results against encoder-only language models and feature-based supervised machine learning models.

Among the LLMs, zero-shot GPT-4.1 run at a temperature of 0 performed best, reaching a quadratic weighted kappa (QWK) of 0.578, the metric used to score how closely predicted difficulty levels matched the true ones. That result still fell short of ConvBERT, an encoder-only model, which scored a QWK of 0.625 and outperformed both every LLM tested and the best feature-based supervised machine learning model.

Further analysis found that all LLMs struggled specifically with hard items, tending to misjudge them. The current advanced GPT-5.4 stood out for tending to underestimate item difficulty, rating items as easier than they were. When the researchers reduced item embeddings to lower dimensions and visualized them, embeddings from different difficulty levels came out mixed together rather than separated, which the authors say indicates that the semantic content of an item's text alone is likely insufficient to predict its difficulty.

The authors conclude that because the data shows LLMs do not reliably understand item difficulty, and because they seem to treat more items as easy as their own underlying capabilities increase, caution should be exercised when using LLMs to generate test items aimed at a specific target difficulty.

Key facts

  • Zero-shot GPT-4.1 at temperature 0 was the best-performing LLM at predicting item difficulty, reaching a quadratic weighted kappa (QWK) of 0.578.
  • ConvBERT, an encoder-only model, outperformed every LLM tested and the best feature-based machine learning model, scoring a QWK of 0.625.
  • All LLMs tested struggled to correctly label hard items, and GPT-5.4 in particular tended to underestimate item difficulty.
  • Dimension-reduced item embeddings from different difficulty levels overlapped instead of separating, suggesting an item's text content alone is not enough to predict how hard it is.
  • The study's authors say caution is needed when using LLMs to generate test items at a targeted difficulty level, since more capable models appeared to treat more items as easy.

Why it matters

Item difficulty prediction sits at the core of both formative assessment and high-stakes standardized testing: get it wrong and a test can be miscalibrated, unfairly hard or unfairly easy for the population it targets. Automated item generation using LLMs is being explored as a way to produce test questions at scale, and this study is a direct check of whether the models understand difficulty well enough to be trusted with that job. The finding that a specialized encoder-only model still beats every LLM tested, and that the most advanced LLM in the study underestimates difficulty, is a concrete data point against assuming general-purpose LLMs are ready to take over this task.

Who it affects

Testing organizations and ed-tech companies building automated item generation pipelines, researchers working on NLP for education and psychometrics, and ultimately test-takers whose exams could be miscalibrated if difficulty prediction is handed to a model that gets it wrong.

How to use it

The study is a signal, not a deployment guide: it names no specific product or price. The practical takeaway is that teams generating test items with LLMs at a targeted difficulty level should treat the model's difficulty labeling with caution, and consider encoder-only models like ConvBERT or feature-based approaches, or human expert review, as a check rather than relying on zero-shot LLM judgment alone.

How solid is it

The comparison covers multiple LLMs, encoder-only models and feature-based machine learning baselines against items from one large-scale Reading and Writing test, scored with quadratic weighted kappa, a standard agreement metric. The abstract does not name the authors, their institution, the exact sample size beyond calling the test "large-scale," the study date, the full list of LLMs tested beyond GPT-4.1 and GPT-5.4, or a specific number for how much GPT-5.4 underestimated difficulty, which limits how far the claims can be checked from the source alone.

Risks and caveats

No author names or institutional affiliation are disclosed in the available text, so provenance and any conflicts of interest cannot be assessed. Sample size and dataset detail beyond "large-scale Reading and Writing test" are not given, and the extent of GPT-5.4's underestimation is described only qualitatively as a tendency, without a number. The results come from one test domain, reading and writing, and may not generalize to item difficulty prediction in other subjects.