Glonce

Paper finds .916 overall F1 hid scorer errors in LLM follow-up prompt tests

researchThe Glonce editorsarXiv🔥57Glonce rating57Impact45Novelty55Relevance90Surprise35Players10Credibility80

Many LLM studies compare how models answer different follow-up prompts. One prompt might neutrally ask a model to reconsider; another insists that an invented film really exists. The gap between the two is read as the effect of conversational pressure. An automatic scorer decides whether each response makes an unsupported factual claim, and researchers commonly check that scorer with a single overall accuracy or F1 score.

The paper argues that this check is not enough. High overall performance can hide prompt-specific mistakes: if the scorer makes more errors after one prompt, the measured difference can be too large, too small, or point in the wrong direction. An audit made up mostly of difficult responses can also misrepresent how the scorer performs across the full study. The authors derive how such errors change a prompt comparison, then test the problem with six models, 180 invented-entity questions and six prompt conditions.

In a balanced audit of 288 responses, the scorer's overall F1 was .916. Yet, depending on the prompt, the scorer flagged between 2.6% and 33.3% of responses that the primary LLM rater judged not to contain an unsupported claim. The authors describe these prompt-specific estimates as uncertain.

A second audit had the scorer and two separate LLM raters evaluate the same 300 model-question cases under each prompt. The scorer showed an apparent 9.3-point increase from asserting certainty rather than neutrally asking the model to reconsider. Under both LLM raters, that increase fell to 2.3 points. The interval for the primary rater included zero, so it did not establish an increase at all. Two other prompt comparisons kept their direction under both raters. An attempted prompt-specific correction remained too uncertain for a firm conclusion.

The recommendation: validate the exact difference you plan to report, not just the scorer as a whole. Audit both prompts on matching cases, and carry scorer and rater uncertainty into the final result.

Key facts

  • In a balanced audit of 288 responses, the scorer's overall F1 was .916, yet depending on the prompt it flagged 2.6%-33.3% of responses that the primary LLM rater judged free of unsupported claims (estimates uncertain).
  • An apparent 9.3-point increase from asserting certainty versus neutrally asking the model to reconsider fell to 2.3 points under both of two other LLM raters, across the same 300 model-question cases per prompt.
  • The primary-rater interval for the 2.3-point increase included zero, so it did not establish an increase.
  • Two other prompt comparisons kept their direction under both raters; an attempted prompt-specific correction stayed too uncertain for a firm conclusion.
  • The study used six models, 180 invented-entity questions and six prompt conditions; the authors advise auditing both prompts on matching cases and reporting scorer and rater uncertainty.

Why it matters

Comparisons between follow-up prompts are how many studies measure the effect of conversational pressure on LLMs. If the scorer behind them errs differently after different prompts, the headline effect can be too large, too small or even reversed. Here, a scorer-based 9.3-point increase shrank to 2.3 points under two other raters, while the scorer's overall F1 of .916 looked reassuring.

Who it affects

Researchers who evaluate LLMs by prompting models with follow-ups and using an automatic scorer to label responses. The abstract describes validating a scorer with one overall accuracy or F1 score as common practice, so any such comparison is in scope. Readers of those studies should also know what was audited.

How to use it

The authors' advice is practical. Validate the exact difference you plan to report, not just the scorer overall. Audit both prompts on matching cases. Include scorer and rater uncertainty in the final result. Avoid audits made up mostly of difficult responses, which can misrepresent performance in the full study.

How solid is it

This is an arXiv preprint, and this account rests on its abstract. The evidence is a specific design: six models, 180 invented-entity questions, six prompt conditions, a balanced audit of 288 responses and a second audit of 300 model-question cases per prompt. The six models and six prompt conditions are not named, and the scorer and the LLM raters are not named either. Only one comparison, certainty assertion against neutral reconsideration, has its effect size given.

Risks and caveats

The prompt-specific error estimates (2.6%-33.3%) were themselves uncertain, and the abstract does not say which prompts sit at each end of that range. The primary-rater interval for the 2.3-point increase included zero, so the result is no proof of a real increase either. The attempted prompt-specific correction was too uncertain to support a firm conclusion. The raters are other LLMs, and the 9.3-to-2.3 shift concerns one comparison; the other two kept their direction.

“high overall performance can hide prompt-specific mistakes: if the scorer makes more errors after one prompt, the measured difference can be too large, too small, or point in the wrong direction”

— arXiv abstract, paper 2610.31678