Instruction tuning alters confidence, shrinks rationale diversity

Instruction-tuned language models, the versions of base models fine-tuned to follow prompts and answer questions directly, perform well across a range of generation tasks. Prior work has already shown that this fine-tuning step can leave models sounding more confident than warranted, a pattern researchers call verbalized overconfidence. The authors of this paper suspect that in question answering, that overconfidence may be tied to how consistent a model's own supporting rationale is from one attempt to the next. This paper tests that connection directly, asking whether the shift in confidence that instruction tuning produces comes with a matching shift in the lexical diversity of the rationales a model generates to support its answers.

To test this, the researchers compare three matched pairs of base and instruction-tuned models across question-answering benchmarks. Each pair shares the same underlying architecture, with one version left untuned and the other fine-tuned on instructions; the paper does not name the specific model families or benchmarks used.

On confidence, instruction tuning consistently alters how confident the models sound in their answers across all three pairs, even though predictive accuracy changes little. At the same time, likelihood-based calibration, meaning how well the model's internal probability estimates track whether it is actually correct, gets worse rather than better.

On rationale diversity, the effect is not uniform. Cross-rationale diversity, how much the different justifications a model offers for an answer vary from each other, consistently decreases after instruction tuning. Surface-level lexical diversity within a single rationale behaves differently: it moves in both directions and by different amounts depending on the model and the benchmark, rather than shifting in one consistent way.

Both patterns hold up after the researchers control for which answer the model selected and for how long its rationale runs, which the authors say confirms that confidence and rationale diversity are two distinct effects of instruction tuning rather than two readings of the same underlying change. The paper does not offer an explanation or a fix for why instruction tuning produces this pattern; it documents the pattern itself.

Key facts

The study compares three matched pairs of base and instruction-tuned language models on question-answering benchmarks.
Instruction tuning consistently alters how confident the models sound in their answers, even though predictive accuracy barely changes.
Likelihood-based calibration, how well a model's confidence matches whether it is actually correct, gets worse under instruction tuning.
Cross-rationale diversity, how much a model's justifications for an answer vary from one another, consistently decreases after instruction tuning.
Surface-level lexical diversity within a single rationale moves inconsistently, up or down depending on the model and benchmark, and both effects survive controls for answer choice and rationale length.

Why it matters

Instruction tuning is the fine-tuning step that turns a raw base model into an assistant that follows prompts and answers directly, and it is standard practice before a model ships. Prior work has already shown that this step can leave models sounding more confident than they should, a pattern researchers call verbalized overconfidence. This paper asks a sharper question: whether that shift in confidence comes with a matching shift in how varied the model's own explanations for its answers are. That matters for anyone who treats a model's stated confidence, or the explanation it gives, as a signal of whether the answer can be trusted.

Who it affects

Anyone who fine-tunes base models into instruction-following assistants, and anyone building products such as question-answering tools or chat assistants that show a model's confidence or its explanation to end users. Researchers working on calibration and on interpreting model outputs are a direct audience too, since the paper treats confidence and rationale diversity as two separate measurements rather than one.

How to use it

The paper's own findings double as a caution: a model's stated confidence should not be read as a stand-in for whether it is actually right. Confidence shifts under instruction tuning even when accuracy barely moves, and calibration between confidence and correctness gets worse rather than better. On the explanation side, a narrower spread of rationales for the same answer is not evidence that the answer itself is more correct or more consistent; it may simply mean the model is repeating a narrower set of phrasings. Anyone building on instruction-tuned models should treat confidence scores and rationale variety as separate signals to check, not as proxies for accuracy.

How solid is it

The paper tests three matched base and instruction-tuned model pairs on question-answering benchmarks and reports that the confidence and cross-rationale diversity effects hold consistently across all three, and that both survive after controlling for which answer was chosen and how long the rationale runs. That is a reasonable robustness check for a study this size. Against that, the abstract gives no numeric values for the accuracy, confidence or calibration changes it describes, only qualitative language such as "consistently alters" and "limited changes," so outside readers cannot judge the size of the effects from the text alone. It also does not name the specific model families or benchmarks used. On Hugging Face's papers board the piece has drawn modest attention so far: 7 points and 2 comments.

Risks and caveats

Three model pairs is a small sample, even though the pattern holds across all of them, and without the model or benchmark names, it is hard to know how far the finding generalizes. The paper documents the effects on confidence and rationale diversity but proposes no mechanism or fix for either one; there is no remedy on offer here, only an observed pattern. Readers should also not assume that instruction tuning makes confidence rise in every case: the text states that the change is consistent, not which direction it moves in, for each of the three pairs tested.

“These differences persist after controlling for answer selection and rationale length, confirming that confidence and rationale diversity capture distinct effects of instruction tuning.”

— the paper's abstract