New diagnostic ladder separates decision-rule and readout gaps in speech models

Speech language models are usually graded on paralinguistic tasks, such as recognizing emotion from audio, by whether their prompted answer is correct. A new paper argues that this single accuracy number hides several distinct failure points along the path from audio to answer, and introduces a generation-aligned diagnostic ladder to separate them. The method compares four things computed at the same answer token: the answer the model actually emits, the logits over the answer options, an affine readout built from those logits, and a linear readout of the model's hidden state. Looking at the successive differences between these four measurements isolates an endpoint gap, a decision-rule gap, and a readout-coverage gap.

The authors ran this ladder across five speech language model systems and two emotion corpora, neither of which is named in the abstract. Decoding directly from the hidden state exceeded the model's own generated answer accuracy by 27.8 accuracy points on average, and both the decision-rule gap and the readout-coverage gap were positive in all ten system-corpus conditions tested. In other words, the information needed for a correct answer was present in the model's internal state far more often than the model actually produced it.

The authors also tested a label-free logit correction, a fix that needs no labeled data, and found it improved generated accuracy in every one of the ten conditions, which they take as evidence that part of the decision-rule gap is fixable rather than a hard ceiling. In a separate rank-matched analysis, they show that emotion-relevant information sitting outside a model's native readout generalizes to speakers the model was not trained on and holds up even after controlling for measured acoustic descriptors. Yet when the authors replace the specific readout-external directions they identified, the model's emitted answers usually barely change. The overall picture the authors draw is that information availability and a model's behavioral use of that information are two different things, and their ladder can localize where performance is actually lost, in the decision rule or in the readout from state to answer, rather than lumping it all into one accuracy score.

Key facts

The diagnostic ladder compares four values at the same answer token, the emitted answer, the option logits, an affine readout of those logits, and a linear readout of the hidden state, to separate endpoint, decision-rule, and readout-coverage gaps.
Across five speech language model systems and two emotion corpora, decoding the hidden state exceeds the model's own generated-answer accuracy by 27.8 accuracy points on average.
Both the decision-rule gap and the readout-coverage gap are positive in all ten system-corpus conditions tested.
A label-free logit correction, requiring no labeled data, improves generated accuracy in every one of the ten conditions.
Emotion information sitting outside a model's native readout generalizes to held-out speakers and survives controls for acoustic descriptors, but replacing those readout-external directions usually leaves emitted answers largely unchanged.

Why it matters

Judging speech language models purely by whether their prompted answer is right conflates several different things that can go wrong between audio input and spoken answer. This paper's diagnostic ladder shows that a large share of what looks like an accuracy problem is actually a decision-rule or readout problem: the relevant information can be sitting inside the model already, just not reaching the answer the model gives. That distinction matters for anyone trying to improve or fairly compare these systems, since fixing a readout problem is a different engineering task from fixing what the model actually perceives.

Who it affects

Researchers who build or evaluate speech language models on paralinguistic tasks like emotion recognition, and anyone who uses prompted-answer accuracy as a proxy for what a speech model does or does not understand from audio.

How to use it

The ladder is applied as a diagnostic overlay on existing systems, comparing the emitted answer, the option logits, an affine logit readout, and a hidden-state linear readout at the same answer token; the resulting gaps show where accuracy is being lost. The paper also demonstrates a label-free logit correction, a lightweight fix that needs no additional labeled data and improved generated accuracy in all ten tested conditions, indicating that some of the decision-rule gap can be closed without retraining.

How solid is it

The pattern held consistently: state decoding beat generation in all ten system-corpus combinations, and both the decision-rule and readout-coverage gaps were positive in every one of them, across five different systems and two different emotion corpora. The rank-matched analysis additionally controlled for measured acoustic descriptors and tested generalization to speakers held out of training, which strengthens the claim that the extra information found outside the native readout is real rather than an artifact of the test set.

Risks and caveats

The abstract names neither the five evaluated systems nor the two emotion corpora, and gives no authors or institutions, so it is not possible from this text alone to judge how representative the tested systems are or where the underlying data comes from. The paper also does not report the decision-rule and readout-coverage gaps separately, only that both are positive in all ten conditions, so the relative size of each failure mode is unclear. Availability is not the same as use: the authors note that replacing the readout-external directions they identified usually, not always, has little effect on the model's emitted answers, meaning that giving a model access to information does not reliably change its behavior.

“state decoding exceeds generation by 27.8 accuracy points on average, and both the decision-rule and readout-coverage gaps are positive in all ten conditions”

— the paper's abstract