Glonce

CORD repairs calibrated confidence scores without changing predictions

researchThe Glonce editorsHugging Face🔥64Glonce rating64Impact50Novelty70Relevance95Surprise40Players20Credibility80
CORD repairs calibrated confidence scores without changing predictions

Post-hoc calibration is a standard technique for correcting the confidence scores a trained classifier reports, but on multiclass problems a calibrator can also change which class the model picks as its top-1 prediction, not just its confidence. Standard accuracy metrics only capture the net effect of these prediction changes on whether the classifier is right or wrong; they do not reveal how often the predicted class flips at all. The authors introduce the Top-1 Prediction Change Rate (TPCR) as a metric measuring exactly that frequency.

To address it, the authors propose CORD, Calibrator-Output Repair for Top-1 Decision Preservation, described as the first post-fit adapter to enforce exact prediction preservation by repairing the entire calibrated probability vector, rather than only its top value. Using nothing but the original, uncalibrated output and the calibrated output, CORD determines how much probability mass should be assigned to the original top-1 class, then uses the calibrated conditional distribution to spread the remaining mass across the other classes. The result is a repaired probability vector whose own argmax always recovers the original, pre-calibration prediction. On the calibration split, CORD also coordinates the repaired masses to keep the calibrated outputs' mean mass on the original predictions whenever that is attainable. The adapter does not alter the fitted calibrator or its direct output, does not fit any additional supervised mapping, and needs no hyperparameter tuned by a user or on a validation set.

Across CIFAR-10, CIFAR-100, and ImageNet-1K, CORD attains zero TPCR by construction, meaning the repaired top-1 prediction always matches the original one, and it lowers mean expected calibration error (ECE), negative log-likelihood (NLL), and Brier score relative to the corresponding uncorrected calibrated outputs on every dataset tested. These paired gains hold up under distribution shift and across different calibration-set sizes. The authors frame the result as separating two previously entangled jobs: calibrator fitting no longer has to carry the burden of preserving the original decision, since that is handled afterward, exactly, by CORD's repair step. Code is available at https://github.com/labhai/CORD.

Key facts

  • Post-hoc calibrators can silently flip a classifier's top-1 predicted class while fixing its confidence scores; the authors introduce the Top-1 Prediction Change Rate (TPCR) to measure how often this happens, since accuracy alone only captures the net effect.
  • CORD (Calibrator-Output Repair for Top-1 Decision Preservation) is a post-fit adapter that repairs the full calibrated probability vector so its argmax always matches the original, uncalibrated prediction.
  • CORD attains zero TPCR by construction on every dataset tested, while still lowering mean ECE, NLL, and Brier scores compared to the uncorrected calibrated outputs.
  • The gains hold across CIFAR-10, CIFAR-100, and ImageNet-1K, and persist under distribution shift and across different calibration-set sizes.
  • CORD needs no extra supervised training and no user- or validation-tuned hyperparameters, and does not modify the underlying calibrator; the code is public at https://github.com/labhai/CORD.

Why it matters

Post-hoc calibration is standard practice for making a trained classifier's reported confidence trustworthy, but it has a side effect the field has not measured directly: correcting the probabilities can also change which class the model calls its top choice. The authors argue that accuracy metrics hide this because they only capture the net effect of such changes on correctness, not how often the predicted class itself flips. CORD closes that gap by making prediction preservation an explicit, guaranteed property of the repair step rather than an implicit hope left to the calibrator.

Who it affects

Anyone building or deploying systems that rely on post-hoc calibration of multiclass classifiers, where confidence scores are corrected after training without disturbing the model's actual decision. Researchers working on calibration methods also gain a new metric, TPCR, to report alongside standard measures like ECE, NLL, and Brier score, making prediction changes visible instead of hidden inside accuracy figures.

How to use it

CORD runs after any existing post-hoc calibrator rather than replacing it: it does not alter the fitted calibrator or its direct output, and it fits no additional supervised map of its own. Given only the original, uncalibrated output and the calibrated output, it determines the mass to assign to the original top-1 class and redistributes the remaining mass over the other classes using the calibrated conditional distribution, producing a repaired vector whose argmax always equals the original prediction. No hyperparameter needs tuning by a user or on a validation set, and the code is available at https://github.com/labhai/CORD.

How solid is it

The claims rest on experiments across three standard image-classification benchmarks, CIFAR-10, CIFAR-100, and ImageNet-1K. On every one, CORD attains zero TPCR by construction and lowers mean ECE, NLL, and Brier scores relative to the corresponding uncorrected calibrated outputs. The paper states these paired gains persist under distribution shift and across different calibration-set sizes. The source text does not give specific numeric values for the ECE, NLL, or Brier scores, for either the baseline calibrated outputs or CORD; it states only that CORD's mean values are lower on every dataset tested.

Risks and caveats

The zero-TPCR result holds by construction, meaning CORD is built so its repaired output's top class always matches the original prediction rather than this being an empirical finding that could fail on new data. The source text gives no author names, institutions, affiliations, publication venue, or date, and no release date for the code at the linked GitHub repository, so those details cannot be verified from this article. No specific ECE, NLL, or Brier figures are reported, only that CORD's mean values are lower than the direct outputs' on every dataset tested.