Glonce

XConf estimates LLM confidence from the model's own past episodes

researchThe Glonce editorsHugging Face🔥59Glonce rating59Impact70Novelty78Relevance100Surprise55Players25Credibility80
XConf estimates LLM confidence from the model's own past episodes

Researchers propose XConf, short for eXperiential Confidence, a way to estimate how confident a language model should be in its own answer. Existing confidence estimators, the authors argue, share one design flaw: they only read the current inference process, whether by introspecting on it, scoring its token probabilities, or resampling it. The authors argue that the current inference alone is not a sufficient basis for confidence. XConf instead pairs the current inference with a stored record of the model's own past episodes: each episode holds the task, the model's reflection on it, its stated confidence at the time, the actual outcome, and a lesson written once that outcome was graded. Given a new task, XConf's Recall stage retrieves past episodes on similar tasks that were met with a similar stated confidence and reads off their historical success rate. Its Reflect stage then shows the model this record, has it name its recurring failure mode, and has it restate a confidence informed by that track record. The method is format-general, needs no access to the model's logits and no weight updates, and requires generating only one answer. Tested across nine benchmarks spanning reasoning, coding, multimodal QA, and interactive agent tasks, on four models drawn from three model families, XConf beat or matched ten-sample self-consistency on discrimination (AUROC) in 23 of 24 comparisons, with much lower calibration error, while costing a tenth of the generation budget that ten-sample self-consistency needs. Used for selective prediction, where the system abstains on its least-confident cases, abstaining on the 10% least-confident episodes raised the delivered success rate by up to 8.7 points on agent tasks. The authors present experiential confidence estimation, learning to judge one's own reliability from a graded history rather than from the current answer alone, as a new paradigm for general-purpose confidence estimation.

Key facts

  • XConf estimates a language model's confidence by combining the current inference with a stored record of the model's own past graded episodes, rather than reading only the current inference as prior methods do.
  • It needs no access to the model's token probabilities and no weight updates, and requires generating just one answer, a tenth of the generation cost of ten-sample self-consistency.
  • Across nine benchmarks covering reasoning, coding, multimodal QA, and agent tasks, on four models from three model families, XConf beat or matched ten-sample self-consistency on discrimination (AUROC) in 23 of 24 comparisons, with much lower calibration error.
  • Abstaining on the 10% least-confident episodes raised the delivered success rate on agent tasks by up to 8.7 points.
  • The paper names neither the specific benchmarks nor the four models and three families tested, nor any institution behind the work.

Why it matters

How confident a model is in its own answer decides what gets shipped automatically, what gets escalated to a human, and what gets retried. The authors argue that every prior confidence estimator shares the same blind spot: it only reads the current inference, whether by introspecting on the model, scoring its token probabilities, or resampling the same question. XConf's premise is that the current answer alone is not enough evidence; a model's own history of being right or wrong on similar tasks is additional signal that existing methods throw away.

Who it affects

Anyone building or operating systems that need to know when to trust a language model's output rather than treat every answer as equally reliable, across reasoning, coding, multimodal question answering, and interactive agents. It is most directly relevant to teams building selective-prediction pipelines, where a system should abstain or hand off rather than act on a low-confidence result.

How to use it

XConf is described as format-general: it does not need access to the model's internal token probabilities and does not require any weight updates. It costs only one additional answer generation per query, which the authors put at a tenth of the generation cost of ten-sample self-consistency, the comparison method. The source gives no release date, no code or model availability, and no institution or funding source for the work.

How solid is it

The method was tested across nine benchmarks spanning reasoning, coding, multimodal QA, and interactive agent tasks, on four models drawn from three model families. Against ten-sample self-consistency, XConf beat or matched it on discrimination, measured by AUROC, in 23 of the 24 comparisons run, while also producing a much lower calibration error, at a tenth of the generation cost. In a selective-prediction test, having the system abstain on the 10% of episodes it was least confident about raised the delivered success rate on agent tasks by up to 8.7 points.

Risks and caveats

The source names neither the nine benchmarks nor the four models and three model families the method was tested on, and it names no comparison baseline other than ten-sample self-consistency, so how XConf holds up against other confidence-estimation approaches is untested here. No institution, affiliation, or funding source is given, and there is no stated release date or indication that code or models will be made available. The approach also depends on having a record of the model's own graded past episodes to draw on, and the source does not say how that record gets built up for a system newly put into use.

“We argue that the current inference is not a sufficient basis for confidence.”

— XConf paper