Hugging Face blog explains decision models that return probabilities, not text

Hugging Face's blog post "Decision Models in Hugging Face" starts from a customer support ticket: "I was charged twice for my subscription. I need a refund today." The system has to decide which team handles it, how urgent it is, and whether the customer wants a refund. The usual route is to ask an LLM for a JSON object. The post shows this with a Gemma model (ggml-org/gemma-4-e4b-it-GGUF:Q4_0) run through llama cli, which, after a thinking phase, returns a team ("Billing"), an urgency ("High") and a refund flag (true).

The post then asks whether a model could instead evaluate the available options directly and return a probability distribution over them, without generating any text. Its answer is a decision model. The definition: it takes some input context (the state), evaluates a question with a set of predefined possible answers (primitives), and returns a structured decision along with probabilities. In the post's words, instead of asking a model to generate an answer token by token, you ask it to assign probabilities to the answers the application already understands. The interface is summarised as State + Questions + Allowed Answers → Decisions + Probabilities.

On naming, the post says TypeSafe AI introduced Jev and described it as a System One model, a nod to the split between fast, intuitive System 1 thinking and slower, deliberative System 2. The post says the community has converged on "decision model" because it describes the model's computational role rather than invoking a particular theory of cognition, and it uses that term from then on.

The post gives three reasons not to use LLMs for everything. First, LLMs are not guaranteed to follow the requested schema: even with explicit instructions a model might produce invalid JSON, add unexpected keys, or answer outside the allowed choices. Second, constrained decoding with libraries such as Outlines and XGrammar, which restrict the tokens allowed at each step, enforces the supported structural constraints but adds work. Third, when an application only needs to pick among predefined answers, autoregressive decoding can be unnecessary work.

It then compares decision models with two close relatives. A traditional classifier, for example one routing tickets to Billing, Technical support and Sales, learns a mapping from input to fixed labels, with a classification head giving one logit per class and softmax turning them into probabilities. Its limit is that the classes are baked into the head. Adding a fourth department such as Legal may mean modifying the head, collecting training data, and retraining or adapting the model. Zero-shot classification removes part of that limit by supplying candidate labels at inference time. The post runs facebook/bart-large-mnli on "My payment was deducted twice." With labels billing, technical support and sales, billing scores 0.8467, then sales 0.0962 and technical support 0.0571. Adding "legal" as a fourth label moves billing to 0.8186, with sales at 0.0930, technical support at 0.0552 and legal at 0.0332. One common implementation uses a natural language inference model, turning each label into a hypothesis and measuring how strongly the input entails it. The post concedes this "already sounds quite similar to a decision model. And in many ways, it is."

What decision models add, according to the post, is that one request can carry several named questions about the same state, each with its own answer type: choice (pick between alternatives), score (rate on an ordered scale) or noul (estimate the probability that a statement is true). The post is explicit that the advantage is not that they invented runtime-defined labels or probabilistic classification. It is that they organise these capabilities into a consistent interface, potentially backed by architectures and training procedures optimised specifically for decisions.

A request has two parts: a state and questions. The state is the information to evaluate: a string, a structured JSON object (the post's example has a customer message, a subscription tier "pro" and a count of two previous tickets), or other modalities for supported omni-modal models.

The code uses pipeline("decision", model="autotrust/GEV-26B-Decide") from transformers. In the choice example the state is "I was charged twice for my order and want a refund." and the question "department" has criteria billing (payments and refunds), technical (bugs and technical issues) and sales (pricing and subscriptions). The output gives choice "billing" with confidence 0.9974 and probabilities of 0.9983 for billing, 0.0010 for technical and 0.0007 for sales. The post notes the remaining probabilities are useful too, because they show how the model spreads its belief among the alternatives.

In the score example, the question "urgency" has levels 0 (low), 1 (medium) and 2 (high). The model returns probabilities of 0.0149, 0.2648 and 0.7203, a confidence of 0.5581 and a score of 1.7054, reported in the prose as 1.7. The score is the expected value of the level index, not the most likely level. The post stresses this does not mean the model picked a fictional level halfway between medium and high; it is the expectation of the probability distribution.

The third primitive, noul, covers a binary question, shown with "Is the customer requesting a refund?" described as a Bernoulli decision. The text of the post available here breaks off at the start of that example, so its output and anything after it are not covered in this retelling.

Key facts

• Hugging Face defines a decision model as one that takes a state plus questions with predefined answers and returns a structured decision with probabilities, rather than generating text token by token.
• The demo uses pipeline("decision", model="autotrust/GEV-26B-Decide") in transformers; in the choice example billing gets 0.9983 against 0.0010 for technical and 0.0007 for sales.
• Three answer types are described: choice, score (the expected value of the level index, 1.7054 for urgency, reported as 1.7) and noul (a yes/no question answered probabilistically).
• The post argues LLM JSON output can break the schema and constrained decoding (Outlines, XGrammar) still adds overhead, while classifiers have labels baked into the head and zero-shot classifiers such as facebook/bart-large-mnli take labels at inference time.
• The post says the advantage is a consistent interface, not new ideas like runtime-defined labels or probabilistic classification, and that such an interface is potentially backed by models optimised for decisions.

Why it matters

Many applications already use LLMs for classification and routing by asking for JSON. The post argues that when the application only needs to choose among answers it already understands, generating text is unnecessary work and a model that scores those answers directly fits better. It frames this as a shared interface (State + Questions + Allowed Answers → Decisions + Probabilities) and says the community has settled on the name "decision model" over "system one model", the term TypeSafe AI used for Jev. The post is careful about novelty: it says the idea is not runtime-defined labels or probabilistic classification, which zero-shot classifiers already offer, but packaging them as one interface, potentially backed by architectures and training built for decisions.

Who it affects

The scenarios in the post are developers building routing, triage and classification features, such as a customer support system that must pick a team, rate urgency and spot refund requests. It speaks to people who currently get such answers from an LLM in JSON, and to those who use fixed-label classifiers or zero-shot classification in transformers.

How to use it

The post shows code against the transformers library: create decide = pipeline("decision", model="autotrust/GEV-26B-Decide") and pass a dictionary with a "state" (a string, structured JSON, or other modalities for supported omni-modal models) and a "questions" mapping. Each question has a type, instructions and criteria. A choice question lists options with short descriptions; a score question lists ordered levels such as ["low", "medium", "high"]; a noul question is a yes/no statement. The output holds, per question, the result, a confidence and the probabilities, plus a legend of level names for score questions. The post does not state a transformers version for the pipeline, and no price or licence is given.

How solid is it

This is an explanatory blog post from Hugging Face with worked examples, not a benchmark. The outputs shown are single examples on short inputs. The post gives no accuracy, latency or cost comparison between decision models and LLMs or classifiers, and nothing about how GEV-26B-Decide was trained or who is behind the autotrust namespace. The available text ends partway through the third (noul) example, so the rest of the post is not covered here. The post does not explain why the choice example's confidence (0.9974) differs from the billing probability (0.9983).

Risks and caveats

The claims about LLM weaknesses are stated in the post as possibilities (a model might produce invalid JSON; decoding can be unnecessary work), with no measurements behind them. The claimed benefit of purpose-built architectures is hedged by the post itself with "potentially". The urgency score of 1.7 is an expectation over levels, not a chosen level, and its confidence in the example is 0.5581, much lower than in the choice example, so readers should not treat a score as a category. The zero-shot example shows how adding a label shifts the other scores (billing fell from 0.8467 to 0.8186), which is part of the contrast the post draws.

“Instead of asking a model to generate an answer token by token, we ask it to assign probabilities to the answers that our application already understands.”

— Hugging Face blog post "Decision Models in Hugging Face"