Blog

Why a decision model answers with probabilities, not text

How Ardana's choice, score and noul questions turn a support ticket into a probability per option, and how an agent thresholds that instead of parsing text.

A paper ticket rides a conveyor toward three coloured doors while glass tubes above the doors show how full each option is.

Parsing a decision out of generated text is the wrong contract

Every agent I have looked at makes a decision the same way. It builds a prompt, asks a chat model to answer “billing, technical or sales”, and then parses the reply. Sometimes the reply is one word. Sometimes it is a sentence with the word somewhere inside it, or a JSON object with a trailing comment, and the code retries until the regex matches. TypeSafe describes it exactly: “coercing a text-generation system into outputting structured decisions, then parsing the results back into something your code can depend on”. I wrote this code more than once and it never felt right.

The confidence is the worst part. The usual fix is to ask the model to also write a number from 0 to 1, and then the agent thresholds on a number that was typed, not measured. Tian et al. (2023) found that RLHF chat models “produce conditional probabilities that are very poorly calibrated”, and that for ChatGPT, GPT-4 and Claude the verbalized confidence is typically better calibrated than their own token probabilities. So the raw probabilities of a chat model are not something you can trust either. Both halves of the contract are broken.

A decision model should not write text at all. It reads a state and typed questions and returns a calibrated probability for every question, so there is nothing to parse and nothing to retry. A probability is what an agent can threshold, sort and log. Text is what a human reads.

Routing a support ticket from the command line

Take the ticket from our README. A customer writes: “My card was charged twice for order A-104. Please refund the duplicate.” An agent needs two things from it: which department owns it, and whether the customer is asking for money back. With ardana that is one command, the state as plain text and the two questions as typed flags.

ardana run decider "My card was charged twice for order A-104. Please refund the duplicate." --choice "Which department should handle this?" billing technical sales --noul "Does the customer ask for a refund?"

--choice takes the question and then the options. --noul takes a yes-or-no question. There is also --score QUESTION LEVEL... for 2 to 10 ordered levels, lowest first, and a --request FILE flag; the state itself can be text or JSON. The model behind decider here is decider:2b: 1.9B parameters on Qwen/Qwen3.5-2B-Base, version v11, released 2026-09-24, default quant q4_k_m. The first run downloads it, 1.3 GB, the file decider-2b-v11-Q4_K_M.gguf. After that the README puts this answer at about a tenth of a second on an M1 MacBook.

This is the README’s sample output:

Which department should handle this?
  * billing    93.5%
    technical   0.5%
    sales       6.1%
    confidence 0.90, p max 93.5%, certainty 0.76

Does the customer ask for a refund?
    probability of yes  0.93

Each question comes back as a title, exactly as you typed it, because ardana run names inline questions q1, q2 in flag order and puts your text in instructions. Under the choice every option gets a row, probability as a percent with one decimal, names padded to the widest, and the pick marked with *. The last line of a choice is confidence 0.90, p max 93.5%, certainty 0.76: confidence with two decimals, p max the pick’s own percent with one decimal. The noul is one row, the probability of yes, with no figures line under it. Blocks keep the order of the request, one blank line apart. Want the raw API response instead? Pass --json. Pass --verbose and you also get the model name, token counts and timings.

What happens inside: a readout, not a generation

The model never writes a token. Each question becomes one prompt row, and the row ends in an open answer slot:

Context:
<state>

Question: <q>
Options:
(A) <opt>
(B) <opt>
Answer: (

That final ( is the answer slot. At that position the model has a logit for every token in its vocabulary, and we read only the ones at the option labels, A, B and so on. Everything else is ignored. Those few logits go through softmax(logits / T), where T is the model’s temperature for the answer type. For decider:2b the profile sets 1.164 for choice, 1.624 for noul and 1.124 for score; a model shipped without a decider_config.json gets 1.0. A noul is the same row with two options, no then yes, and the answer is the probability at yes.

Why read it this way? Because multiple choice is where calibration has been measured. Kadavath et al. showed in 2022 that larger models are well-calibrated on diverse multiple choice and true/false questions “when they are provided in the right format”. That is the bet behind the row above: ask a multiple choice question, read the answer at one slot, and keep the format fixed so every readout is comparable. We built the engine around reading that one slot.

Each row is decoded alone. The planner makes one row per question, so the ticket request above is two readouts, usage.input_tokens counts the shared state once, and output_tokens is always 0. We do not pack rows into one batch, because packing moved Q4_K_M probabilities by up to 0.16. The limits are checked before any decode: 1024 rows, 36,864 tokens per row, 1,048,576 tokens per request. Too much is a 413 request_too_large, never a silent truncation.

A decision is a readout of one position’s logits, not a generation. There are no output tokens, so there is nothing to parse.

Choice, score and noul are three readouts of one mechanism

Once the readout exists, a question type is only a way of naming the options and reading the result. I built three of them, and they share one slot.

choice is the plain one. You give it 2 to 255 named options and it returns the option picked plus a probability for every option. That is the department question from the ticket. The answer carries choice, confidence, probabilities, x_p_max and x_certainty.

noul is a choice in disguise. The question is rendered as two options, no then yes, and you can describe each side with criteria.false and criteria.true. The answer is one number, noul, the probability of yes. Nothing else is needed, since the probability of no is one minus it.

score is the one that bends the mechanism. You give it 2 to 10 ordered levels, lowest first. The result is the expected level, sum(i * p_i) over levels 0..n-1, rounded to 2 places, with a legend that maps "0", "1" and so on back to each level’s description. On decider:0.8b and decider:2b I do not ask one row with all the levels in it. Each level gets its own yes/no row, Proposed answer: <level> followed by Does the proposed answer fit?, and the yes probabilities are normalised into one distribution. The raw fit of each level comes back as x_level_fit and their sum as x_fit_mass, so you can see what the normalisation did.

The three answer shapes, side by side:

choice  { choice, confidence, probabilities, x_p_max, x_certainty }
score   { score, confidence, legend, probabilities, x_p_max, x_certainty, x_level_fit, x_fit_mass }
noul    { noul }

Answers keep the order of the questions in the request. Route with a choice, check the refund with a noul, grade urgency with a score, all in one call.

Confidence is computed from the distribution, not typed by the model

Look at the department answer again. The model gave billing 0.9346, sales 0.0605 and technical 0.0049. The confidence is not in there. We compute it.

"department": {
  "type": "choice",
  "choice": "billing",
  "confidence": 0.9019,
  "probabilities": { "billing": 0.9346, "technical": 0.0049, "sales": 0.0605 },
  "x_p_max": 0.9346,
  "x_certainty": 0.7643
}

confidence is TypeSafe’s formula, (n * p_max - 1) / (n - 1), clipped to 0..1. With three options that is (3 * 0.9346 - 1) / 2 = 0.9019: 1 when all the probability sits on one option, 0 when it is spread evenly. x_p_max is the largest probability. x_certainty is one minus the normalised entropy, 0.7643 here, so it weighs the whole distribution. A score gets its own confidence, from the expected distance to the most likely level.

Why does this matter? A number the model typed is a token it predicted. A number we computed is arithmetic. TypeSafe’s tiers: high, act automatically; medium, confirm or flag for review; low, route to a human, request clarification, or fall back to a different system. TypeSafe names the tiers, not the cutoffs; those are ours to pick.

Confidence is arithmetic over the probabilities, not a claim the model makes about itself.

Wiring the decision into an agent

An agent talks to the server over HTTP: the same ticket, each question keyed by a name you choose.

curl http://127.0.0.1:8000/v1/systemone -H 'content-type: application/json' -d '{
  "model": "decider:2b",
  "state": "My card was charged twice for order A-104. Please refund the duplicate.",
  "questions": {
    "department": {
      "type": "choice",
      "instructions": "Which department should handle this?",
      "criteria": ["billing", "technical", "sales"]
    },
    "refund": { "type": "noul", "instructions": "Does the customer ask for a refund?" }
  }
}'

Answers come back keyed the same, department with its probabilities and "confidence": 0.9019, refund as one float, 0.9261. The agent code is a few comparisons, with no regex and no retry loop. High confidence, assign to billing. Middle, assign and flag for a human. Low, route to a person.

In Python it is the TypeSafe client pointed at the local server by one environment variable. The SDK insists on a key; a server started without --api-key ignores it.

pip install typesafe-sdk
TYPESAFE_BASE_URL=http://127.0.0.1:8000 TYPESAFE_API_KEY=any python decide.py
from typesafe_sdk import TypeSafeClient

with TypeSafeClient() as client:
    response = client.system_one(
        model="decider:2b",
        state="My card was charged twice for order A-104. Please refund the duplicate.",
        questions={"refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"}},
    )
print(response.answers["refund"].noul)

One caveat: TypeSafeClient() keeps the SDK default timeout of 10.0 seconds. A first request that still has to pull the model may not fit in that, so our playground snippet passes timeout=120.0.

Limits are checked before any decode: 1024 rows, 36,864 tokens per row, 1,048,576 per request. Over them is a 413 request_too_large, never a truncated state. The agent fails loudly or it decides.

Run the curl, or ardana run with --json, and pick a model at ardana.ai/models/. Then tell me how you threshold it.