Cascade routing with a local decision model's confidence
How to put ardana serve and decider:2b in front of a chat model, which returned field to threshold, and how to check the threshold on your own labelled tickets.
The threshold is the part everyone hand-waves
Every agent I have wired up ends with the same cost shape. Most tickets are easy. A few are not. Sending all of them to the big chat model means paying the big price for the easy ones, and I did exactly that for too long. The fix is a cascade: a cheap model answers first, and only the tickets it is unsure about go up to the expensive one.
Walker Miller’s guide on dev.to puts it as “Done right, this cuts the token bill 40-70% with no quality loss.” His arithmetic: the expensive model costs 15x the cheap one, 70% of the traffic is genuinely easy. Everything through the expensive model costs N × 15. The cascade costs 0.7N × 1 + 0.3N × 16 = 5.5N, because the 30% that escalate pay for both calls. A 63% reduction. He also warns that “A cascade with a bad escalation trigger is worse than not having one”.
The whole saving rests on “unsure about”, and that is a threshold on a number. Strands released a 2 billion parameter decision model on October 1 that returns “a high-quality reliability score” with each decision. Good. But the UCCI paper from May says “most deployed routers use uncalibrated confidence scores and require per-workload threshold tuning”. So which number do you threshold, at what value, for your tickets? That is the question here, with a local server you can run today.
The cheap stage is ardana serve and one request
The cheap model in my cascade is decider:2b. It is 1.9B parameters on a Qwen/Qwen3.5-2B-Base base, 1.3 GB in the default q4_k_m quant, Apache-2.0, and it is the library default. The first run downloads it. After that the whole cheap stage is one process.
ardana serve --port 8000 --default-model decider:2b--port defaults to 8000 and --host to 127.0.0.1. There is --api-key, and without one the API is open, which is fine on localhost. --keep-alive is 5m, --max-loaded-models is 1, and --max-queued-rows is 4096, and anything bigger gets a 503. The endpoints are POST /v1/systemone, GET /v1/models, GET /health, and a playground at /. The request is the one from the README: a state, and typed questions about it.
curl http://127.0.0.1:8000/v1/systemone
-H 'content-type: application/json'
-d '{
"model": "decider:2b",
"state": "My card was charged twice for order A-104. Please refund the duplicate.",
"questions": {
"department": {
"type": "choice",
"instructions": "Which department should handle this?",
"criteria": ["billing", "technical", "sales"]
},
"refund": {
"type": "noul",
"instructions": "Does the customer ask for a refund?"
}
}
}'This is what my machine returned on 2026-10-09, ardana 0.2.2 on a Ryzen 7 3800X, CPU only. It is a real response, not a sample.
{
"model": "decider-2b-v11",
"answers": {
"department": {
"type": "choice",
"choice": "billing",
"confidence": 0.8973,
"probabilities": { "billing": 0.9316, "technical": 0.0047, "sales": 0.0637 },
"x_p_max": 0.9316,
"x_certainty": 0.7571
},
"refund": { "type": "noul", "noul": 0.9255 }
},
"usage": { "input_tokens": 72, "output_tokens": 0 }
}The first request took 3.33 s with the model load inside it, the second 1.40 s with an identical body. The README says about a tenth of a second on an M1 MacBook, and my CPU-only box is not that. Look at the choice answer. One question gave me confidence, x_p_max, x_certainty, and the full probabilities. The noul gave me one number, 0.9255 for yes. Which of these to put against a threshold is the next section.
In Python I use the typesafe-sdk client, because the API is compatible with TypeSafe clients. Two environment variables point it at the local server. The key has to be non-empty, any string will do, because an empty one raises “No API key was provided.” before a request is made.
pip install typesafe-sdk
TYPESAFE_BASE_URL=http://127.0.0.1:8000 TYPESAFE_API_KEY=any python decide.pyfrom typesafe_sdk import TypeSafeClient
with TypeSafeClient() as client:
response = client.system_one(
model="decider:2b",
state="My card was charged twice for order A-104. Please refund the duplicate.",
questions={
"department": {
"type": "choice",
"instructions": "Which department should handle this?",
"criteria": ["billing", "technical", "sales"],
},
"refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"},
},
)
print(response.answers["department"])
print(response.raw_http_response.json()["answers"]["department"])One catch cost me a few minutes. The SDK’s answer models are pydantic with extra="ignore", so the typed answer on this run was ChoiceAnswer(type='choice', choice='billing', confidence=0.8973, probabilities={'billing': 0.9316, 'technical': 0.0047, 'sales': 0.0637}). x_p_max and x_certainty are dropped. The raw body still has them: response.raw_http_response.json()["answers"]["department"] held 'x_p_max': 0.9316, 'x_certainty': 0.7571 on the same call. If you would rather have them typed, system_one(..., response_model=MyModel) takes your own pydantic model for the body.
Which number to threshold
The response above carries four numbers for the department question, and they do not say the same thing. The formulas sit in the server code. For a choice question with n options, 2 to 255 of them:
confidence = (n * p_max - 1) / (n - 1)
x_p_max = the largest probability
x_certainty = max(0, 1 - H / ln(n)), where H = -sum(x ln x) over x > 0Internally the server computes on normalised probabilities and clips confidence to [0, 1]. A score question, 2 to 10 levels, has its own confidence: 1 minus the expected distance from the most likely level over the mean distance of the levels from the middle of the scale. A noul returns only noul, the probability of yes.
Does the server really do this? I checked my response by hand:
(3 * 0.9316 - 1) / 2 = 0.8974 confidence 0.8973
1 - H / ln 3 on the three probabilities = 0.757 x_certainty 0.7571The gap is rounding: the server rounds to 4 places after computing on unrounded values. Both numbers are reproducible from the probabilities alone.
So which one goes in the if? TypeSafe’s docs say confidence “collapses that shape into a single number from 0 to 1”, with a catch right after: “Only the top probability counts, so (0.6, 0.3, 0.1) and (0.6, 0.2, 0.2) both have confidence 0.4.” x_certainty reads the whole distribution, so those two differ there. The docs also list p_max, thresholded per question, and the ratio p_max / p_second.
confidence says whether the winner won by enough. x_certainty says whether the rest of the mass is one real rival or just noise. For a cascade I threshold both.
A noul has no confidence at all, and TypeSafe suggests |2p - 1| for it. Each type gets its own signal, so the thresholds in the next section are per question.
What threshold, then? TypeSafe’s guidance is flat: “Start with conservative thresholds, test with your own data, and adjust as you observe results.” Their routing pattern gives two examples, theirs and not mine: “Below 0.6 confidence on any action, route to a human”, and approving a transfer “requires very high confidence (>0.85), otherwise the system should ask the user to confirm.” I treat both as starting points for a loop, nothing more.
The loop: answer, threshold, escalate, log
Here is the whole cascade in one Python file. The SDK finds the local server through TYPESAFE_BASE_URL, and the key must be non-empty, because the client raises on an empty one before it sends anything, so any will do. The typed answer gives me choice, confidence and probabilities. The x_certainty I take from response.raw_http_response.json(), because the SDK’s pydantic models drop extra keys. Either one under its floor calls the chat model. ask_chat_model is your chat client.
# pip install typesafe-sdk
# TYPESAFE_BASE_URL=http://127.0.0.1:8000 TYPESAFE_API_KEY=any python cascade.py
import json
from typesafe_sdk import TypeSafeClient
QUESTIONS = {
"department": {
"type": "choice",
"instructions": "Which department should handle this?",
"criteria": ["billing", "technical", "sales"],
},
"refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"},
}
# 0.85 and 0.6 are TypeSafe's two examples, 0.3 is my catch-net floor.
# All three get replaced by the check in the next section.
THRESHOLDS = {
"department": {"confidence": 0.85, "x_certainty": 0.6},
"refund": 0.3,
}
def route(ticket_id, ticket, client, log):
response = client.system_one(model="decider:2b", state=ticket, questions=QUESTIONS)
department = response.answers["department"]
x_certainty = response.raw_http_response.json()["answers"]["department"]["x_certainty"]
p_refund = response.answers["refund"].noul
floors = THRESHOLDS["department"]
row = {
"id": ticket_id,
"department": department.choice,
"confidence": department.confidence,
"x_certainty": x_certainty,
"probabilities": department.probabilities,
"refund": p_refund,
"chat": None,
}
if department.confidence < floors["confidence"] or x_certainty < floors["x_certainty"]:
row["chat"] = ask_chat_model(ticket, "Which department should handle this?")
row["refund_queue"] = p_refund >= THRESHOLDS["refund"]
log.write(json.dumps(row) + "\n")
return row
with TypeSafeClient() as client, open("decisions.jsonl", "a") as log:
for ticket_id, ticket in tickets:
route(ticket_id, ticket, client, log)Three things here are deliberate. The thresholds sit in a dict keyed by question, not in one constant at the top. Future AGI measured why: a 0.7 on product-question was right 95 percent of the time and a 0.7 on fraud-report 55 percent, so one global threshold ships a different precision per class. The 0.85 is TypeSafe’s example for approving a transfer, the 0.6 on x_certainty their example for sending anything below it to a human. Both are placeholders until the next section replaces them with numbers from your tickets.
The second is the refund question. A noul is one probability, no confidence field, and here a yes sends the ticket to a human in the refund queue. That makes it a catch-net question. Future AGI’s rule for those: they need high-recall floors, so set them low and let the system over-route to safe paths rather than guess. So no symmetric distance from 0.5 here. Mine starts at 0.3, because a wasted look by a human costs less than a missed refund.
Third, the log. Every row keeps the probabilities, the local answer, and the chat answer when there was one, under a ticket id. The chat model runs only inside the if, so the threshold is the whole saving, or the whole bill. Labels come later, from whoever closes the ticket, and the id joins them to these rows.
Check the threshold on your own tickets
When the Strands Decider 2B post landed on Hacker News, a top-level comment by soltanov was one sentence: “Benchmark calibration does not establish reliability on unfamiliar production inputs.” I agree. The refund ticket above is one ticket. The only calibration that counts is the one you measure on yours.
This is why the loop logs the probabilities. Each ticket has its top probability, its answer, and, once labelled, whether it was right. Pavlovic gives the definition: a model is confidence-calibrated if, for all confidences c, it is correct c proportion of the time. Split the n tickets into M equally spaced bins by top probability and compare, per bin, the fraction correct against the mean confidence:
ECE = Σ_{m=1}^{M} (|B_m|/n) |acc(B_m) − conf(B_m)|Plot the bins, confidence against accuracy, and you have a reliability diagram; the diagonal is perfect calibration. Future AGI runs ten bins per intent on production data, bin index min(int(conf * num_bins), num_bins - 1), because a 0.7 on product-question is right 95 percent of the time and a 0.7 on fraud-report 55 percent. Your threshold per question is the bin where accuracy clears what you need; they use 0.95.
Two caveats from Pavlovic. The ECE moves with the binning, so pick ten bins and keep them. And low ECE does not equal high accuracy: a model that says 0.5 and is right half the time is calibrated and useless. Walker Miller adds the last rule: re-validate on a schedule, and whenever you change the cheap model.
Calibration is something you measure on your tickets, not something a model ships with.
Run the loop on a few hundred of your own tickets, draw the diagram, and tell me where your thresholds landed.