Blog

Choose local decision actions by expected loss

Turn local model probabilities into actions with explicit error costs, derive a binary threshold, and compare multiclass policies on held out data.

A small brown groundhog compares two trays of plain blocks on a tabletop balance.

Choosing an action takes more than the most likely label

I would give a missed refund request and an unnecessary refund tag different costs in a ticket workflow. Both are classification mistakes, but I would not assume they waste the same amount of time. The event I want to estimate is whether the customer asked for a refund. Detecting that request establishes neither eligibility nor permission to send money.

Choosing the most probable label minimizes expected loss when correct classifications cost nothing and every mistake has the same positive cost. For unequal costs, I would score each action across the possible true classes, weighting each loss by its probability. Then I would choose the action with the lowest expected loss. This is the decision rule in Elkan’s cost-sensitive learning paper.

Our cascade workflow deals with escalating between models. Here I am choosing how the application handles a ticket after it has a probability estimate. Ardana’s request schema has no cost matrix or action execution field, so I would keep that policy in application code. The first step is to read a probability for the exact event whose mistakes I priced.

Read the probability for the event you priced

I would start with the documented request and save its JSON body as request.json. It asks which department should handle the ticket and whether the customer requests a refund.

{
	"model": "decider:2b",
	"state": "My card was charged twice for order A-104. Please refund the duplicate.",
	"questions": {
		"department": {
			"type": "choice",
			"instructions": "Which department should handle this?",
			"criteria": ["billing", "technical", "sales"]
		},
		"refund": {
			"type": "noul",
			"instructions": "Does the customer ask for a refund?"
		}
	}
}

Next, pull the model and run the saved request. The second command prints the API response as JSON. This file input is supported in Ardana v0.3.2.

ardana pull decider:2b
ardana run decider:2b --request request.json --json

I can also pass the complete request through stdin. For HTTP, I would start the server in a separate terminal. Then I send the same JSON body to its default local endpoint.

ardana run decider:2b --request - --json < request.json
ardana serve
curl http://127.0.0.1:8000/v1/systemone -H 'content-type: application/json' -d '{"model":"decider:2b","state":"My card was charged twice for order A-104. Please refund the duplicate.","questions":{"department":{"type":"choice","instructions":"Which department should handle this?","criteria":["billing","technical","sales"]},"refund":{"type":"noul","instructions":"Does the customer ask for a refund?"}}}'

For the policy, I would read these response fields. The refund value estimates the probability of a yes answer to the refund-request question. It establishes neither eligibility nor permission to pay. The department map gives a probability for every option.

answers.refund.noul
answers.department.probabilities

Our department policy needs the class probabilities. The chosen label is the most probable option, while confidence is a transformed score, not a class probability. The v0.3.2 implementation rounds probabilities to four decimal places, so their returned sum can differ slightly from one.

I would keep error costs out of the event question and apply them in application code. The request schema has no cost-matrix or action-execution field. In When Policies Change Probabilities, Kudum and colleagues tested four LLM reviewer interfaces on 720 patches: changing the joint cost-and-threshold prompt changed reported risk with evidence held fixed. That preprint’s balanced sample of archived test outcomes does not establish production behavior or Ardana performance. For our refund-request policy, the next step is to price the two possible tagging mistakes.

Derive the binary threshold from two error costs

I start by assigning zero loss to both correct decisions: tagging a real refund request and leaving a message without one untagged. A false positive means I tag a message that contains no refund request. A false negative means I leave a real refund request untagged. I give those errors nonnegative costs, with a positive sum, measured in the same units.

Let p be the probability that the message contains a refund request. Action 1 adds the tag, and action 0 leaves it untagged. I weight each possible error cost by its probability, following Elkan’s binary decision rule. With zero losses for correct decisions, the comparison becomes:

R1=C_FP*(1-p)
R0=C_FN*p
R1<=R0 gives p>=C_FP/(C_FP+C_FN)

For this example, I choose a false-positive cost of 2 and a false-negative cost of 8 in arbitrary loss units. I also invent a probability of 0.30. The threshold is 2/(2+8)=0.2. Leaving the message untagged has expected loss 80.30=2.4, while tagging it has expected loss 20.70=1.4. Our policy adds the tag. None of these numbers come from model output or measured business costs.

At exactly the threshold, both actions have equal expected loss, so I need an explicit policy for the tie. Equal positive error costs would put that threshold at 0.5. This calculation still depends on usable probability and cost estimates. For more than two classes, I apply the same expected-loss comparison to every available action.

Apply a loss matrix to every possible class

I apply the same expected-loss rule when there are more than two classes. Each row is an action, and each column is a true class. For this calculation, billing, technical, and sales must be mutually exclusive and exhaustive. I multiply every column’s probability by its row’s loss and add the products.

For this example, I use made-up probabilities and costs. The costs are in arbitrary loss units, and neither input comes from a model inference run or business measurements. I keep the full probability vector because every possible class contributes to each action’s expected loss. The policy calculation belongs in application code.

from decimal import Decimal as D

p = {"billing": D("0.55"), "technical": D("0.35"), "sales": D("0.10")}
loss = {
    "assign_billing": {"billing": 0, "technical": 8, "sales": 4},
    "assign_technical": {"billing": 2, "technical": 0, "sales": 3},
    "assign_sales": {"billing": 4, "technical": 6, "sales": 0},
}
risk = {action: sum(row[label] * p[label] for label in p) for action, row in loss.items()}
action = min(risk, key=risk.get)

The sample computed losses are 3.2 for assigning billing, 1.4 for assigning technical, and 4.3 for assigning sales. Billing has the highest probability at 0.55, but I choose the technical action. Assigning billing carries a loss of 8 when the true class is technical, weighted here by 0.35. That consequence changes which action minimizes the sum, even though the probability ranking stays the same.

Before using this calculation on a response, I require validated inputs. Probabilities must be finite, within the probability range, and normalized within an explicit rounding tolerance; every loss row must match the expected label set and contain finite losses. The compact example assumes those checks have passed. I also make ties explicit: the first action in configured row order wins.

Ardana rounds returned probabilities to four decimal places, so their sum can differ slightly from one. I account for that rounding when validating inputs and comparing close risks. The top label alone is sufficient under zero correct-classification loss and equal positive losses for every mistake. Our unequal rows require the full calculation.

Evaluate the policy after fixing its costs

I would fix the application costs before comparing policies. With known costs and usable probabilities, the analytic rule determines the action without a threshold search. If I tune a binary threshold empirically, our application losses would determine the score. Scikit-learn’s threshold tuner defaults to balanced accuracy, so I would supply a scorer that rewards lower application loss.

For empirical tuning, I would keep the tuning data separate from the data used to fit the classifier. The final comparison would use an untouched evaluation set. On that set, I would compare the most-probable-label and cost-sensitive policies against labeled outcomes, using the same fixed loss matrix.

The arithmetic depends on usable probability and cost estimates. I would check that the probabilities are usable on our workload before relying on the derived threshold. Changing a threshold does not repair the probabilities underneath it. A lower calculated expected loss does not establish realized savings or guarantee that each action is correct.

For later drift, I would use the separate probability monitoring workflow. We also need the loss units and assumptions recorded with each matrix version. Start with Ardana on GitHub, adapt the fixture above to your own labeled cases, and version the application loss matrix alongside it.