Interpret ordinal scores from decision models

The same score can hide opposite answers

I cannot read an ordinal score of 1.0 as belief in the middle level. In a synthetic example, the distributions [0,1,0] and [0.5,0,0.5] over levels 0, 1 and 2 both have means of 1.0. The middle-level probability is 1 in the first distribution and 0 in the second.

In the first distribution, the middle level also has the highest probability. In the second, the two extremes tie. Rounding the mean to an integer cannot recover that difference. Our confidence calculation separates these cases too: 1 for the first, 0 for the second.

Ardana’s ordinal score is an expected level index, computed by weighting each index by its probability. That average can land on a level with zero probability, so I need the distribution to see where the probability sits.

I covered the basic answer shapes in the probability introduction. Here I care about what we lose when we keep only the mean. Its meaning starts with the rubric and the indices we assign to its levels.

The rubric supplies order, not equal distances

I start with the descriptions because they define what the levels mean. Ardana accepts 2 to 10 ordered descriptions, lowest first. It assigns them positions from 0 through n−1 and computes the expected position from normalized probabilities, rounded to two decimal places. The result is measured in level indices.

That ordering gives us a sequence, but it does not establish equal distances between meanings. A move from the lowest description to the middle one need not represent the same change as a move from the middle to the highest. We can compute a mean of assigned numbers. Its interpretation still depends on the numbers we assign.

Numeric object keys are an easy place to get this wrong. I might try to express wider gaps by writing the levels like this. The object is accepted, but those keys do not set the weights:

{ "0": "low", "10": "medium", "100": "high" }

Ardana sorts the keys numerically, then discards them and reindexes the descriptions as positions 0, 1, and 2. The weights are therefore 0, 1, and 2, not 0, 10, and 100. Changing the key values is not a way to supply a different distance scale.

Here is constructed arithmetic, not model output. For distribution A [0.6,0,0.4] and B [0,1,0], codes [0,1,2] give means of 0.8 and 1. Recoding the same ordered categories as [0,1,4] gives 1.6 and 1. A now ranks above B even though the category order has not changed.

I read our score as an expected level index with that coding attached. The ordering alone does not justify treating the gaps as equal changes in the underlying property. To see what the answer contains, I need the distribution behind that index.

Inspect the distribution behind the mean

I start with a customer message and a rubric I can inspect. This complete illustrative request asks how upset the customer is: calm, annoyed, or furious. The message is the input to the question. The three descriptions define the levels whose probabilities I want to examine.

{
	"model": "decider:2b",
	"state": "The app crashes on login. Fix it now!",
	"questions": {
		"mood": {
			"type": "score",
			"instructions": "How upset is the customer?",
			"criteria": ["calm", "annoyed", "furious"]
		}
	}
}

For this walkthrough, I use synthetic sample output calculated from the repository test’s level fits. It illustrates the arithmetic; these values did not come from running the request above. This object contains only the answer portion. The API wraps question answers with the model and token usage.

{
	"type": "score",
	"score": 1.3,
	"confidence": 0.25,
	"legend": { "0": "calm", "1": "annoyed", "2": "furious" },
	"probabilities": { "0": 0.1, "1": 0.5, "2": 0.4 },
	"x_p_max": 0.5,
	"x_certainty": 0.1413,
	"x_level_fit": { "0": 0.1, "1": 0.5, "2": 0.4 },
	"x_fit_mass": 1.0
}

I read the full probabilities before the mean: calm has 0.1, annoyed 0.5, and furious 0.4. Weighting their indices gives 0×0.1+1×0.5+2×0.4=1.3, the expected level index. Annoyed is the mode because its probability is largest. We have no separate selected-level field here, and rounding the expected index is not how I find the mode.

Confidence needs a different calculation. For this distribution, the mode is index 1, the probability-weighted distance from it is 0.1+0.4=0.5, and the normalization denominator is 2/3. We get 1−0.5/(2/3)=0.25. That measures distributional spread; it does not establish how often the answer is correct.

The maximum probability is 0.5, while the entropy-based certainty is 0.1413. I keep those separate from confidence because each compresses the distribution differently. Ardana rounds the expected score to two decimal places and probabilities, confidence, and extras to four, so recomputing from displayed probabilities can differ. The sample also includes individual level fits and fit mass, which we need to read alongside the normalized probabilities.

Level fit and probability answer different questions

I inspect isolated level fits alongside the normalized probabilities. In isolated scoring, we give each level its own yes/no row asking whether that proposed answer fits. The row contains neither neighboring levels nor a numeric prefix. Ardana decodes every row separately, so a question with n levels needs n rows instead of the single row used by listwise scoring.

Listwise scoring puts all numbered levels into that one row and normalizes their logits with a temperature-scaled softmax. Our pinned decider:2b profile enables isolated scoring. Its score temperature is 1.124, and the isolated rows keep that temperature despite having yes/no options.

Within a score question, I can select listwise scoring with the following field. Doing so also omits the fit extras. Setting the option to true cannot enable isolation when the model profile disables isolated levels.

"isolated":false

For isolated scoring, we take each level’s yes probability as its fit. Fit mass is the sum of those fits, and we divide each fit by that mass when the sum is positive. Consider the constructed arithmetic examples [0.01,0.05,0.04], [0.1,0.5,0.4], and [0.2,1,0.8]. Their fit masses are 0.1, 1, and 2, respectively. All three produce the same normalized probabilities: [0.1,0.5,0.4].

The first set has small absolute fits across all levels, yet normalization produces the same distribution as the other two. I cannot recover that lost magnitude from the normalized probabilities. A mass of 2 also shows why fit mass is not itself a probability. Confidence still describes distributional spread, so it cannot recover the absolute fits either. Before assigning an action, I want to see both the level fits and the distribution they produce.

Try the rubric before assigning an action

I start with the mood request above and save it as request.json. It keeps the login complaint and the calm, annoyed, furious rubric together. Run it through the CLI so we can inspect the answer before choosing what the application should do.

ardana run decider:2b --request request.json --json

For HTTP, I start the local server first. The request stays in the same file. We can use it to inspect the response through the interface our application will call.

ardana serve

Leave the server running and open another terminal. Send the request from the directory containing the saved file. I keep the rubric unchanged here so we are asking the same question through either interface.

curl http://127.0.0.1:8000/v1/systemone -H 'content-type: application/json' -d @request.json

In the playground, I look at the probabilities beside their level descriptions. We can also inspect isolated fits, fit mass, and the other summaries there. If our action depends on reaching a particular level, I add the probabilities at that level and above. That gives us a threshold probability, a different application policy from acting on the expected level index.

For the mood rubric, we can make that boundary furious. Choosing when its probability should trigger an action is a decision we make in the application. Try the request with your own descriptions, then use the Ardana repository to inspect the implementation.