Multi-label classification with local decision models

One ticket can have both labels

I would give this example ticket both labels: “The export crashes and my card was charged twice.” The export failure is a technical problem, and the duplicate charge is a billing problem. If I make billing and technical compete for one winning label, I lose part of the report.

That is the distinction between multiclass and multilabel classification: multiclass assigns one class, while multilabel permits any subset of labels. I need the latter here. A ticket about an export failure alone can carry only the technical label. Adding the duplicate charge should add billing without displacing technical.

In Ardana, an exclusive choice returns the highest-probability option from the supplied options. For overlapping labels, I want a yes or no question for each label. Each question has its own yes probability, and these probabilities do not need to sum to one across labels. I can ask about billing and technical problems separately while giving both questions the same ticket.

Ask one yes or no question per label

I give each label its own yes or no question and keep the report as the shared input. The question keys are names we choose for our application. Here I use billing and technical, so the application can look up each answer by its label. Save this as labels.json; it is a synthetic request derived from the schema, and I have not executed it.

{
	"model": "decider:2b",
	"state": "The export crashes and my card was charged twice.",
	"questions": {
		"billing": {
			"type": "noul",
			"instructions": "Does this report a billing problem?"
		},
		"technical": {
			"type": "noul",
			"instructions": "Does this report a technical problem?"
		}
	}
}

Each question asks about one problem in the same report. The question type tells Ardana to return the probability of yes. I keep the keys stable when I change the wording of a question, so our application still knows which answer belongs to which label.

I install Ardana using the command for my operating system. The Bash command below is for macOS or Linux. The PowerShell command is the Windows alternative.

curl -fsSL https://ardana.ai/install.sh | sh

irm https://ardana.ai/install.ps1 | iex

I pull the model, then run the saved request through the CLI. The request file carries both named questions, so I do not need to rebuild them in the command. JSON output gives our application the response structure directly.

ardana pull decider:2b
ardana run decider:2b --request labels.json --json

For HTTP, I start the local server with the command below. I then send the same file from another terminal. Both entry points use this request, so I can keep the label names and question wording together in one place.

ardana serve

curl http://127.0.0.1:8000/v1/systemone -H 'content-type: application/json' -d @labels.json

I read each result through its label key, as shown below. The value is the probability of yes for that question. A yes or no answer has only its type and that probability; there is no separate confidence field. I do not add example scores here because this request has not been run.

response["answers"][label]["noul"]

Separate answers do not define a joint probability

I normalize each question separately in Ardana. A yes answer competes with its own no answer, and the returned probability is rounded to four decimal places. It does not compete with yes answers from the other labels. Several labels can therefore have high probabilities at the same time, without sharing a total that must add up to one.

We can read each score as an estimate of a label’s marginal probability: how likely that label is to apply, given the input. Our response assembles separate answers and exposes no joint label distribution or rules that enforce consistency between labels.

What would I need to calculate the probability that both labels apply? Let A and B stand for label events, with x as the input. The conditional probability identity gives us the relationship:

P(A and B | x)=P(A | B,x)P(B | x)

The first factor includes information that our separate answers do not provide. It asks how likely A is when B applies, given the same input. I can replace that factor with the marginal probability of A only if A and B are conditionally independent given x.

Separate scoring does not establish independence. I cannot infer a joint label distribution from calibrated marginals alone, and multiplying those scores establishes neither the probability that the complete label set is correct nor joint calibration.

For decisions that minimize expected individual label errors, conditional marginals are enough, even when labels depend on each other. The decision that maximizes exact-set accuracy can differ. We need to decide which errors matter before turning each probability into an application action.

Make each label an application decision

I keep the rule for emitting a label in the application. Each label gets a threshold, and the caller decides what action follows when that threshold is met. This is the comparison I would use for each label:

response["answers"][i]["noul"] >= thresholds[i]

Changing that threshold changes our decision without changing the predicted probability. I would choose thresholds on validation data, using a metric that reflects the cost of a missed label or an extra label. Tuning on classifier-training data risks overfitting, as the threshold tuning guide explains. Any rule that constrains which labels can appear together also belongs to the caller.

For validation, I want fixtures with overlapping expected labels. We need to check that the application can retain every expected label when several comparisons pass. Cases with a single expected label and cases with no expected labels also exercise the contract. I would record the complete expected set for each fixture, so an additional correct label does not become an error because the reference omitted it. For fixture design, see How to build a trustworthy classification test set.

I inspect errors per label before reading an aggregate score. A confusion matrix for each label separates missed labels from false alarms. Precision, recall and F1 give us another view of those decisions. A macro average weights labels equally, while a micro average pools their counts, so I would name the averaging method alongside the result.

The complete set needs its own check. Exact-set accuracy accepts a prediction only when every label matches, while Hamming loss counts incorrect individual label decisions. A decision that minimizes individual label errors can differ from one that maximizes exact-set accuracy. I would choose which objective drives our thresholds according to what the downstream action requires.

Run overlap cases through the local model

I put the label questions in one HTTP request, but that does not mean they share inference work. In v0.3.2, each yes or no question becomes one scoring row. The engine decodes each row separately and clears the llama.cpp context between rows. Adding labels adds scoring work, even when the text stays the same.

The reported input token count includes shared state once, even though each row is decoded separately. I would not take that count as evidence of a single shared model pass. Neither one request nor separate decoding tells us whether the labels are statistically independent.

Our default limits are 1024 scoring rows, 36864 tokens per row, and 1048576 tokens across rows. Context length also constrains what fits. Oversized requests return HTTP 413. I keep those limits in mind when adding labels to a long piece of text.

I would start with a small set of labeled overlap cases from the application you are building. Include text where you expect multiple labels to apply, so you can inspect those answers together. Run those cases through Ardana using the request command from above, and compare the answers with the labels you expected.