Select local classifier examples for human review

The hardest cases are not a traffic sample

I do not use our review queue’s unweighted error rate as a traffic estimate. When we choose examples because the model is uncertain, we change the sampled distribution. That selection helps us choose where to request human labels. It also limits what I can conclude from the resulting error rate.

We select unlabeled examples to send to humans for acquisition. I keep evaluation separate: I take a uniform random sample from a defined decision population and hold the model fixed. We can then average its errors without using uncertainty to decide which examples enter the audit.

Building this queue does not train our model. In the Ardana version used here, I still need application code to manage human review, and the CLI exposes no finetuning command. Any training that uses the acquired labels is a separate step. Before choosing which cases to review, I also separate probabilities from human judgments: a model probability is not an observed correctness label.

Choose uncertainty scores for a fixed task

I start with a fixed routing question and the same options for every decision. We also keep the model artifact and label policy fixed across the batch. I save this complete illustrative request as request.json. The input contains the message to route; our human labels stay outside it.

{
	"model": "decider:2b",
	"state": "My card was charged twice. Please refund the duplicate.",
	"questions": {
		"department": {
			"type": "choice",
			"instructions": "Which department should handle this?",
			"criteria": ["billing", "technical", "sales"]
		}
	}
}

For each choice answer, I read the full probability map returned with the selected option. Ardana also returns a confidence score that rescales the largest probability using the number of options. We can read that contract in the response types. Those numbers alone do not establish how often our decisions are correct.

With margin sampling, I subtract the second largest probability from the largest. Smaller gaps go first because the leading two options are closer together. Our ranking describes how closely the options compete; correctness still needs a human label.

For least confidence, I use one minus the largest probability. Entropy uses all the option probabilities. Both rank larger values first. For binary classification, all three methods rank uncertainty equivalently, closest to 0.5. Our request has three options, so I keep the choice of score explicit.

I also account for serialized probabilities being rounded to four decimal places. Their sum can land slightly above or below one. With the task and response contract fixed, we can define the successful-decision pool before selecting review items.

Reserve the audit before filling the review queue

I start with a frozen pool of successful decisions under one question, model, and label policy. The proposed sampler validates that pool before selecting anything. Duplicate IDs, invalid budgets, wrong labels, and invalid probabilities cause rejection. Silently dropping those records would change the population we intended to sample. Freezing the pool gives us a defined set of events for this batch.

For the audit, I sort the complete pool by ID and sample uniformly without replacement before ranking any margins. Every eligible event gets the same chance of inclusion, regardless of its scores. Our fixture is hypothetical: eight records, an audit budget of two, and an acquisition budget of three. Each event has an audit inclusion probability of 0.25. These inputs and sampling results are examples. They did not come from an inference run or human labeling.

I use score-independent audit selection to mean that model scores never decide audit membership. The two review streams are disjoint but depend on each other because reserving an event for audit changes what acquisition can select.

Only after reserving the audit do I rank the remaining records by ascending margin, breaking ties by ID. I skip every application-defined group represented in the audit. From each other group, I take at most one record for acquisition. Those exclusions can leave the acquisition budget underfilled, even when records remain in the pool. Group filtering never removes or replaces an audit record.

We evaluate only that frozen successful-decision pool. Inference failures sit outside it, so the audit cannot describe their outcomes. I also keep the acquisition queue’s raw error rate separate: uncertainty selection changes the sampled distribution. Once the records reach a reviewer, we still need to distinguish a uniformly sampled event from one selected through ranking and group exclusions.

Keep selection history beside human annotations

I keep the original decision beside a separate annotation record. That record holds review status, the human label, reviewer identity, and review time. Selection history tells us why the example entered the queue and whether it belongs to acquisition or audit. I leave both streams’ human labels out of the model input used to produce the decisions we evaluate.

Our manifest identifies the frozen pool, model artifact, question, label policy, and grouping rule. I also document the collection window, collectors, and sampling procedure, following the provenance questions in Datasheets for Datasets. The response reports a model profile name, so I retain artifact identity separately to make the decision reproducible.

A review budget can remain partly unused. Our acquisition rule skips groups already represented in the audit and takes at most one example from each remaining group. If those exclusions leave too few candidates, I record the shortfall. Filling every slot is not a reason to change audit membership.

Unresolved annotations stay visible in our records. Difficult examples can be harder to label, so dropping unresolved audit items can introduce non-response bias. I record their status and keep the missing labels visible when reporting results. A completed subset needs its own explanation of what remains unresolved.

Our audit covers the frozen population of successful decisions. Inference failures are outside that population. I state that scope beside any reported error rate. If I use audit examples to revise prompts, rubrics, or thresholds, those examples have entered development. Under the usual data leakage rule, I can no longer call them an untouched test set for the revised system.

Start with a saved local decision request

I start with the saved request from earlier. It contains the input text and department question, ready to send to local inference. The CLI source I inspected supports these commands.

ardana pull decider:2b
ardana serve --host 127.0.0.1 --port 8000 --default-model decider:2b

With the server running, I send that saved request from another terminal. I keep the response as the original decision record, separate from the human annotation. The returned choice and its probabilities provide the prediction data for the review workflow. I can also send the saved request through the CLI.

curl http://127.0.0.1:8000/v1/systemone -H 'content-type: application/json' -d @request.json

ardana run decider:2b --request request.json --json

I budget inference and human review separately. The server’s inference queue limit controls admission to model execution; it does not reserve reviewer time or create annotation tasks. In the inspected version, review management belongs to the application, and the CLI has no finetuning command. Filling a queue does not train the model.

My next step is to have a reviewer apply the label policy to a queued record. I save their label, identity and review time beside the preserved decision. Any audit result stays scoped to the successful decisions in its defined population. To connect your saved request to local inference, start with Ardana on GitHub.