Evaluate rare events with local decision models

A strong test score can fill the wrong queue

I do not trust a precision number without the class balance behind it. In our hypothetical example, a balanced test gives 98.77% precision at 80% recall and a 1% false-positive rate. If positive cases make up 1% of traffic and those rates stay unchanged, precision falls to 44.69%. We need to check the accounting before we judge the model.

My hypothetical workflow tags customer requests for a VAT invoice. I count a ticket as positive when the customer requests that invoice. Every positive prediction enters an invoice-review queue, including tickets tagged incorrectly. The question supplies routing evidence, not a security guarantee. These numbers are assumptions for the example, not measured Ardana accuracy.

At 10,000 tickets per day, our assumptions produce 179 reviews: 80 true positives and 99 false positives. A queue with capacity for 100 reviews is already short by 79. In evaluating prompt-injection detectors, I focused on recall at a chosen false-positive rate. For this example, I need to relate precision and queue volume to event prevalence. I start with a score for one explicit question.

Collect a score for one explicit event

I start with one explicit event: the customer asks for a VAT invoice. That gives our score a specific meaning before we choose any routing threshold. The request follows Ardana’s v0.3.2 request schema. I save it as request.json.

{
	"model": "decider:2b",
	"state": "Please send a VAT invoice for order A-104.",
	"questions": {
		"vat_invoice": { "type": "noul", "instructions": "Does the customer request a VAT invoice?" }
	}
}

For a local run, I pass the saved request file and ask for JSON output. The selected model runs the question against the supplied ticket text. I have not executed this example, so there is no measured score to show.

ardana run decider:2b --request request.json --json

The HTTP path uses the same file. I start the local server, then send the file from another terminal with curl once the server is running.

ardana serve

curl http://127.0.0.1:8000/v1/systemone -H 'content-type: application/json' --data-binary @request.json

I read the probability of yes from this exact response path. It belongs to our invoice question, so a larger value means a higher returned probability that the customer requested a VAT invoice. This is the score I keep for evaluation.

answers.vat_invoice.noul

Our readout rounds that probability to four decimal places after normalizing the options. I retain the returned scores, keeping the reference labels separate because the model’s answer does not supply the label. When choosing thresholds, I handle equal scores consistently. The local probability is evidence for routing a ticket into review; it provides no security guarantee. With a score and a separate label for each case, we can count what a chosen threshold sends to review.

Recompute precision at the deployment prevalence

I start with the denominators because each metric answers a different question. Precision counts correct invoice requests among tickets sent to review. Recall counts detected requests among all actual requests, while the false-positive rate counts incorrect routes among tickets without a request.

precision = TP/(TP+FP)
recall/TPR = TP/(TP+FN)
FPR = FP/(FP+TN)

For deployment prevalence π, recall r and false-positive rate f, I calculate precision with the expression below, provided its denominator is positive. I keep the threshold fixed and assume the class-conditional score distributions remain unchanged. That means changing how common invoice requests are, while preserving the score distribution within each class. A change in the kinds of requests or other tickets can break this assumption.

πr/[πr+(1−π)f]

My hypothetical balanced evaluation has 2,000 tickets: 1,000 invoice requests and 1,000 other tickets. With recall 0.80 and false-positive rate 0.01, I get 800 true positives and 10 false positives. Precision is 800/810, approximately 98.77%, and recall is 80%. I chose these assumptions and calculated the results; they are not measured Ardana performance.

Now I apply the same rates to a hypothetical 10,000 tickets per day with 1% invoice requests. That gives 100 actual requests and 9,900 other tickets, producing 80 true positives and 99 false positives. Precision becomes 80/179, approximately 44.69%, while recall stays at 80%. If I route every positive prediction to review, the queue receives 179 tickets per day. The 1% prevalence is an assumption for this example, not an estimate of real traffic.

Suppose I can review 100 tickets per day and there are no other arrivals. Those 179 routed tickets exceed capacity by 79 per day. Keeping only the top 100 changes the routing policy, and these aggregate counts cannot tell me its precision or recall. I need to evaluate that capped policy before assigning it a quality score.

Read average precision with its baseline attached

I use a precision-recall curve to see how precision changes as we recover more positive cases. ROC and PR describe related confusion matrices, but precision puts false positives beside true positives, which is what our review queue receives. Davis and Goadrich show that optimizing ROC area does not necessarily optimize PR area.

For our assumed deployment mix, the population no-skill PR reference is 0.01, because prevalence is 1%. I keep that reference beside the curve because its height changes with class balance. It is not the exact expected AP of a finite random ranking; finite random rankings can average above it. We need the evaluation prevalence attached to AP, and cannot carry balanced-test AP unchanged into rare-event traffic.

To calculate AP, I use true labels and positive-class scores, before applying a routing threshold. Scikit-learn’s average precision weights each precision value by the increase in recall at that step. This estimator does not interpolate between points. Trapezoidal PR area uses linear interpolation, which can be optimistic. I report which estimator I use because a number labelled only “PR area” leaves that choice unclear.

Our returned yes probabilities are rounded to four decimal places, so I retain those scores and handle ties consistently. Cases with the same score cross a threshold together when we include scores at or above it. Scikit-learn’s PR curve ends at precision 1 and recall 0, but that final point has no corresponding threshold. I leave it out of the candidate operating points. PR and AP do not certify calibrated probabilities or choose a review capacity; we still have a routing policy to set.

Freeze the routing policy before final evaluation

I would choose the model, question and routing threshold on validation data, with daily review capacity beside the quality metrics. Then I would freeze them and evaluate the final labeled holdout without retuning. The holdout needs cases from both routes. If we label only tickets sent for review, we cannot count missed positives or establish recall across all tickets.

For that final run, I would report true positives, false positives, false negatives and true negatives, plus total positives, total negatives and prevalence. My report would include the PR curve and AP computed from labels and scores. It would also show precision, recall and review volume against capacity at the frozen threshold. A cap of 100 reviews per day changes the hypothetical policy. The earlier aggregate counts cannot tell us its new precision or recall.

That gives us a static evaluation of one frozen policy. I cover label provenance and leakage and ongoing calibration monitoring separately. To build your own evaluation, start with Ardana’s quick start and adapt the request fixture to your routing question. When you run it, record both the numerator and denominator beside every rate.