Monitor decision probability calibration drift

A rollout changes the population being measured

We leave a decision threshold in place, then fresh traffic arrives. The cutoff is unchanged, but the population behind our decisions may be different. Rerunning the old labeled set cannot tell us how those new cases behave. I want evidence from recent traffic before relying on the old calibration result.

Ovadia and colleagues found that conventional post-hoc calibration could fail under dataset shift in their classification experiments. I take that as a reason to check reliability after rollout. At the same time, distribution changes can be benign. A changed score distribution alone gives us no basis for declaring that performance has degraded.

In the cascade post, I covered threshold routing, prediction logs, later label joins and scheduled revalidation. The quantization comparison checked accuracy, Brier and ECE on benchmark items. Do our probabilities still hold up on the traffic reaching this deployment? I keep a fixed labeled reference for regression checks and assess recent traffic once its labels are mature.

Record the probability for the event being labeled

I start with the event written in the question. For this refund request, our label records whether the customer asked for a refund, not whether we issued one. I save the documented request as request.json:

{
	"model": "decider:2b",
	"state": "My card was charged twice for order A-104. Please refund the duplicate.",
	"questions": {
		"refund": { "type": "noul", "instructions": "Does the customer ask for a refund?" }
	}
}

Then I run the request and keep the JSON response so our ledger stores the returned probability directly. Human CLI percentages use one decimal place, while probability readout uses four decimal places. I use the documented command:

ardana run decider:2b --request request.json --json

I start the server below for the separate HTTP alternative. It exposes POST /v1/systemone at http://127.0.0.1:8000.

ardana serve

For the noul question above, I compare its probability of yes with labels for refund requests. Choice confidence rescales the largest choice probability; it is not a probability of correctness. An ordinal score is an expected level, so I use the individual level probabilities to check how often each level occurs. I check the response definitions before choosing which returned number to monitor.

Our application owns the prediction and label ledger. I would store the original request and response under an application identifier so we can join them to labels when they become available. Each record would include prediction time, label arrival time and the matching label. The inspected API supplies neither those tracking fields nor a label-ingestion or calibration-monitoring endpoint. I would also record the deployed artifact separately, because the response identifies a model profile rather than a checksum.

Wait for comparable labels and inspect missing ones

I would keep missing labels out of the negative count. In conversion prediction, an event that has not happened yet can still happen later. A short observation window can create false negatives, while a longer wait makes the usable data older. Chapelle’s delayed-feedback research describes this tradeoff and why elapsed time affects which labels are missing.

For our refund question, the label remains whether the customer requested a refund. I would keep an unfinished review pending instead of treating it as evidence that no refund was requested. Waiting for a refund to be issued would measure a different event.

I would use the application ledger to join each label to its original prediction by an application identifier. We preserve both prediction and label-arrival times. For an outcome that can occur later, we define the observation horizon before comparing groups. I would group cases by prediction time and compare equally mature cohorts, rather than whichever labels arrived most recently. We keep pending counts beside labeled totals to see how much evidence is still missing.

But waiting does not fix every missing label. Research on selective labels shows how decisions can determine which outcomes become observable, leaving an unrepresentative labeled subset. If we review only escalated cases, we cannot assume those labels describe automatically accepted cases. Waiting cannot recover an outcome that our action prevents us from observing.

Before comparing groups, I would report label coverage by score group and action route. Where feasible, I would sample accepted cases for review alongside escalations. We need consistent question definitions and segment rules so those comparisons refer to the same things.

Keep score groups stable and traffic mix visible

I want each probability group to cover the same range across mature cohorts. For that comparison, I would freeze the boundaries, question definitions and segment rules. For each group, show the count, mean probability, observed event frequency and uncertainty; leave empty groups unestimated. Fixed boundaries improve comparability, but they do not eliminate estimation error or the sensitivity to binning choices.

For a binary event, I use p for its predicted probability and y for the observed label, zero or one. The binary Brier score averages the squared gap between them. Keep this normalization fixed: it is half the score obtained by summing over both classes.

mean((p−y)²)

Lower Brier is better, but I need to know which component changed. Proper-score decomposition separates miscalibration, discrimination and outcome uncertainty. A worsening total alone does not identify calibration drift. For expected calibration error, I also need the group counts: its bin weights change when traffic moves between groups, so the aggregate can move even with unchanged within-bin errors.

Overall calibration gives me no guarantee about calibration within each subgroup. Report segment proportions and within-segment reliability and losses alongside the aggregate, checking predicted classes separately. For a mix diagnostic, I would weight recent mean losses for disjoint segments by their reference proportions and compare the result with the actual recent mean loss. Keep absent segments visible; do not silently renormalize the remaining weights.

I would compare a frozen labeled reference set with recent mature labeled traffic. Recompute reliability and proper scores on the reference for regression checks, then inspect the recent cohort for deployment performance. Unchanged reference predictions and labels cannot expose new traffic drift. A changed score histogram alone also does not establish degraded performance; distribution changes can be benign.

Revalidate thresholds when the evidence changes

Revalidate a cutoff when mature labels show sustained deterioration beyond our chosen risk tolerance. Changed decision costs or model, question, and policy definitions also warrant review. I would check false positives, false negatives, accepted-case errors, and action volumes before choosing another threshold. Drift alone does not specify the replacement cutoff.

Repeated ordinary significance tests inflate false alarms. Podkopaev and Ramdas instead compare a time-uniform target-risk lower bound against a source-risk upper bound plus a prespecified tolerance. Their guarantee requires bounded loss and appropriate sampling assumptions. We cannot attach it to an arbitrary rolling-window alert.

I would fit any proposed recalibration on held-out validation data and evaluate it on separate labeled data. Guo and colleagues separate training, validation, and testing while assuming a shared distribution. That assumption matters when we apply their approach to changing traffic.

Ardana supplies probabilities, while this monitoring workflow lives in our application. The API has no label-ingestion or calibration-monitoring endpoint. Start with the Ardana repository quick start, record the API probabilities for refund requests, and join mature labels for whether a refund was requested.