Build conformal sets from local decision probabilities

A probability does not specify a covered label set

I want to route a support request, but two departments look plausible. Picking the highest-scoring department gives the application one destination and discards the competing candidate. For this setup, I assume each request has one true department.

I want a candidate label set that can keep both departments under consideration without asserting that both are correct.

Ardana already returns a probability for every option in a choice question. I would build a split conformal wrapper around those probabilities in application code. The application would handle this post-processing; Ardana’s released schema has no conformal calibration or prediction-set fields. The wrapper needs labeled calibration examples held apart from model training.

Under exchangeability of calibration and future test cases, split conformal prediction provides a marginal coverage guarantee for the candidate set. Angelopoulos and Bates give the construction and its assumptions. Coverage averages over calibration and test randomness; it is not a certificate for the particular request I am routing. The model’s probabilities do not need to be perfectly calibrated, so I can start with its existing choice scores.

Collect the same choice scores for calibration and use

I would start our wrapper with the local choice scorer. If needed, I first install Ardana with the optional setup command below. Then I pull the model and start the service with the documented commands. I leave the server running while preparing requests.

curl -fsSL https://ardana.ai/install.sh | sh

ardana pull decider:2b
ardana serve

For this routing example, I save the reduced request below as request.json. Our candidates are billing, technical, and sales. I keep those labels fixed for calibration and later use.

{
	"model": "decider:2b",
	"state": "My card was charged twice for order A-104. Please refund the duplicate.",
	"questions": {
		"department": {
			"type": "choice",
			"instructions": "Which department should handle this?",
			"criteria": ["billing", "technical", "sales"]
		}
	}
}

Our request body goes to the local endpoint with the documented content type. I read the full probability map from the parsed response using the lookup below.

curl --fail-with-body -H 'content-type: application/json' --data-binary @request.json http://127.0.0.1:8000/v1/systemone > response.json

body["answers"]["department"]["probabilities"]

The response also includes the separate scalar named below. I keep it out of our set construction. Its definition rescales the largest option probability using the number of options, and it is not a conformal coverage guarantee.

confidence

Before collecting conformal scores, I fix the weights, quantization, instructions, taxonomy, preprocessing, and any probability calibration. Our labeled conformal holdout must be independent of model training. I keep it out of tuning those choices too, so our score function stays fixed.

Returned probabilities are rounded to four decimal places. I use identical transformations and score arithmetic for calibration and deployment, even when rounded values do not sum to exactly one. Renormalizing only one side would break that consistency. Our application owns this post-processing; the released API has no conformal prediction-set fields.

Select the finite sample rank exactly

I turn each labeled calibration case into a score by subtracting the probability of its true label from one. A larger score means the model agreed less with that label. This is the LAC score used by MAPIE; in real calibration, I compute it from independently labeled held-out cases.

With n calibration scores and a predetermined error level α between zero and one, I multiply n + 1 by 1 − α and round up to get rank k. The finite sample construction in Appendix D takes the kth smallest score when k ≤ n, and positive infinity otherwise. I use that exact order statistic because a generic quantile call can use a different index convention.

I keep a candidate label when one minus its probability is at most the cutoff, including equality. Save the complete example as conformal_example.py. I use an integer-built fraction for α so the ceiling uses the exact fraction, and validate the scores, probabilities, and fixed taxonomy before constructing sets.

from fractions import Fraction
from math import ceil, inf, isfinite

LABELS = ("billing", "technical", "sales")
SCORES = [0.05, 0.10, 0.15, 0.20, 0.25, 0.30, 0.40, 0.65, 0.80]


def cutoff(scores, alpha):
    # Pass alpha as a Fraction built from integers, not from a binary float.
    if not isinstance(alpha, Fraction):
        raise TypeError("alpha must be a Fraction")
    if not scores or not 0 < alpha < 1:
        raise ValueError("need scores and 0 < alpha < 1")
    if any(not isfinite(s) or not 0 <= s <= 1 for s in scores):
        raise ValueError("scores must be finite values in [0, 1]")
    n = len(scores)
    k = ceil((n + 1) * (1 - alpha))
    return k, sorted(scores)[k - 1] if k <= n else inf


def prediction_set(probabilities, qhat):
    if set(probabilities) != set(LABELS):
        raise ValueError("probability labels must match the fixed taxonomy")
    if any(not isfinite(p) or not 0 <= p <= 1
           for p in probabilities.values()):
        raise ValueError("probabilities must be finite values in [0, 1]")
    # Use the same subtraction during calibration and prediction; keep ties.
    return [label for label in LABELS if 1 - probabilities[label] <= qhat]


def main():
    alpha = Fraction(1, 5)
    k, qhat = cutoff(SCORES, alpha)
    print(f"n={len(SCORES)} alpha={float(alpha)} k={k} qhat={qhat:.2f}")
    examples = {
        "singleton": (0.70, 0.20, 0.10),
        "multiple": (0.45, 0.40, 0.15),
        "empty": (0.34, 0.33, 0.33),
        "boundary": (0.35, 0.35, 0.30),
    }
    for name, values in examples.items():
        print(name, prediction_set(dict(zip(LABELS, values)), qhat))
    k, qhat = cutoff(SCORES, Fraction(1, 20))
    print(f"alpha=0.05 k={k} qhat={qhat}")
    print("infinity", prediction_set(dict(zip(LABELS, examples["empty"])), qhat))


if __name__ == "__main__":
    main()

These nine scores and four probability vectors are authored arithmetic examples. With α = 0.2, the eighth score is 0.65, giving a probability cutoff of 0.35 and an 80% target. I run the file with the command below.

python3 -B conformal_example.py

The boundary vector keeps both labels at 0.35 because equality passes. With α = 0.05, rank 10 exceeds the nine observed scores, so the cutoff becomes infinity and the set contains the whole taxonomy. I keep that branch explicit; replacing it with the largest observed score would change the construction.

n=9 alpha=0.2 k=8 qhat=0.65
singleton ['billing']
multiple ['billing', 'technical']
empty []
boundary ['billing', 'technical']
alpha=0.05 k=10 qhat=inf
infinity ['billing', 'technical', 'sales']

What the coverage guarantee means

I read the guarantee as a statement about repeated calibration and test draws. Without ties, exchangeability makes the test score’s rank uniform among the calibration scores plus the test score. For a finite cutoff, counting accepted ranks gives at least our target coverage; inclusive ties preserve that lower bound. The infinity branch includes our entire fixed taxonomy. Appendix D gives the rank argument.

That promise averages over both calibration and test randomness. I cannot read it as a probability certificate for this input, a guarantee for each subgroup, or the target coverage for every realized calibration sample. Perfectly calibrated model probabilities are not required, though score quality affects how informative our sets are. Conformal theory makes these distinctions.

My application still needs a policy for each set shape. An empty set means no candidate passed, so I would send it for review. Multiple candidates are alternatives for one true label, and I would preserve them for the next decision. Keeping only the top label does not inherit the set’s coverage guarantee, while automating only singletons gets no conditional error guarantee from the marginal theorem alone. MAPIE explains the coverage scope.

I cannot use an empty set to certify an unknown label. If the true label is outside our taxonomy, no returned subset can contain it. That limitation also applies when the wrapper returns the whole taxonomy.

Our calibration observations must be independent of model training and jointly exchangeable with the future test pair. I would fix the model, instructions, taxonomy, preprocessing, and score transformation before collecting conformal scores.

Measure set size beside held-out coverage

I would start with independently labeled cases held out from model training. Fix the scorer, instructions, taxonomy, and score arithmetic before collecting their probabilities. Use a calibration holdout to choose the cutoff, then evaluate on a separate, untouched test set. Every true label must belong to the fixed taxonomy. Keep the test cases out of any adjustments to the scorer or cutoff.

Then I would report empirical coverage: the fraction of test cases whose true label appears in the returned set. I would also report the average set size and how often sets are empty, singletons, or contain several candidates, so we can see how many choices the application still has to resolve. Keep the entire set for the routing policy. Reducing it to the top label does not inherit its coverage guarantee.

We have checked the example’s arithmetic, but have not run live model inference for this example. Deployment coverage and the calibration sample size needed for useful sets remain unmeasured. The authored scores give us no model benchmark or evidence that a production sample is adequate. To collect that evidence, start the local scorer from the Ardana repository, save held-out choice probabilities with their independently assigned true labels, and apply the wrapper to those scores.