Recalibrate local decision probabilities on your labels

A threshold cannot repair a probability estimate

I can move an automation’s decision threshold and leave its probability problem untouched. When I route a refund request by its estimated probability, the cutoff decides which cases pass. It does not repair the estimate itself. Probability estimation and action selection are separate jobs.

I propose fitting a map that the caller owns, for one fixed binary event. We use Ardana’s returned score as input and fit a regressor against labels for whether the customer requested a refund. The map estimates that event’s probability. We keep downstream action selection separate from this fit.

Ardana’s shipped request has no calibrator-fitting field. I fit the map outside the model, using its returned probability as a score. Ardana rounds that value to four decimal places and exposes no raw logits. A map cannot recover the discarded precision. We keep calibration labels separate from classifier training data. Final evaluation labels stay outside both fitting and method selection. We do not assume calibration will improve the result.

Collect the probability for one fixed event

I collect scores for one fixed question: did the customer ask for a refund? A positive label means a request was made, regardless of whether a refund was issued. First I install Ardana, pull the model, and keep the server running in this terminal.

curl -fsSL https://ardana.ai/install.sh | sh
ardana pull decider:2b-q4_k_m
ardana serve

Next I save this authored example as request.json. It pairs one customer message with the refund question. I keep that event definition fixed when collecting labels.

{
	"model": "decider:2b-q4_k_m",
	"state": "My card was charged twice for order A-104. Please refund the duplicate.",
	"questions": {
		"refund": { "type": "noul", "instructions": "Does the customer ask for a refund?" }
	}
}

In a second terminal, I send the saved request and capture the body. The curl flag keeps error bodies while returning failure for HTTP status 400 or greater. I check HTTP success and valid JSON before extracting a score.

curl --fail-with-body http://127.0.0.1:8000/v1/systemone -H 'content-type: application/json' -d @request.json > response.json

For each answer, I require the same type as the question and a finite numeric score between 0 and 1. The exact field access is below. Ardana rounds this yes probability to four decimal places and returns no raw logits. I treat it as the map’s input score; fitting cannot recover discarded precision.

payload["answers"]["refund"]["noul"]

Finally I record artifact provenance with the collected scores. The response’s model name is a profile name, so it cannot identify the weights by itself. For this pull, the catalog gives the file, repository, and revision below; the file is 1274396800 bytes.

decider-2b-v11-Q4_K_M.gguf
hf.co/Mapika/decider-2b-GGUF
ff2e5e687327eda9ac34e9a3ca84d3f400672c87

Fit and save a caller owned isotonic map

I wrote these fixtures so we can inspect the fitting and persistence steps. All scores and labels are authored synthetic data: 16 fitting cases with 7 positives and 8 separate test cases with 4 positives. Repeated scores belong to different invented cases. We run no Ardana inference here, and these inputs establish no real model performance.

For the map, I use isotonic regression to minimize squared error while requiring a nondecreasing function. We save its learned knots and use linear interpolation after reload. The code also checks that the reloaded map reproduces the fitted estimator on the embedded test inputs. Save the complete file as calibration_demo.py:

"""Authored synthetic arithmetic only; no Ardana inference or performance claim.

Publish the complete file if its invocation is used in the article.
Requires numpy and scikit-learn; all example inputs are embedded below.
"""

import hashlib
import json
import platform
from pathlib import Path

import numpy as np
import sklearn
from sklearn.isotonic import IsotonicRegression
from sklearn.metrics import log_loss


# Each repeated score represents a different invented case, not a repeated label.
p_fit = [0.1, 0.1, 0.1, 0.1, 0.4, 0.4, 0.4, 0.4,
         0.7, 0.7, 0.7, 0.7, 0.9, 0.9, 0.9, 0.9]
y_fit = [0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1]
# Separate invented case IDs; never passed to fit or used to select the method.
p_test = [0.05, 0.2, 0.4, 0.55, 0.7, 0.8, 0.9, 0.95]
y_test = [0, 0, 1, 0, 1, 1, 0, 1]

# Proposed deployment identity, not a claim that these scores came from it.
contract = {
    "ardana_source_sha": "445fbc945b02b641eeaef65a3ae0f0252f571aba",
    "model": "decider:2b-q4_k_m",
    "gguf_repo": "hf.co/Mapika/decider-2b-GGUF",
    "gguf_revision": "ff2e5e687327eda9ac34e9a3ca84d3f400672c87",
    "gguf_file": "decider-2b-v11-Q4_K_M.gguf",
    "question_id": "refund",
    "question": {"type": "noul", "instructions": "Does the customer ask for a refund?"},
    "label_rule": "1 means a refund was requested; 0 means it was not requested",
    "preprocessing": "plain-text-v1",
}
iso = IsotonicRegression(y_min=0.0, y_max=1.0, increasing=True, out_of_bounds="clip")
iso.fit(p_fit, y_fit)
fit_json = json.dumps({"p": p_fit, "y": y_fit}, sort_keys=True, separators=(",", ":"))
artifact = {
    "format_version": 1,
    "map_id": "refund-isotonic-demo-v1",
    "synthetic": True,
    "contract": contract,
    "fit_count": len(p_fit),
    "fit_positives": sum(y_fit),
    "fit_data_sha256": hashlib.sha256(fit_json.encode()).hexdigest(),
    "software": {"python": platform.python_version(), "numpy": np.__version__,
                 "scikit_learn": sklearn.__version__},
    "method": "isotonic",
    "interpolation": "linear",
    "out_of_bounds": "clip",
    "x": iso.X_thresholds_.tolist(),
    "y": iso.y_thresholds_.tolist(),
}
Path("refund-map.json").write_text(json.dumps(artifact, indent=2) + chr(10))
loaded = json.loads(Path("refund-map.json").read_text())
if loaded["format_version"] != 1 or loaded["contract"] != contract:
    raise ValueError("Map contract mismatch")


def apply_score(p_raw):
    if isinstance(p_raw, bool) or not isinstance(p_raw, (int, float)):
        raise ValueError("Expected a numeric score")
    if not np.isfinite(p_raw) or not 0.0 <= p_raw <= 1.0:
        raise ValueError("Expected a finite score in [0, 1]")
    # np.interp clamps to the first/last y outside the fitted x interval.
    p_recalibrated = float(np.interp(p_raw, loaded["x"], loaded["y"]))
    return {"p_raw": p_raw, "p_recalibrated": p_recalibrated,
            "map_id": loaded["map_id"]}


mapped = np.array([apply_score(p)["p_recalibrated"] for p in p_test])
# Check the saved JSON map reproduces the library prediction on these inputs.
np.testing.assert_allclose(mapped, iso.predict(p_test), rtol=0.0, atol=1e-15)
report = {
    "synthetic": True,
    "fit_count": len(p_fit), "fit_positives": sum(y_fit),
    "test_count": len(p_test), "test_positives": sum(y_test),
    "x": loaded["x"], "y": loaded["y"],
    "test_pairs": [apply_score(p) for p in p_test],
    "raw_brier": float(np.mean((np.array(p_test) - y_test) ** 2)),
    "mapped_brier": float(np.mean((mapped - y_test) ** 2)),
    "raw_log_loss": float(log_loss(y_test, p_test, labels=[0, 1])),
    "mapped_log_loss": float(log_loss(y_test, mapped, labels=[0, 1])),
}
print(json.dumps(report, indent=2))

Next I create the environment and install the pinned dependencies. These commands require Python 3.12; the documented run used Python 3.12.15, NumPy 2.5.4, and scikit-learn 1.9.1. Run them in order from the directory containing the file:

python3.12 -m venv .venv
.venv/bin/python -m pip install numpy==2.5.4 scikit-learn==1.9.1
.venv/bin/python calibration_demo.py

Our program writes refund-map.json in that directory, then reads it back. Alongside the knots, I record the fitting data hash, case counts, software versions, and interpolation behavior. The saved contract binds the artifact revision, question, label rule, preprocessing, and shipped source. Reload rejects a contract mismatch before we apply a score.

For this fixture, I include a proposed deployment identity. Its identifiers describe the intended artifact and task, while every score remains invented. We fit only our external map; there is no native model fitting in this program. I keep that distinction explicit because an artifact revision beside synthetic scores is not evidence that the artifact produced them.

I keep the generating source and data provenance with the map, following scikit-learn’s persistence guidance. Scikit-learn does not support loading pickled estimators across versions. Our JSON stores the coordinates and interpolation rule explicitly. With the map reloaded, we can examine its effect on the separate synthetic test cases.

Check the frozen map on independent labels

I freeze the map before looking at final evaluation labels. Calibration cases stay separate from classifier training cases, and final test cases stay outside both map fitting and method selection. Our eight invented test cases demonstrate the calculation; they establish nothing about Ardana performance.

Our authored test makes both losses worse. Brier loss increases from 0.20593750000000002 to 0.23573133680555555. Log loss increases from 0.6153665195535283 to 0.6641906590609915. Lower is better for both, but these losses combine calibration, discrimination, and outcome uncertainty, so I do not read either as a calibration verdict alone.

The saved map interpolates linearly between knots. With endpoint clipping, our input 0.05 becomes 0.25 and 0.95 becomes 0.75. I treat those as endpoint predictions, not evidence about scores outside the fitted range. Clipping decides how our map behaves there; it supplies no additional labels.

Pooling our middle observations creates a plateau from 0.4 through 0.7 at 0.375. Distinct inputs now share an output, so we can introduce ties and change ROC-AUC. I check those ties when evaluating the frozen map.

Sixteen invented fitting cases are only a fixture. Scikit-learn warns that isotonic fitting can overfit with calibration samples much smaller than 1000. I do not treat 1000 as an adequacy guarantee. We still need independent evaluation, with reliability diagrams comparing mean predictions against observed positive fractions and histograms showing the counts behind them.

I keep the returned score beside its mapped value and map identity. Our reload check requires the same artifact revision, question, label rule, preprocessing, and shipped source. If that contract differs, I reject the map before applying it.

Bind the map to the decision contract

My next step is to collect real labels for the fixed refund question. I keep those calibration cases separate from classifier training data and reserve final-test cases that stay outside map fitting and method selection. Once the method and map are frozen, I compare raw and mapped probabilities on those labels and save the evidence with the map.

For that comparison, I freeze the model artifact revision, question, label rule, preprocessing, and shipped source together. The response’s model name identifies a profile, so I cannot use it alone as provenance. I keep the fitting data’s provenance, generating code, and dependency versions beside the saved map. Each application retains both probabilities and the map identity, so the final-test report can point to the exact transform.

Before applying it, I check that the loaded contract matches. I choose downstream action thresholds separately from fitting the probability map. For the local decision API behind this workflow, start with our Ardana repository.