Test the evaluator before scoring a decision model

The grading code can change the apparent result

I test the grading code before I compare decision models. If removing a Markdown fence changes the grade without changing the answer, I need to know what that grade measures. My first test uses answers I wrote myself, with expected scores I can calculate by hand.

The October 1 Evidence Boundary report describes a fence-removal diagnostic that changed the apparent result without changing answers. The October 11 Last Responsible Moment report describes identical correct central decisions with different composite scores and an overly broad filename check. I have not independently reproduced either report.

Our site’s benchmark delegates execution and scoring to JevBench. That makes the adapter and aggregation code part of what I need to verify. Passing those software checks still leaves a separate question about label provenance: who assigned each expected answer and why it is correct.

When I design controls, I keep those questions separate and start by checking that each answer reaches the question it belongs to. The expected grade should be obvious before any model runs.

Match question IDs and option names before scoring

I start with two questions whose answers I can check from the state. The duplicate charge belongs to billing, and the app crash belongs to technical. I save this request as request.json. The command below runs that saved request.

{
	"model": "decider:2b",
	"state": "Billing handles the duplicate charge; technical handles the app crash.",
	"questions": {
		"department": {
			"type": "choice",
			"instructions": "Who handles the duplicate charge?",
			"criteria": ["sales", "technical", "billing"]
		},
		"handoff": {
			"type": "choice",
			"instructions": "Who handles the app crash?",
			"criteria": ["sales", "technical", "billing"]
		}
	}
}

ardana run decider:2b --request request.json --json

In my scorer, I match answers by question ID and probability entries by option name. Reordering stored question or option keys must leave the authored control’s results unchanged. Swapping the two answer objects still passes its structural checks, but accuracy becomes zero.

Our benchmark adapter requests one question named decision and reads answers.decision. That path does not exercise arbitrary multi-question mapping. I need a separate control to catch an answer attached to the wrong question, even when every option name is valid.

For probability losses, I read probabilities, not confidence. Choice confidence is (K*p_max-1)/(K-1), so treating it as the winning probability changes the input to the metric. I also keep hard choices separate from ordinal scores: the ordinal score is an expected level index rounded to two decimals, not a categorical pick.

My scorer needs an explicit tie policy. Ardana selects the first maximum in request order before rounding; JevBench’s probability argmax breaks ties lexicographically and does not check agreement with the returned choice. Ardana rounds output probabilities to four decimals, so their sum need not equal one exactly. I account for that rounding when checking the distribution before calculating a loss.

Check probability losses against known answers

I use the site’s unscaled multiclass Brier convention: B = (1/N) ΣᵢΣₖ(pᵢₖ − 1[yᵢ = k])². I sum squared errors across classes and average across examples, with no division by class count or by two. For one-hot categorical truth, log loss is L = −(1/N) Σᵢ ln(pᵢ,yᵢ), using natural logs. The excerpt below checks per-example distributions and losses; the question-level bookkeeping described later belongs to the longer research fixture.

My first inputs are authored distributions with answers I can check by hand. A perfect prediction has Brier and log loss both zero. A uniform three-class prediction gives 2/3 and ln(3); a certainly wrong prediction gives 2 and infinity. I leave zero-probability log loss infinite in this control, while scikit-learn clips probabilities using dtype precision. Matching its Brier calculation also requires disabling its default binary halving, so the convention belongs beside the test.

Save this complete standard-library self-test as scorer_controls.py; it is a smaller probability check drawn from the research fixture. I require exact option names, finite numeric values that are not booleans, and probabilities between zero and one. Ardana rounds probabilities to four decimals, so I allow accumulated rounding error of K × 0.00005 + 1e-12, then normalize accepted distributions. That tolerance is an authored policy, separate from JevBench’s policy. I run perfect, uniform and certainly wrong vectors in both key orders, rounded uniform once, and require deliberately missing or positional keys to fail.

import math

LABELS = ("sales", "technical", "billing")


def distribution(a):
    assert isinstance(a, dict) and a.get("type") == "choice", "answer type"
    p = a.get("probabilities")
    assert isinstance(p, dict) and set(p) == set(LABELS), "option IDs"
    assert all(type(v) in (int, float) and math.isfinite(v) and 0 <= v <= 1
               for v in p.values()), "probability values"
    total = sum(p.values())
    assert abs(total - 1) <= len(LABELS) * 0.00005 + 1e-12, "probability sum"
    assert a.get("choice") in p, "choice ID"
    assert p[a["choice"]] == max(p.values()), "choice/probability mismatch"
    return {k: p[k] / total for k in LABELS}


def losses(p, gold):
    brier = sum((p[k] - int(k == gold)) ** 2 for k in LABELS)
    nll = -math.log(p[gold]) if p[gold] > 0 else math.inf
    return brier, nll


def close(actual, expected):
    assert math.isclose(actual, expected, rel_tol=1e-12, abs_tol=1e-12)


def expect_assertion(name, action, message):
    try:
        action()
    except AssertionError as exc:
        assert str(exc) == message, (name, str(exc))
        print(f"EXPECTED ASSERTION {name}: {message}")
    else:
        raise AssertionError(f"{name}: bad fixture escaped")


for p, brier, nll in [
    ({"sales": 0, "technical": 0, "billing": 1}, 0, 0),
    ({"sales": 1 / 3, "technical": 1 / 3, "billing": 1 / 3}, 2 / 3, math.log(3)),
    ({"sales": 1, "technical": 0, "billing": 0}, 2, math.inf),
]:
    for options in (p, dict(reversed(list(p.items())))):
        a = {"type": "choice", "choice": max(p, key=p.get),
             "probabilities": options}
        b, n = losses(distribution(a), "billing")
        close(b, brier)
        close(n, nll)

uniform = {"type": "choice", "choice": "billing",
           "probabilities": {k: 0.3333 for k in LABELS}}
b, n = losses(distribution(uniform), "billing")
close(b, 2 / 3)
close(n, math.log(3))
print("PASS perfect, uniform, wrong-certain and rounded-uniform controls")

for name, p in [
    ("position IDs", {"0": 0.1, "1": 0.1, "2": 0.8}),
    ("missing option", {"billing": 0.8, "technical": 0.2}),
]:
    bad = {"type": "choice", "choice": "billing", "probabilities": p}
    expect_assertion(name, lambda: distribution(bad), "option IDs")

I run this with assertions enabled; Python’s optimization mode disables the checks. Every probability above is authored, and the command performs no model inference. The expected sample output includes rejected controls because accepting either bad map must fail the test, even when its values sum to one.

python3 scorer_controls.py

PASS perfect, uniform, wrong-certain and rounded-uniform controls
EXPECTED ASSERTION position IDs: option IDs
EXPECTED ASSERTION missing option: option IDs

Keep failed attempts in the stated denominator

For each metric, I keep the denominator beside the result. In the JevBench revision pinned by our benchmark, attempted scorable failures count as wrong for accuracy. Unattempted tasks stay out of that calculation. Brier and ECE use only scorable records with probabilities, and their sample count is reported separately:

calibration_n

The upstream test makes this difference easy to check by hand. One valid answer assigns .25 to option a and .75 to the correct option b, and a second attempt fails. Accuracy is .5, Brier is .125, and ECE is .25. Those probability metrics describe the single eligible record, so I read them with their sample count. The failed attempt contributes to the accuracy denominator without contributing a probability distribution to score.

A parsed probability dictionary can be operationally successful and still fail probability validation. Our site’s error counter checks operational status, so it does not count every distribution-contract failure. NaN or Infinity can also fail serialization before any per-item record exists. For a dictionary that reaches scoring, these fields can appear together:

status="ok"
valid=False

My authored fixture uses its own question-level policy. One missing answer leaves two attempted questions, one valid, one invalid, and one correct: accuracy .5 and Brier .06. Malformed JSON leaves zero valid and two invalid questions, accuracy zero, and no probability losses. I count questions rather than HTTP calls here, and use different probability tolerances from pinned JevBench. I state that policy with the results so a reader knows what each number counts.

Keep the controls beside the scorer

I want the controls beside the mapping and aggregation code, so changes to either run them again. Start with perfect, uniform, and certainly wrong distributions whose losses we can check by hand. Then reorder question and option keys, swap answers between questions, and remove an answer. I want a failing test when the scorer accepts the swap as correct or hides the missing answer from accuracy.

Our fixture validates only the choice fields it consumes. Full response validation and transport handling need their own tests. It also counts questions and uses its own rounding tolerance, which differs from the pinned benchmark dependency. I want those policies explicit in the tests before adapting the controls to another evaluator.

I read accuracy separately from probability losses because they answer different questions. The hard decision tells me whether the selected answer was correct. The probability losses assess the distribution assigned to the known answer. Passing these authored cases checks evaluator behavior; it does not establish model quality.

For the next comparison, I start with the Ardana repository and inspect the released API and readout. Match that contract to the fields your evaluator consumes. Add the controls at that boundary so a mapping mistake becomes a test failure before you compare models.