Evaluate classifiers when human annotators disagree

A majority label changes the question we measure

When I collapse subjective judgments into one majority label, I change what my evaluation measures. The winning label hides how the panel split. It also hides which reviewers consistently made different judgments.

I need to declare which human target I want. An adjudicated policy target asks whether a decision follows our resolved interpretation of the rules. An empirical panel target represents the label distribution among the people who judged the case. A reviewer perspective target represents an individual reviewer’s judgments. Those systematic differences cannot be recovered from a per-item histogram.

Baan et al. show that majority-based calibration can penalize predictions matching human disagreement. Their empirical case study uses ChaosNLI-SNLI, which does not establish the same result for arbitrary policy decisions. I keep that scope explicit when choosing how to score our classifier.

I declare what our evaluation target means when deciding what the model should match. Neither the majority vote nor the distribution of votes establishes objective correctness on its own.

Keep the judgments before choosing an aggregate

I keep a caller-owned annotation ledger before choosing an aggregate. Each row preserves the case, pseudonymous reviewer identifier, rubric version, label, status and reason. We keep original judgments separate from corrections and adjudication. This row is synthetic:

{
	"case_id": "demo-1",
	"annotator_id": "r1",
	"rubric_version": "priority-v1",
	"label": "routine",
	"status": "valid",
	"reason": null
}

Our protocol collects judgments independently, before showing predictions or other votes. We keep current-case votes and adjudication out of inference inputs. Following Prabhakaran et al., I document reviewer recruitment, selection and assignment, with demographic information only when collecting it is responsibly feasible.

A per-item histogram gives us more than a majority label. It preserves the label split, but cannot recover systematic differences between individual reviewers. I retain the individual records so we can examine those patterns, as Prabhakaran et al. recommend.

In safety annotations, Oesterling et al. distinguish task-execution failures, ambiguous policy and differing safety perspectives. I would address execution failures with quality control and ambiguity with clearer policy. Differing perspectives need deliberation about which perspectives to incorporate. The authors’ models expose patterns, without establishing why people differ.

Before aggregating, I declare how reviewers were sampled and whether we weight each item or each annotation event equally. With equal-item weighting, each case contributes equally; with annotation-event weighting, cases with more recorded judgments contribute more. That choice is part of what our reported result means. We preserve missing-label and failure counts, flag majority ties, and keep hard agreement separate from distribution distance.

Collect the named probability vector

I use Ardana here to collect a frozen task prediction. The annotation ledger and evaluator stay in our code. Our inference inputs exclude current-case votes and adjudication, and reviewers judge independently before seeing predictions or other votes.

On macOS or Linux, I install Ardana and start the local server with these commands. The installer can change, so it does not pin a release. This example follows the API shipped in v0.3.2.

curl -fsSL https://ardana.ai/install.sh | sh
ardana serve

With the server running, I save the complete request I wrote as request.json. It includes the message, rubric instructions and both named criteria. The first request downloads the library model unless it has already been pulled.

{
	"model": "decider:2b",
	"state": "The export fails intermittently. A workaround exists. No deadline is stated.",
	"questions": {
		"priority": {
			"type": "choice",
			"instructions": "Apply priority-v1. Which priority does this message warrant?",
			"criteria": {
				"routine": "May wait until the next business day.",
				"urgent": "Requires human review before the next business day."
			}
		}
	}
}

From another terminal, I capture the response in decision.json and keep the request beside it. That gives our evaluation a frozen input and prediction. I keep the full response, including the reported model and token usage, when extracting the vector.

curl http://127.0.0.1:8000/v1/systemone -H 'content-type: application/json' -d @request.json > decision.json

For comparison with human frequencies, I read the named vector at the path below. Confidence rescales the largest option probability according to the option count. It does not retain the probability assigned to each label. I keep the returned choice too: Ardana selects it before rounding probabilities to four decimals, and their serialized sum can differ from one.

answers["priority"]["probabilities"]

I record the installed binary version and actual model artifact with the frozen pair for provenance. Our local catalog defaults to decider:2b with q4_k_m, at the model revision below. The example follows the shipped source commit below. I keep the actual artifact identity even when I use the catalog default.

Model revision: ff2e5e687327eda9ac34e9a3ca84d3f400672c87
Shipped source commit: 445fbc945b02b641eeaef65a3ae0f0252f571aba

Compare distributions as well as winning labels

I use five invented judgments to make the difference visible. In our synthetic panel, r1, r2 and r3 choose routine, while r4 and r5 choose urgent. That gives routine 3 votes and urgent 2, so the empirical distribution assigns routine 0.6 and urgent 0.4. These are label frequencies for this panel, computed from each count divided by the number of judgments.

Against that target, I compare two authored candidate vectors, A and B. These are synthetic inputs, never recorded Ardana responses. Both select routine and score 1 for majority agreement, but I also compare the full vectors using total variation.

TV(p,q) = 0.5 * sum(abs(p[k] - q[k]))

Candidate	Routine	Urgent	Agreement	TV
A	0.99	0.01	1	0.39
B	0.6	0.4	1	0

Total variation ranges from 0 to 1. I get 0.39 for A and 0 for B, even though their winning labels are identical. B reproduces the observed split; A assigns almost all probability to routine. Our majority score has no way to express that difference.

In Baan et al.’s EMNLP 2022 study, majority-based calibration can penalize predictions matching human disagreement. Their empirical case study uses ChaosNLI-SNLI. I do not treat that result as evidence about arbitrary policy decisions or Ardana performance.

I interpret the distance with the panel’s finite size in mind. Their estimate assumes independent judgments from the same categorical distribution, and finite counts limit its precision. Matching our five invented votes establishes neither objective correctness nor population calibration. I need to say whose judgments this panel is meant to represent before treating a lower distance as useful.

Write down whose judgment the model should match

For the next evaluation, I want a written target: the empirical judgments of a declared reviewer population. We name who belongs in that population and document recruitment, selection and assignment. We keep the original judgments with pseudonymous reviewer identities, rubric versions, statuses and reasons. Corrections and adjudication stay separate.

Before anyone sees predictions or other votes, we collect each judgment independently. We keep current-case votes and adjudication out of inference inputs. I give each case equal weight and record that choice, so extra annotation events do not silently change what we measure. We report majority agreement and distribution distance separately, with majority ties flagged and missing labels and failures counted.

I keep this evaluator and its annotation ledger in our calling code. We use Ardana to collect frozen local predictions, then compare them against our declared human target. Agreement with this panel does not establish objective correctness. To try the frozen local request above in your own evaluation, start with the Ardana repository.