How to build a trustworthy classification test set

A classifier can pass without reading the input

I cannot trust a classification score if label formatting gives away the answer. In Decision Models Under Pressure, maintainer gazelle93 reports that an earlier dataset let a classifier identify answers without item text. Answer position was coupled to label identity too. The project’s v3 dataset was frozen on September 23, 2026.

Seven of its eight dataset checks passed. Without item text, the classifier still reached 0.260 on DBpedia, against chance of 0.0625 and a threshold of 0.125. I would keep that failed check beside any score I reported from this dataset.

That result identifies a residual shortcut in this evaluation. It does not establish a universal model flaw. The study also acknowledges that undisclosed Jev training data prevents it from separating stronger learning from prior exposure to its public examples.

For my own test set, I want evidence for each correct answer. Before comparing classifiers, I would record where each item came from and how its label was assigned. Without those records, I cannot tell what earned the score.

Give every label a traceable origin

I would start our label audit with the questions in Datasheets for Datasets and adapt them to our evaluation. They cover collection, processing, relationships between examples, and known errors. The questions distinguish observed, reported, and inferred information, including how inferences were verified.

For our test set, I would keep a provenance ledger beside the examples, with the source, processing history, and reason for each label. A source observation, a label inferred from it, and a human judgment would remain separate entries in that history. For an inferred label, I would record the verification; for a human judgment, the instruction the reviewer applied. Missing support would stay visible for review.

The September dataset card gives me a concrete example of traceability. It preserves original texts, records text hashes, and documents label normalization. I would carry that history into our ledger whenever we transform an example, so the reviewer can recover what changed.

With that record, our reviewer could accept a supported label, request verification, or set an example aside. Ardana’s typed request has no dedicated fields for gold labels or adjudication, so I would maintain this ledger separately. I would also record relationships between examples to help us review the split.

Split related examples together

I would compare example text for exact matches before assigning splits. Then I would review near duplicates, because small edits can leave us evaluating another version of the same example. Katherine Lee and colleagues found near duplicates and repetitive substrings in language-model datasets, and deduplication reduced overlap between training and test data.

Scikit-learn’s grouped-validation guide explains why dependent examples break the independent-sample assumption. I would keep versions of a ticket, conversation, or document in the same split, and group template families when shared wording is part of what we need to hold apart. Our grouping should follow the question: what material do we expect the model to handle unseen?

For the final evaluation, I would reserve a holdout at the same group level before tuning the model. Scikit-learn’s leakage guidance says test data must stay out of model choices and fitted preprocessing, because leakage can inflate evaluation estimates. We can use the development splits to make those choices, then evaluate the frozen setup on the holdout. I would record the grouping rule alongside the labels so a reviewer can see what the score measures.

Review the label before blaming the model

My rule is to review a disputed label before counting it as a model failure. In Pavlick and Kwiatkowski’s NLI study, human disagreement persisted after more ratings and changes to the supplied context. For textual inference, that gives us a reason to preserve plausible judgments when reviewers disagree.

I want reviewers to distinguish close alternatives from ambiguous ones. The September dataset card treats opposing actions as potentially close distractors, while excluding competing labels under its adjudicated ambiguity rules. For our own test set, I would ask reviewers to check whether the written rule separates the options. Ardana supports descriptions alongside option names, so we can spell out the distinction reviewers should apply.

For synthetic labels, I would record who generated and checked the answer. Tateno’s S1MB disclosure says GPT-6-Astra generated its 600 synthetic questions, supplied the intended answers, and checked them without independent human validation. That describes their provenance; it does not establish that the labels are wrong. I would keep conclusions scoped to the instructions, contexts, and criteria the set tests.

Our proposed review process keeps each reviewer’s judgment and reasoning before adjudication. If the rubric resolves a dispute, I would save that rationale with the label. If it does not, I would preserve the case as unresolved and report it separately. We should not turn disagreement into certainty just to finish the score.

Separate familiar domains from test exposure

I separate familiarity with a task from evidence that a model saw the exact evaluation examples. In his S1MB disclosure, Yuichi Tateno reports 77 shared subset names between the benchmark and bekko training manifests. He says bekko used training splits from datasets represented in S1MB and did not train on their test sets. The suite includes public data and synthetic tasks beyond the original datasets’ official test splits. Shared subset names alone cannot establish whether the actual evaluation examples overlap with training.

For our evaluation report, I would record task familiarity, observed example overlap, and unknown training history separately, with supporting evidence for each statement. A shared domain would get a familiarity note. For a verified example match, I would include the matching source and the scope of the check.

When training history is unavailable, I would leave that part unresolved and name exactly which material we could inspect. A search of available files would support a claim about those files. I would not extend it to training data we cannot see.

Our score report would carry those limits beside the result. I would reserve claims about unfamiliar work for evaluations designed to test that claim. We also need to specify which information reaches the model in the request we grade.

Keep the answer key outside the request

I keep the evaluation ledger outside the request I send to Ardana. Our request schema has no dedicated field for gold labels or adjudication notes. Putting them inside the JSON state still exposes them: Ardana renders object keys and values into the input used to build the prompt. I would keep those answers in the evaluator and compare them with the returned choice afterward.

Our answer key belongs with the evaluator, outside the model input.

For a reference request, I would first start the local server. These commands come from the Ardana HTTP API example. I have not run them for this article.

ardana serve

With the server running, I would send this request from another terminal. The criteria supply the available options; they do not declare which one is correct for this example. There is no measured response to report here.

curl http://127.0.0.1:8000/v1/systemone -H 'content-type: application/json' -d '{"model":"decider:2b","state":"My card was charged twice for order A-104. Please refund the duplicate.","questions":{"department":{"type":"choice","instructions":"Which department should handle this?","criteria":["billing","technical","sales"]},"refund":{"type":"noul","instructions":"Does the customer ask for a refund?"}}}'

Before the next evaluation, I would inspect the actual request body sent to the model. I would remove evaluator-only answers and review notes from that input. Try the command with a request you have checked the same way.