How to test text classifiers across languages

What an English score leaves unanswered

When I move a classifier from English to another language, I need more than its English score. I need to know whether it still makes the same decision under the same label policy. A different language suite can change the tasks too.

AIB’s DecisionBench-KJ has 2,000 Korean and Japanese items from eight dataset families. Its Korean accuracy covers seven datasets, while Japanese covers five, with equal dataset weighting within each language. The collections include language-specific tasks, rather than parallel translations. I would not read the gap between those totals as a language effect.

S1MB uses different scopes too. English v1 covers 137 benchmarks across 106 dataset subsets, while Japanese v1 covers 27 benchmarks and 8,067 judgments. Its Japanese evaluation data is private, so I cannot reproduce it from public results alone. S1MB also says its scores do not establish performance on unseen tasks or rule out training overlap. For our own evaluation, I would first fix the task and what each label means, then test that decision in each language.

Start with the label policy in each language

I would start with a label policy for every language I plan to test. What belongs in each class, and what happens when an input is ambiguous? Proficient reviewers would check those boundaries in each language. I would resolve disagreements before treating an example as a test case.

Parallel translations let me compare decisions on corresponding content across languages. I would keep native examples separate to test material originally written in each language. Translation can change the task’s cues: Artetxe and colleagues found that translating an inference premise and hypothesis independently can reduce lexical overlap and affect the studied models. I would retain their distinction between original, human-translated and machine-translated data.

JGLUE was built from Japanese material without translation. Its historical MARC-ja policy assigned negative sentiment to one- and two-star reviews and positive sentiment to four- and five-star reviews. It excluded three-star reviews. Development and test selection also required agreement from at least seven of ten crowdworkers. MARC-ja is now unavailable because distribution of its underlying MARC data stopped.

For my own evaluation, that history makes the policy part of the task definition. I would have reviewers check whether translations preserve the intended label and record wording they find ambiguous. The accepted examples would stay with their label rules and the decisions about which cases to exclude. With those meanings settled, I can vary the language used around the input.

Change one language variable at a time

I would start with an English baseline and hold the model, task, label policy, and option order fixed. My first variant would translate only the text being classified. A second would translate only the question instructions, and a third only the option descriptions. Each variant would start from the same baseline, so we could inspect that change without carrying over another.

Our comparison would change one language variable at a time before testing the combinations we expect to use.

Once those comparisons are available, I would test combinations that match the intended use. That could mean input in one language with instructions and descriptions in another. We would keep parallel translations separate from examples originally written in the target language. I would also distinguish human and machine translations so we know how the text in each comparison was produced.

Mixed-language input needs its own evaluation. LinCE studies code-switching, where languages alternate within a sentence or utterance, across four language pairs and four tasks, including sentiment. Its authors caution that narrow task, domain, and language coverage leaves broader generalization unclear. I would therefore collect a separate set reflecting the language combinations and switching patterns we expect to handle. We would evaluate that set directly before drawing conclusions about mixed-language quality.

For Ardana, I would make these changes in the input text, question instructions, and option descriptions. The request has no separate language selector. Described options let us retain the answer identifiers while changing the text that explains each choice. I would keep those identifiers fixed across every variant so our comparison still refers to the same decisions.

Keep answer identifiers stable

I would keep answer identifiers fixed across every language variant. Ardana accepts choice options as a map from names to descriptions, and preserves those names as the returned answers. I can translate a description while keeping the key that my application uses. That gives the evaluator the same label identifiers to compare across requests.

Here is unexecuted synthetic input to save as request.json. I wrote it to pair Spanish message text with English instructions and option descriptions. The request shape is checked against the schema, but I have no observed response for this fixture. It is not native customer data or evidence of multilingual performance.

{
	"model": "decider:2b",
	"state": "Me cobraron dos veces. Quiero un reembolso.",
	"questions": {
		"department": {
			"type": "choice",
			"instructions": "Which department should handle this message?",
			"criteria": {
				"billing": "Payments, charges and refunds",
				"technical": "Errors using the product",
				"sales": "Questions about buying the product"
			}
		}
	}
}

The request type has no separate language field. I put the message in the state and define the decision through the question’s instructions and criteria. For a description-language experiment, I would change only the map values, leaving the answer keys and the message fixed.

To run this locally, I would start the server in one terminal. It accepts requests at the endpoint used below. If the named model has not been pulled, the first request downloads it automatically.

ardana serve

With the server running, I would send the saved file from another terminal. The model named in the fixture is the library default. I would still evaluate its performance separately in each language.

curl http://127.0.0.1:8000/v1/systemone -H 'content-type: application/json' -d @request.json

For a choice answer, the response schema includes these fields. I would retain the selected option and full probability distribution. The choice is the most probable option, and the maximum-probability field holds its probability. Confidence is computed from that probability and the number of options, so I would not read it as the probability that the answer is correct.

choice
probabilities
confidence
x_p_max
x_certainty

Read each language and task separately

I would give each task and language its own report row. Keep the sample count, class counts, wrong answers, missing responses, and request failures beside the score. Each case should retain its input source and whether it is synthetic, with model and dataset revisions attached to the run. I would also record the inference settings so we know what produced those answers.

For error review, I would save the full option distribution beside the expected and selected labels. That gives us something concrete to inspect when the selected answer is wrong. Failed requests should remain visible in the report, with the number of completed cases stated explicitly.

I would not interpret a difference between unmatched task collections as a language effect. S1MB’s Japanese data requires authorized access, so we cannot validate individual predictions from its public results alone. AIB also warns about possible training exposure; I would mark that exposure unknown when the evidence is missing.

My next step would be to commit the synthetic request above as a fixture in the evaluation repository. It has no observed response yet, so I would leave its result empty until we run it. Add the expected label and provenance beside it, then save the actual response and run revisions with that case.