Blog

Instruct model vs decision model on the same questions

We sent the same typed questions to gemma-4, qwen3.5 and decider through ardana, shuffled the options, and measured where a zero-shot instruct model breaks.

A small groundhog holds the same question card up to two judges, a big chatty one and a small quiet one, and compares the scores they hold up.

Ignore the hype and train a classifier?

The best question under Strands Decider 2B on Hacker News came from weinzierl. He asked: “If I have a sizeable amount of labeled data and need decisions calibrated to that data should I 1. Ignore the hype and train a traditional classifier 2. Finetune an LLM based decision model 3. Shoehorn (probably a small subset of) the data into the context of the LLM classifier somehow”. Three options, all reasonable.

mjb, one of the authors, answered: “Yes, at this size (2B) you can get better performance and calibration fine tuning on your questions.” Then he added: “It’s a single pass through a 2B model, no reasoning, so it’s never going to be particularly ‘intelligent’.” It is an honest answer.

The classifier camp has real numbers. A February 2026 cost study (arXiv:2602.06370) measured fine-tuned BERT at 94.43 macro F1 on AG News against 87.93 for GPT-4o zero-shot, at $5.73 to $10.44 per million requests versus $276.00 to $1271.58. Alex Jacobs ran 32 experiments and found Qwen2.5-1.5B zero-shot at 93.8% on SST-2, above BERT-base at 91.5%, while “BERT is ~20x faster.” His verdict: “With a few thousand labeled examples, fine-tuned DeBERTa will beat small LLMs.”

You probably already have an instruct model on disk. Ardana sends it the exact same typed questions it sends to a trained decision model, and reads both the same way. So I measured what it does next to a trained one.

How ardana reads an instruct model it was never trained for

The layout comes from one file. A pulled model with a decider_config.json gets the plain layout, the one every decider-format model is trained on. A model without it gets the chat layout and a stock profile: temperature 1.0 for every answer type, nothing fitted. The question text is the same in both cases. Only the wrapping around it changes.

A plain row is the decider format, token for token:

Context:
<state>

Question: <q>
Options:
(A) <opt>
(B) <opt>
Answer: (

For an instruct model we render the tokenizer’s own chat template around one user message that holds a sentinel, with the generation prompt on, and split it at the sentinel into a head and a tail. If the template leaves a thinking block open, ardana closes it. Two start-up checks make sure the letters A to J stay single tokens after Answer: ( and that the tail boundary holds. On gemma-4-e2b the head decodes to <bos><|turn>user\n and the tail to <turn|>\n<|turn>model\n, so the row reads:

<bos><|turn>user
Context:
<state>

Question: <q>
Options:
(A) <opt>
(B) <opt><turn|>
<|turn>model
Answer: (

Then nothing is generated. We read the logits at that last (, keep only the option letters, and softmax them at the profile’s temperature. A noul question is just the options no and yes, and the answer is the yes probability. At 1.0 you get gemma’s raw next-letter distribution, renormalised over A, B, C.

Decider uses the same readout, its card says it plainly: letters are never generated. The difference is the numbers around it. decider:2b divides by temperatures fitted by NLL, 1.164 for choice, 1.624 for noul and 1.124 for score, and it judges every score level in its own yes/no row, where a stock model reads the levels listwise. ardana show tells you which one you got:

$ ardana show decider:2b
layout          plain (decider format)
score levels    isolated (one yes/no row per level)

$ ardana show gemma-4-e2b
layout          chat (instruct model read zero-shot)
score levels    listwise

One ticket, four models

The ardana repository has a request fixture for a support ticket, tests/fixtures/requests/ticket.json, with two questions: which department should handle it, and does the customer ask for a refund. The file names "model": "jev-latest", but the positional model of ardana run overrides it, so the identical body goes to every model. All numbers below are one run on my machine, a CPU with no GPU, ardana 0.2.2. Here is the trained model first.

ardana run decider:2b --request tests/fixtures/requests/ticket.json
Which department should handle this?
  * billing    93.2%
    technical   0.5%
    sales       6.4%
    confidence 0.90, p max 93.2%, certainty 0.76

Does the customer ask for a refund?
    probability of yes  0.93

Same file, now the stock gemma read zero-shot.

ardana run gemma-4-e2b --request tests/fixtures/requests/ticket.json
Which department should handle this?
  * billing    98.8%
    technical   1.1%
    sales       0.2%
    confidence 0.98, p max 98.8%, certainty 0.94

Does the customer ask for a refund?
    probability of yes  1.00

Both pick billing and both say yes. The difference is how sure they are. decider:2b keeps 6.4 percent on sales and puts the refund at 0.93, while gemma-4-e2b gives billing 98.8 percent and the refund 0.9977 in the JSON, which prints as 1.00. It is near-certain about everything, and that pattern repeats.

The two small models split. decider:0.8b goes even higher on billing, 99.7 percent, with the refund at 1.00. qwen3.5:0.8b still picks billing, but at 82.5 percent with 14.2 percent on technical, certainty 0.50, and a refund probability of 0.63. Three models put the refund at 0.93 or above. The small instruct model is barely past a coin flip.

The sentiment fixture, “I absolutely love this product, it works perfectly.”, is easier. decider:2b says positive at 95.7 percent with certainty 0.82. gemma-4-e2b and qwen3.5:0.8b both print positive 100.0 percent, and decider:0.8b 99.8 percent.

So billing wins on all four. But billing is also the first option in the fixture. What happens when it is not?

Shuffle the options and watch the pick

On the same thread fxwin ran the test I wanted to see. He sent the sample query from the Strands post with billing, sales and retail in five different orders. Billing won every time, but the score moved a lot: 0.843 with billing first, and 0.470 for “retail,billing,sales”, where retail sat right behind at 0.468. In his words, “it looks like there is a pretty big bias toward the first option listed”. He added later that those numbers came from the v19 checkpoint, and that “v21 is much less sensitive to ordering, but still shows a first position bias.”

So we did the same thing with ardana. The ticket’s department question went through all six orders of billing, technical and sales, one run each, for all four models. decider:2b kept p(billing) between 0.9316 and 0.9619, decider:0.8b between 0.9929 and 0.9971, and gemma-4-e2b between 0.985 and 0.9983. Then qwen3.5:0.8b swung from 0.4487 to 0.9452. In the order sales, technical, billing it picked technical at 0.4548 over billing at 0.4487, the same ticket getting a different department only because a line moved.

We also sent fxwin’s own five orders through POST /v1/systemone. On decider:2b p(billing) stayed between 0.9098 and 0.9379, on gemma-4-e2b between 0.9776 and 0.9897. Neither came anywhere near his 0.470.

None of this surprises the people behind JevBench. Their README reports one model scoring 21 percent instead of 72 percent on answer-judging items once its yes and no options were swapped, and says flat out that “small models are very sensitive to option order.” The decider card describes its training this way: “Large label sets were sub-sampled to at most 10 options per training example (gold always kept, order shuffled), so the model conditions on the supplied candidates rather than on a fixed head.” gemma-4-e2b did not flip either, but at 0.98 and up there is not much room left to move.

Thirty public items, then ten hard ones

I wanted questions somebody else wrote. JevBench publishes its items under MIT: 48 in easy.jsonl, 72 in original.jsonl, 111 in the public half of hard.jsonl. I sent the first 30 of the first two to all four models once, on this CPU, the 0.8b pair through the CLI:

ardana run decider:0.8b --request - --json

These are counts from one run. They are illustrative, not a ranking.

On easy-30, decider:2b, decider:0.8b and gemma-4-e2b each got 30 of 30, and qwen3.5:0.8b got 25. On original-30, decider:2b and gemma-4-e2b got 29, decider:0.8b got 25, and qwen3.5:0.8b fell to 19 of 30 with a mean Brier of 0.5517. Size hurts the stock model much more than the trained one.

The two bigger models missed the same item, original-policy-02-1, where the expected answer was yes. decider:2b said no at 0.6875, gemma-4-e2b said no at 0.9093. The instruct model was far more sure of its mistake.

Then I ran the first 10 items of the hard tier. decider:2b got 6 of 10. gemma-4-e2b got 1, missed all six choice items, and still gave its own pick a mean probability of 0.7914, against 0.7224 for decider:2b. On hard-opus-a-long_policy-17 it put 0.9128 on the wrong option and 0.0134 on the right one.

On hard items the zero-shot instruct model stayed confident while wrong, and a probability that does not drop when the answer is wrong is nothing to route on.

None of this replaces the real board. JevBench v1.4.2.2 puts decider-2b at rank 41 with public accuracy 0.710 and hard 0.473, next to Laya at rank 43 (0.584, hard 0.341) and GLiNER2 at rank 75 (0.580, hard 0.364). The decider card admits its own limit: “Still overconfident on hard items”, with calibration error 0.156 where the release limit is 0.08.

Why the library still lists instruct models

So why does ardana keep gemma-4, qwen3.5 and smollm3 in the library next to decider? Because the request does not change. The same /v1/systemone body goes to both kinds and the same readout comes back. Serve both and ask GET /health, and one visible difference sits in the reply:

ardana serve --port 8012 --max-loaded-models 2
curl http://localhost:8012/health
"x_temperature_by_type":{"gemma-4-e2b":{"choice":1.0,"noul":1.0,"score":1.0},"decider:2b":{"choice":1.164,"noul":1.624,"score":1.124}}

You may already have one of these models on disk, and that makes it the honest baseline to measure before you trust a trained one.

A trained model is not the last step either. mjb, one of the Strands Decider authors, said it in the same thread: at 2B “you can get better performance and calibration fine tuning on your questions”, and recalibrating alone already helps on your problem type. Start smaller than that and put both kinds on one ticket (gemma-4:e2b is the library name for the same gemma-4-e2b):

ardana run decider:2b --request tests/fixtures/requests/ticket.json
ardana run gemma-4:e2b --request tests/fixtures/requests/ticket.json

Swap the fixture for your own JSON body. Every family with its sizes is on ardana.ai/models/. Then tell me which of the two you would put in front of your agent.