Quantizing a 4B decision model, q4_k_m against q8_0

Six items ahead at 4B, three behind at 2B

The gap between q4_k_m and q8_0 changes sign when you go from decider:2b to decider:4b. We ran six entries, three quants at each size, on the same 231 JevBench items, as a follow-up to the earlier 2B post. At 4B, q8_0 got 191 right and the default q4_k_m got 185, so q8_0 is six items ahead. At 2B, q4_k_m got 178 and q8_0 got 175, so there the lower quant is three items ahead.

Is either gap real? This run cannot say so. The 95% interval at 4B is 0.748 to 0.851 for q4_k_m and 0.776 to 0.874 for q8_0, and the two overlap. All three 2B intervals overlap too. bf16 does not break the tie: it scored 191 at 4B, level with q8_0 in every tier, and 176 at 2B.

What changed since the 2B post

The earlier post ran JevBench’s own CLI with the typesafe adapter and its fixed 120 s client timeout.

python -m jevbench.cli run --adapter typesafe

On that run decider:2b-bf16 stopped after three timeouts in a row at 166 of 231 items, and the 65 items it never attempted were all hard. The q4_k_m and q8_0 quants each lost three items at about 120.1 s. So I could only compare the 161 items all three quants answered validly: 48 easy, 72 original and 41 hard.

This time the client timeout is 600 s, because 120 s on a CPU loses long hard items. Our runner passes it to the same adapter, and the rule that three failures in a row stop a run is still JevBench’s own. All six entries are dated 2026-10-09, on ardana 0.2.2 and JevBench commit bb05a335. Each one attempted all 231 items (48 easy, 72 standard, 111 hard) and is marked complete, and no 2B quant timed out. Items go one request at a time to ardana serve with --keep-alive 2h.

ardana serve --port 8011 --default-model <model> --keep-alive 2h

Two things are missing compared to that post. The per-item answers live in a temporary directory and are discarded, so I cannot tell you which items two quants disagree on. No host is recorded either, where the earlier post named its machine.

The ardana version, the JevBench commit and the items stayed the same. Reproducing the whole family takes two commands.

npm run bench:setup
npm run bench:run -- --family decider --quants all

Three quants of decider:4b on 231 items

The 4B numbers first. q4_k_m got 185 of the 231 items right, q8_0 and bf16 got 191 each. The two interval rows are the ends of the 95% interval on accuracy.

Measure	q4_k_m	q8_0	bf16
Correct of 231	185	191	191
Accuracy	0.801	0.827	0.827
Interval low	0.748	0.776	0.776
Interval high	0.851	0.874	0.874
Brier	0.267	0.242	0.235
ECE	0.074	0.073	0.073
calibration_n	231	231	228

Where do the six items come from? The easy tier gives nothing, all three quants are 48 of 48 there. Two are standard items, 69 of 72 at q4_k_m against 71 at q8_0, and four are hard, 68 of 111 against 72. Brier follows the counts, from 0.267 down to 0.242, while ECE moves by less than 0.001.

Going from q8_0 to bf16 leaves the count of correct items the same in every tier: 48, 71 and 72. The bf16 Brier of 0.235 is over 228 items, because three hard items are missing from its calibration figures, so I do not read it against the other two. More on those three further down.

File sizes are 2.7 GB, 4.5 GB and 8.4 GB. The GGUF card calls Q8_0 “same quality as the bf16 weights”, and on item counts that is what we got. The decider-4b card lists v2.1 on JevBench public items at 1.000 / 0.986 / 0.649 for easy, standard and hard. Our q8_0 and bf16 runs land on the same counts: 48 of 48, 71 of 72 and 72 of 111!

The same quants at 2B point the other way

The same run covered the three quants of decider:2b on the same 231 items. Here are both sizes side by side. At 2B the default quant has the most correct items.

Quant	2B correct	4B correct	2B ECE
q4_k_m	178	185	0.077065
q8_0	175	191	0.068287
bf16	176	191	0.061566

At 2B, q8_0 is 3 items behind q4_k_m and bf16 is 2 behind, all on the hard tier: 66, 63 and 64 of 111. Easy stays at 48/48 and standard at 64/72 for every quant. At 4B the hard tier goes the other way, 68 for q4_k_m and 72 for q8_0. The three 2B intervals overlap, of-course, so I am not calling q4_k_m the better 2B quant either.

Calibration does not follow the accuracy. At 2B the ECE falls with precision, a spread of 0.015499 across the three quants, and the hard tier does the same: 0.201002, 0.184886, 0.174674. At 4B the spread is 0.00159, which is flat. So the 2B quant with the most correct items is also the one with the highest ECE.

What do the model cards expect? The 2B card says “Q4_K_M costs the 2B more than it costs the 4B: 0.3 points in-task and 0.5 points held-out”, and its Q4_K_M gives the same answer as bf16 on 95.86% of questions, against 97.06% on the 4B card. These 231 items do not show that cost. The 4B card measured its quants over 144,226 questions, and we have 231.

Is six items a real difference?

One item here is 1/231, or 0.4329 points overall, and 1/111, or 0.9009 points of the hard tier. Six items is about 2.6 points overall. That looks like enough to pick a quant on, but the interval says otherwise.

Accuracy comes with a 95% bootstrap interval over items, paraphrase pairs resampled together. In bench/run.py that is 2000 resamples from random.Random(0) over 195 units: 48 easy items, 36 standard pairs, 111 hard items. No tier has an interval, so per-tier gaps rest on counts alone. The q4_k_m and q8_0 intervals at 4B overlap from 0.775862 to 0.850622, which alone proves little. But each point estimate lies inside the other’s interval, and between separate studies, which ours are not, that means “P > 0.05 for the difference”.

Six items leaves each 4B score inside the other’s interval. Our benchmarks overview says “Sizes whose intervals overlap are not reliably different.” Quants get the same reading.

Would pairing change that? Both quants answer the same items, and Miller’s Adding Error Bars to Evals says it can “reduce the standard error” when question scores are positively correlated. “We therefore recommend using the paired version of the standard error estimate wherever practicable.” It gives no measured correlation, so nobody can say from this run whether pairing would settle the gap.

Here it is not practicable, and that is on our runner. The bench README is plain: “the per-item answers live in a temporary directory and are discarded”. So I cannot say which items differ, and no paired comparison can be made. It is also 1 run per quant, so run-to-run variation is unmeasured.

Three bf16 items still time out at 600 s

Raising the client timeout from JevBench’s fixed 120 s to 600 s did not remove every timeout. decider:4b-bf16 has "timeouts": 3 and the other five entries have 0. All three lost items are hard ones. They count as wrong, so they are already inside its 72 of 111 on the hard tier.

Calibration is where it gets uneven. calibration_n counts “the items that returned a valid distribution, because failures are not in it”, and for bf16 it is 228, with 108 of the 111 hard items. So its Brier and ECE are over 228 items, while the other two quants are over all 231.

Is this better than the 120 s run in the earlier post? Yes, by a lot. There 2B bf16 hit three timeouts in a row, which stops a JevBench run, and it ended at 166 of 231 items with the 65 unattempted all hard. The 2B q4_k_m and q8_0 each lost three items at about 120.1 s.

Which quant of decider:4b to pull

Skip bf16. It is 8.4 GB, it got the same 191 of 231 as q8_0, and the card describes it as “unquantized, for making other quantizations”. The card calls Q8_0 “same quality as the bf16 weights” at 4.5 GB.

ardana pull decider:4b        # q4_k_m, 2.7 GB
ardana pull decider:4b-q8_0   # 4.5 GB

So which of these two? The default is 2.7 GB, and the card puts it “about 0.2 points lower in-task accuracy”. Our six items point the same way and do not clear the interval, so I pick by disk: q8_0 where 4.5 GB is nothing, the default everywhere else. ardana show prints the same profile and the same three temperatures for all three names.

ardana show decider:4b
ardana show decider:4b-q8_0
ardana show decider:4b-bf16

For each quant, the decider benchmark page shows the correct count, accuracy with its 95% interval, Brier, ECE, calibration_n and timeouts, printed to four decimals. If six items do not convince you, good. Re-run the family and tell me what you get.

npm run bench:setup
npm run bench:run -- --family decider --quants all