Does q4_k_m hurt a decision model's calibration?

Three files of the same model, and one question

When you pull decider:2b in Ardana you get the q4_k_m file by default, 1.3 GB on disk. The same model also ships as q8_0 at 2.0 GB and as bf16 at 3.8 GB. All three are decider v11 from Mapika’s GGUF repository, the same 1.9B parameters stored at different precision. So which one should you run?

Mapika’s card answers in accuracy first. Q4_K_M is “0.3 points lower in-task, 0.5 held-out”, and Q8_0 is “same quality as the bf16 weights”. The measured table goes further: over 144,226 questions the in-task ECE is 0.0383 for bf16, 0.0381 for Q8_0 and 0.0385 for Q4_K_M, and Q4_K_M gives the same answer as bf16 on 95.86% of them. Those numbers come from a CUDA llama.cpp build compared with bf16 PyTorch, on the card’s own 95 tasks.

For a decision model the calibration is the product. On the JevBench Show HN thread one commenter put it short: “confident and wrong is worse than an LLM that just hedges.” A 0.9 has to mean right nine times of ten, or the agent reading it acts wrong with full confidence. We wanted to see the per-quant difference on JevBench’s public items, against a local server like the one you run. So we ran it, all three quants.

Running JevBench against ardana serve

JevBench is Benchmark Heaven’s benchmark, an MIT harness at github.com/fstandhartinger/jevbench, and it says plainly it is not affiliated with or endorsed by TypeSafe AI. I used the clone at commit bb05a335 against ardana 0.2.2. Its public split is three files: easy.jsonl with 48 items, original.jsonl with 72 and hard.jsonl with 111, the public half of the hard tier. That is 231 items in all, 139 choice questions, 74 noul and 18 score.

The whole run is three commands per quant. --default-model is “the model for requests that name none (or Jev’s jev-latest)”, so I start the server on the quant I want to measure and point the harness at it:

TASKS=datasets/public/easy.jsonl,datasets/public/original.jsonl,datasets/public/hard.jsonl

ardana serve --port 8011 --default-model "$MODEL"

python -m jevbench.cli run --adapter typesafe --key-env "" --endpoint http://127.0.0.1:8011 --model "$MODEL" --tasks "$TASKS" --results $OUT/results.jsonl --ledger $OUT/ledger.jsonl --raw-dir $OUT/raw --manifest $OUT/manifest.json --run-label "$LABEL"

python -m jevbench.cli summarize --tasks "$TASKS" --results $OUT/results.jsonl --ledger $OUT/ledger.jsonl > $OUT/summary.json

MODEL is decider:2b for q4_k_m and decider:2b-q8_0 for q8_0, and the bf16 file is decider:2b-bf16. One server lists every quant anyway, so --model names the file too. The typesafe adapter posts {"state": ..., "model": ..., "questions": {"decision": {...}}} to /v1/systemone. It maps a noul answer to {"yes": p, "no": 1-p} and takes choice and score probabilities as they come. With --key-env "" it sends no Authorization header, which is what a local server wants.

Does the quant change the calibration settings? No. ardana show prints profile decider-2b-v11 and runtime llama.cpp for all three names, and this is the Calibration block for q8_0:

$ ardana show decider:2b-q8_0
choice temperature 1.164
noul temperature 1.624
score temperature 1.124
score levels isolated (one yes/no row per level)

decider:2b and decider:2b-bf16 print the same block. The quant changes the weights, nothing else, so any difference in the numbers below comes from the weights.

We do the same thing inside the product repo:

cargo xtask e2e jevbench

It builds ardana, pulls decider:2b into a sandbox home, starts ardana serve, runs the same typesafe command over easy, original and hard, then summarize, with output in tmp/evals/jevbench/. It fails on any failed or missing request. Accuracy is reported, not gated.

What the calibration number measures

Accuracy tells you how often the model is right. Calibration tells you whether its probabilities mean what they say. JevBench takes the top-label confidence of each answer, max(p.values()), and counts the item correct when that argmax equals the gold label. Then it puts every item into one of ten equal-width confidence bins and compares how sure the model said it was with how often it was right.

ECE = Σ (n_bin/n)·|acc_bin − mean_conf_bin|

Each bin’s gap is weighted by how many items fall into it, and empty bins are skipped. Brier is the multi-class sum Σ_k (p_k − y_k)² over the label set, so it punishes a confident wrong answer and a timid right one. Failures carry no probabilities, so they drop out of both numbers, and the summary reports that denominator as calibration_n. Accuracy still divides by all items.

Here is the q4_k_m run over the full 231 items, with calibration_n 228:

bin        items  accuracy  mean confidence
0.3–0.4        3     0.000            0.372
0.4–0.5        8     0.125            0.448
0.5–0.6       19     0.579            0.550
0.6–0.7       24     0.750            0.652
0.7–0.8       35     0.600            0.747
0.8–0.9       39     0.769            0.844
0.9–1.0      100     0.950            0.969

Look at the last row. 100 items sit in the top bin, the model said 0.969 on average and was right 0.950 of the time. That is what calibrated looks like! The middle is rougher: at 0.7 to 0.8 it said 0.747 and was right only 0.600. Over the whole run that gives ECE 0.073 and Brier 0.326, with 176 of 231 correct.

Three quants on the same 161 items

The full runs are not comparable, because bf16 stopped early. So I took the 161 items all three quants answered validly: 48 easy, 72 original and 41 hard, exactly the set bf16 answered. Same items, same temperatures, same server, only the weights change.

Quant	Correct	Accuracy	Brier	ECE
q4_k_m	137	0.8509	0.2298	0.0451
q8_0	133	0.8261	0.2321	0.0359
bf16	133	0.8261	0.2330	0.0333

Hard tier only:

Quant	Correct	ECE
q4_k_m	25	0.2333
q8_0	21	0.1999
bf16	21	0.2000

Easy is 1.0000 and original 0.8889 at every quant. ECE goes 0.045, 0.036, 0.033 from the smallest file to the largest, so q4_k_m costs about one hundredth of calibration. The hard tier sits near 0.20 at every quant: that overconfidence belongs to the model, not to the file.

Agreement tells more than the averages. q8_0 and bf16 picked the same answer on all 161 items, with a mean difference in top-label probability of 0.0050. In practice q8_0 is bf16. q4_k_m agrees with bf16 on 152 of 161, which is 94.41 percent, with a mean difference of 0.0361. On the nine splits q4_k_m matches gold on 6 and bf16 on 2, and neither was sure: top probability at most 0.702 for q4_k_m and 0.768 for bf16.

Quantization flips the answers the model was already unsure about. The confident ones stay put.

The card measured 99.16 and 95.86 percent same answer over 144,226 questions. We got 100 and 94.41 over 161, which lines up well. The card also says Q4_K_M is 0.3 points lower in-task, while q4_k_m is 4 items ahead of bf16, all from the hard tier. One item is 0.62 points, and this is one run on a small sample, so I do not read that lead as q4_k_m being better.

The CPU trap is the timeout, not the quant

Why does bf16 have no full run? It was not wrong. It was late. JevBench waits a fixed 120 seconds per request, each failed record adds one to a counter, each good answer resets it, and three in a row stop the run with STOP: access/rate limit or consecutive infrastructure errors; remaining tasks unattempted.

On this AMD Ryzen 7 3800X, 8 cores, 16 threads and no GPU, bf16 timed out at positions 164 to 166, on hard-opus-c-long_policy-05, -08 and -10. That ended the run at 166 of 231 items. The 65 it never attempted are all hard tier.

The other two quants lost the same three items, multi_hop-05, long_policy-05 and long_policy-08, at about 120.1 seconds each. But their long_policy-10 came back in 59.0 seconds on q4_k_m and 115.2 on q8_0, which reset the counter. So q8_0 got its full run with less than five seconds to spare.

Wall time was 1,899 seconds for q4_k_m over 231, 2,139 for q8_0 over 231 and 2,116 for bf16 over 166. Other work shared the CPU during the q4_k_m and q8_0 runs, so this is no benchmark. Still, bf16 needed 66 to 79 seconds on the four longest long_policy items it answered, where q8_0 needed 40 to 49.

On a CPU, the 120 second timeout decides which items a quant gets to answer, long before calibration does.

None of this was a long state: the longest answered input was 3,911 tokens. The card warns about one 32,794-token state, where BF16 and Q8_0 kept the PyTorch answer and Q4_K_M chose another option at 0.53.

Which quant to pull

Mapika’s card says it plainly: use Q8_0 unless the 1.3 GB file matters. Our run agrees. On the items all three quants answered, q8_0 gave the same answers as bf16 with almost the same probabilities, so the 3.8 GB file buys nothing for serving decisions. It is there for making other quantizations.

q4_k_m is close enough on short decisions. The card is honest that it is lower than bf16 on 59 tasks, with paws dropping 4.5 points. If disk or RAM is tight, take it and don’t worry. If not, q8_0 at 2.0 GB is absolute no brainer.

The hard tier is a limit of the model itself, and no quant fixes it. On our full run q8_0 got 61 of 111 hard items right with an ECE of 0.1915, and the card itself says “Still overconfident on hard items”.

ardana pull decider:2b-q8_0

Have your own GGUF? The same three commands run it against ardana serve. Send me the numbers, I want to see them!