ardana pull qwen3 - Software
- Ardana Server/CLI
- Benchmark
- JevBench bb05a335, dataset dc3995d8, 231 public items, 600 s client timeout
- Measured
- 2026-10-13, 1 run per quant.
Against the other models
Accuracy of every measured size at its default quant, best first, with qwen3’s sizes in ink. Then every measured quant’s accuracy against its calibration error and against its size on disk.
- jevk5:9b 0.8658
- qwen3.8:27b 0.8658
- gemma-4:26b-a4b 0.8615
- gemma-4:12b 0.8571
- qwen3.6:35b-a3b 0.8528
- jevk5:4b 0.8442
- qwen3.5:9b 0.8398
- decider:4b 0.8009
- ornith-1.5:9b 0.7879
- phi-4:14b 0.7835
- gemma-4:e4b 0.7706
- qwen3:14b 0.7706
- decider:2b 0.7662
- qwen3.5:4b 0.7662
- gemma-3:12b 0.7576
- jevk5:2b 0.7359
- qwen3:8b 0.7229
- qwen3.5:2b 0.6926
- llama-3.1:8b 0.6797
- gemma-4:e2b 0.6710
- decider:0.8b 0.6667
- ling-3.0:tiny 0.6667
- phi-4:mini 0.6537
- llama-3.2:3b 0.6494
- qwen3:4b 0.6364
- olmo-3:7b 0.6234
- smollm3:3b 0.6234
- lfm2.5:2.6b 0.6061
- gemma-3:4b 0.6017
- minicpm5:1b 0.5887
- minicpm5:2b 0.5844
- qwen3:1.7b 0.5714
- qwen3.5:0.8b 0.5671
- lfm2.5:8b-a1b 0.5584
- qwen3:0.6b 0.5498
- lfm2.5:1.2b 0.5281
- gemma-3:1b 0.5065
- llama-3.2:1b 0.5065
- gemma-4:31b 0.0000
1 run is left out: a stopped run covers fewer items, and a run without a valid distribution has no ECE.
1 run is left out: a stopped run covers fewer items, and a run without a valid distribution has no accuracy.
qwen3:0.6b
hf.co/Qwen/Qwen3-0.6B-GGUF at 23749fef, profile qwen3:0.6b, chat layout
ardana pull qwen3:0.6b | Tag | Attempted | Accuracy, 95% interval | Brier | ECE (calibration_n) | Timeouts |
|---|---|---|---|---|---|
| qwen3:0.6b default Qwen3-0.6B-Q8_0.gguf · 639 MB | 231 / 231 | 127 / 231 0.5498 0.4805–0.6174 | 0.6945 | 0.2919 (231) | 0 |
| Tag | Tier, items | Accuracy | Brier | ECE (calibration_n) |
|---|---|---|---|---|
| qwen3:0.6b | Easy 48 | 45 / 48 0.9375 | 0.0852 | 0.0394 (48) |
| qwen3:0.6b | Standard 72 | 39 / 72 0.5417 | 0.7351 | 0.3197 (72) |
| qwen3:0.6b | Hard 111 | 43 / 111 0.3874 | 0.9316 | 0.3984 (111) |
qwen3:1.7b
hf.co/Qwen/Qwen3-1.7B-GGUF at 90862c4b, profile qwen3:1.7b, chat layout
ardana pull qwen3:1.7b | Tag | Attempted | Accuracy, 95% interval | Brier | ECE (calibration_n) | Timeouts |
|---|---|---|---|---|---|
| qwen3:1.7b default Qwen3-1.7B-Q8_0.gguf · 1.8 GB | 231 / 231 | 132 / 231 0.5714 0.5022–0.6376 | 0.7859 | 0.3867 (231) | 0 |
| Tag | Tier, items | Accuracy | Brier | ECE (calibration_n) |
|---|---|---|---|---|
| qwen3:1.7b | Easy 48 | 47 / 48 0.9792 | 0.0417 | 0.0208 (48) |
| qwen3:1.7b | Standard 72 | 40 / 72 0.5556 | 0.8329 | 0.4270 (72) |
| qwen3:1.7b | Hard 111 | 45 / 111 0.4054 | 1.0773 | 0.5427 (111) |
qwen3:4b
hf.co/Qwen/Qwen3-4B-GGUF at bc640142, profile qwen3:4b, chat layout
ardana pull qwen3:4b | Tag | Attempted | Accuracy, 95% interval | Brier | ECE (calibration_n) | Timeouts |
|---|---|---|---|---|---|
| qwen3:4b default Qwen3-4B-Q4_K_M.gguf · 2.5 GB | 231 / 231 | 147 / 231 0.6364 0.5683–0.6966 | 0.6834 | 0.3314 (231) | 0 |
| Tag | Tier, items | Accuracy | Brier | ECE (calibration_n) |
|---|---|---|---|---|
| qwen3:4b | Easy 48 | 48 / 48 1.0000 | 0.0000 | 0.0000 (48) |
| qwen3:4b | Standard 72 | 46 / 72 0.6389 | 0.7024 | 0.3630 (72) |
| qwen3:4b | Hard 111 | 53 / 111 0.4775 | 0.9666 | 0.4542 (111) |
qwen3:8b
hf.co/Qwen/Qwen3-8B-GGUF at 7c41481f, profile qwen3:8b, chat layout
ardana pull qwen3:8b | Tag | Attempted | Accuracy, 95% interval | Brier | ECE (calibration_n) | Timeouts |
|---|---|---|---|---|---|
| qwen3:8b default Qwen3-8B-Q4_K_M.gguf · 5.0 GB | 231 / 231 | 167 / 231 0.7229 0.6622–0.7835 | 0.4851 | 0.2245 (231) | 0 |
| Tag | Tier, items | Accuracy | Brier | ECE (calibration_n) |
|---|---|---|---|---|
| qwen3:8b | Easy 48 | 48 / 48 1.0000 | 0.0000 | 0.0000 (48) |
| qwen3:8b | Standard 72 | 57 / 72 0.7917 | 0.3587 | 0.1789 (72) |
| qwen3:8b | Hard 111 | 62 / 111 0.5586 | 0.7768 | 0.3533 (111) |
qwen3:14b
hf.co/Qwen/Qwen3-14B-GGUF at 530227a7, profile qwen3:14b, chat layout
ardana pull qwen3:14b | Tag | Attempted | Accuracy, 95% interval | Brier | ECE (calibration_n) | Timeouts |
|---|---|---|---|---|---|
| qwen3:14b default Qwen3-14B-Q4_K_M.gguf · 9.0 GB | 231 / 231 | 178 / 231 0.7706 0.7130–0.8246 | 0.4066 | 0.1864 (231) | 0 |
| Tag | Tier, items | Accuracy | Brier | ECE (calibration_n) |
|---|---|---|---|---|
| qwen3:14b | Easy 48 | 48 / 48 1.0000 | 0.0000 | 0.0000 (48) |
| qwen3:14b | Standard 72 | 65 / 72 0.9028 | 0.1653 | 0.0866 (72) |
| qwen3:14b | Hard 111 | 65 / 111 0.5856 | 0.7389 | 0.3337 (111) |
Accuracy is correct answers over attempted items, and a request that failed or timed out counts as wrong; the interval is a 95% bootstrap over items, resampling paraphrase pairs together. Brier and ECE (top-label, ten equal bins) run over the items with a valid probability distribution, whose count sits beside every ECE. Every figure is computed by JevBench’s own scorer.
Only JevBench’s public items are used (only the public half of the hard tier), so these numbers are not comparable with its published leaderboard, which includes held-out items.