ardana pull decider - Software
- Ardana Server/CLI
- Benchmark
- JevBench bb05a335, dataset dc3995d8, 231 public items, 600 s client timeout
- Measured
- 2026-10-09 to 2026-10-13, 1 run per quant.
Against the other models
Accuracy of every measured size at its default quant, best first, with decider’s sizes in ink. Then every measured quant’s accuracy against its calibration error and against its size on disk.
- jevk5:9b 0.8658
- qwen3.8:27b 0.8658
- gemma-4:26b-a4b 0.8615
- gemma-4:12b 0.8571
- qwen3.6:35b-a3b 0.8528
- jevk5:4b 0.8442
- qwen3.5:9b 0.8398
- decider:4b 0.8009
- ornith-1.5:9b 0.7879
- phi-4:14b 0.7835
- gemma-4:e4b 0.7706
- qwen3:14b 0.7706
- decider:2b 0.7662
- qwen3.5:4b 0.7662
- gemma-3:12b 0.7576
- jevk5:2b 0.7359
- qwen3:8b 0.7229
- qwen3.5:2b 0.6926
- llama-3.1:8b 0.6797
- gemma-4:e2b 0.6710
- decider:0.8b 0.6667
- ling-3.0:tiny 0.6667
- phi-4:mini 0.6537
- llama-3.2:3b 0.6494
- qwen3:4b 0.6364
- olmo-3:7b 0.6234
- smollm3:3b 0.6234
- lfm2.5:2.6b 0.6061
- gemma-3:4b 0.6017
- minicpm5:1b 0.5887
- minicpm5:2b 0.5844
- qwen3:1.7b 0.5714
- qwen3.5:0.8b 0.5671
- lfm2.5:8b-a1b 0.5584
- qwen3:0.6b 0.5498
- lfm2.5:1.2b 0.5281
- gemma-3:1b 0.5065
- llama-3.2:1b 0.5065
- gemma-4:31b 0.0000
1 run is left out: a stopped run covers fewer items, and a run without a valid distribution has no ECE.
1 run is left out: a stopped run covers fewer items, and a run without a valid distribution has no accuracy.
decider:0.8b
hf.co/ardana-ai/decider-0.8b-GGUF at fc65c88e, profile decider-0.8b-v1, plain layout
ardana pull decider:0.8b | Tag | Attempted | Accuracy, 95% interval | Brier | ECE (calibration_n) | Timeouts |
|---|---|---|---|---|---|
| decider:0.8b default decider-0.8b-Q8_0.gguf · 812 MB | 231 / 231 | 154 / 231 0.6667 0.6063–0.7246 | 0.4869 | 0.1682 (231) | 0 |
| Tag | Tier, items | Accuracy | Brier | ECE (calibration_n) |
|---|---|---|---|---|
| decider:0.8b | Easy 48 | 48 / 48 1.0000 | 0.0014 | 0.0119 (48) |
| decider:0.8b | Standard 72 | 60 / 72 0.8333 | 0.2327 | 0.0832 (72) |
| decider:0.8b | Hard 111 | 46 / 111 0.4144 | 0.8618 | 0.3382 (111) |
decider:2b
hf.co/Mapika/decider-2b-GGUF at ff2e5e68, profile decider-2b-v11, plain layout
ardana pull decider:2b | Tag | Attempted | Accuracy, 95% interval | Brier | ECE (calibration_n) | Timeouts |
|---|---|---|---|---|---|
| decider:2b default decider-2b-v11-Q4_K_M.gguf · 1.3 GB | 231 / 231 | 177 / 231 0.7662 0.7074–0.8235 | 0.3284 | 0.0759 (231) | 0 |
| decider:2b-q8_0 decider-2b-v11-Q8_0.gguf · 2.0 GB | 231 / 231 | 175 / 231 0.7576 0.6955–0.8143 | 0.3220 | 0.0683 (231) | 0 |
| decider:2b-bf16 decider-2b-v11-BF16.gguf · 3.8 GB | 231 / 231 | 176 / 231 0.7619 0.7000–0.8182 | 0.3224 | 0.0616 (231) | 0 |
| Tag | Tier, items | Accuracy | Brier | ECE (calibration_n) |
|---|---|---|---|---|
| decider:2b | Easy 48 | 48 / 48 1.0000 | 0.0019 | 0.0190 (48) |
| decider:2b | Standard 72 | 62 / 72 0.8611 | 0.2006 | 0.0866 (72) |
| decider:2b | Hard 111 | 67 / 111 0.6036 | 0.5525 | 0.1802 (111) |
| decider:2b-q8_0 | Easy 48 | 48 / 48 1.0000 | 0.0013 | 0.0158 (48) |
| decider:2b-q8_0 | Standard 72 | 64 / 72 0.8889 | 0.1729 | 0.0653 (72) |
| decider:2b-q8_0 | Hard 111 | 63 / 111 0.5676 | 0.5574 | 0.1849 (111) |
| decider:2b-bf16 | Easy 48 | 48 / 48 1.0000 | 0.0013 | 0.0157 (48) |
| decider:2b-bf16 | Standard 72 | 64 / 72 0.8889 | 0.1745 | 0.0663 (72) |
| decider:2b-bf16 | Hard 111 | 64 / 111 0.5766 | 0.5571 | 0.1747 (111) |
decider:4b
hf.co/Mapika/decider-4b-GGUF at b79f09d9, profile decider-4b-v2.1, plain layout
ardana pull decider:4b | Tag | Attempted | Accuracy, 95% interval | Brier | ECE (calibration_n) | Timeouts |
|---|---|---|---|---|---|
| decider:4b default decider-4b-v2.1-Q4_K_M.gguf · 2.7 GB | 231 / 231 | 185 / 231 0.8009 0.7477–0.8506 | 0.2656 | 0.0642 (231) | 0 |
| decider:4b-q8_0 decider-4b-v2.1-Q8_0.gguf · 4.5 GB | 231 / 231 | 191 / 231 0.8268 0.7759–0.8745 | 0.2421 | 0.0733 (231) | 0 |
| decider:4b-bf16 decider-4b-v2.1-BF16.gguf · 8.4 GB | 231 / 231 | 191 / 231 0.8268 0.7759–0.8745 | 0.2348 | 0.0727 (228) | 3 |
| Tag | Tier, items | Accuracy | Brier | ECE (calibration_n) |
|---|---|---|---|---|
| decider:4b | Easy 48 | 48 / 48 1.0000 | 0.0001 | 0.0032 (48) |
| decider:4b | Standard 72 | 69 / 72 0.9583 | 0.0689 | 0.0750 (72) |
| decider:4b | Hard 111 | 68 / 111 0.6126 | 0.5080 | 0.1749 (111) |
| decider:4b-q8_0 | Easy 48 | 48 / 48 1.0000 | 0.0000 | 0.0028 (48) |
| decider:4b-q8_0 | Standard 72 | 71 / 72 0.9861 | 0.0575 | 0.0755 (72) |
| decider:4b-q8_0 | Hard 111 | 72 / 111 0.6486 | 0.4665 | 0.1781 (111) |
| decider:4b-bf16 | Easy 48 | 48 / 48 1.0000 | 0.0000 | 0.0028 (48) |
| decider:4b-bf16 | Standard 72 | 71 / 72 0.9861 | 0.0578 | 0.0752 (72) |
| decider:4b-bf16 | Hard 111 | 72 / 111 0.6486 | 0.4571 | 0.1789 (108) |
Accuracy is correct answers over attempted items, and a request that failed or timed out counts as wrong; the interval is a 95% bootstrap over items, resampling paraphrase pairs together. Brier and ECE (top-label, ten equal bins) run over the items with a valid probability distribution, whose count sits beside every ECE. Every figure is computed by JevBench’s own scorer.
Only JevBench’s public items are used (only the public half of the hard tier), so these numbers are not comparable with its published leaderboard, which includes held-out items.