ardana pull gemma-3 - Software
- Ardana Server/CLI
- Benchmark
- JevBench bb05a335, dataset dc3995d8, 231 public items, 600 s client timeout
- Measured
- 2026-10-13, 1 run per quant.
Against the other models
Accuracy of every measured size at its default quant, best first, with gemma-3’s sizes in ink. Then every measured quant’s accuracy against its calibration error and against its size on disk.
- jevk5:9b 0.8658
- qwen3.8:27b 0.8658
- gemma-4:26b-a4b 0.8615
- gemma-4:12b 0.8571
- qwen3.6:35b-a3b 0.8528
- jevk5:4b 0.8442
- qwen3.5:9b 0.8398
- decider:4b 0.8009
- ornith-1.5:9b 0.7879
- phi-4:14b 0.7835
- gemma-4:e4b 0.7706
- qwen3:14b 0.7706
- decider:2b 0.7662
- qwen3.5:4b 0.7662
- gemma-3:12b 0.7576
- jevk5:2b 0.7359
- qwen3:8b 0.7229
- qwen3.5:2b 0.6926
- llama-3.1:8b 0.6797
- gemma-4:e2b 0.6710
- decider:0.8b 0.6667
- ling-3.0:tiny 0.6667
- phi-4:mini 0.6537
- llama-3.2:3b 0.6494
- qwen3:4b 0.6364
- olmo-3:7b 0.6234
- smollm3:3b 0.6234
- lfm2.5:2.6b 0.6061
- gemma-3:4b 0.6017
- minicpm5:1b 0.5887
- minicpm5:2b 0.5844
- qwen3:1.7b 0.5714
- qwen3.5:0.8b 0.5671
- lfm2.5:8b-a1b 0.5584
- qwen3:0.6b 0.5498
- lfm2.5:1.2b 0.5281
- gemma-3:1b 0.5065
- llama-3.2:1b 0.5065
- gemma-4:31b 0.0000
1 run is left out: a stopped run covers fewer items, and a run without a valid distribution has no ECE.
1 run is left out: a stopped run covers fewer items, and a run without a valid distribution has no accuracy.
gemma-3:1b
hf.co/ggml-org/gemma-3-1b-it-GGUF at f9c28bcd, profile gemma-3:1b, chat layout
ardana pull gemma-3:1b | Tag | Attempted | Accuracy, 95% interval | Brier | ECE (calibration_n) | Timeouts |
|---|---|---|---|---|---|
| gemma-3:1b default gemma-3-1b-it-Q8_0.gguf · 1.1 GB | 231 / 231 | 117 / 231 0.5065 0.4367–0.5781 | 0.9541 | 0.4747 (231) | 0 |
| Tag | Tier, items | Accuracy | Brier | ECE (calibration_n) |
|---|---|---|---|---|
| gemma-3:1b | Easy 48 | 43 / 48 0.8958 | 0.1925 | 0.0989 (48) |
| gemma-3:1b | Standard 72 | 34 / 72 0.4722 | 1.0570 | 0.5388 (72) |
| gemma-3:1b | Hard 111 | 40 / 111 0.3604 | 1.2167 | 0.6045 (111) |
gemma-3:4b
hf.co/ggml-org/gemma-3-4b-it-GGUF at d0976223, profile gemma-3:4b, chat layout
ardana pull gemma-3:4b | Tag | Attempted | Accuracy, 95% interval | Brier | ECE (calibration_n) | Timeouts |
|---|---|---|---|---|---|
| gemma-3:4b default gemma-3-4b-it-Q4_K_M.gguf · 2.5 GB | 231 / 231 | 139 / 231 0.6017 0.5302–0.6681 | 0.7748 | 0.3900 (231) | 0 |
| Tag | Tier, items | Accuracy | Brier | ECE (calibration_n) |
|---|---|---|---|---|
| gemma-3:4b | Easy 48 | 48 / 48 1.0000 | 0.0000 | 0.0000 (48) |
| gemma-3:4b | Standard 72 | 48 / 72 0.6667 | 0.6632 | 0.3344 (72) |
| gemma-3:4b | Hard 111 | 43 / 111 0.3874 | 1.1822 | 0.5947 (111) |
gemma-3:12b
hf.co/ggml-org/gemma-3-12b-it-GGUF at ec0cbabd, profile gemma-3:12b, chat layout
ardana pull gemma-3:12b | Tag | Attempted | Accuracy, 95% interval | Brier | ECE (calibration_n) | Timeouts |
|---|---|---|---|---|---|
| gemma-3:12b default gemma-3-12b-it-Q4_K_M.gguf · 7.3 GB | 231 / 231 | 175 / 231 0.7576 0.7026–0.8130 | 0.4402 | 0.2186 (231) | 0 |
| Tag | Tier, items | Accuracy | Brier | ECE (calibration_n) |
|---|---|---|---|---|
| gemma-3:12b | Easy 48 | 48 / 48 1.0000 | 0.0000 | 0.0000 (48) |
| gemma-3:12b | Standard 72 | 64 / 72 0.8889 | 0.2093 | 0.1107 (72) |
| gemma-3:12b | Hard 111 | 63 / 111 0.5676 | 0.7804 | 0.3918 (111) |
Accuracy is correct answers over attempted items, and a request that failed or timed out counts as wrong; the interval is a 95% bootstrap over items, resampling paraphrase pairs together. Brier and ECE (top-label, ten equal bins) run over the items with a valid probability distribution, whose count sits beside every ECE. Every figure is computed by JevBench’s own scorer.
Only JevBench’s public items are used (only the public half of the hard tier), so these numbers are not comparable with its published leaderboard, which includes held-out items.