Benchmarks

gemma-4

Gemma 4, a stock instruct model read zero-shot in the chat layout.

Model page
ardana pull gemma-4
Benchmark
JevBench bb05a335, dataset dc3995d8, 231 public items, 600 s client timeout
Measured
2026-10-11 to 2026-10-13, 1 run per quant.

Against the other models

Accuracy of every measured size at its default quant, best first, with gemma-4’s sizes in ink. Then every measured quant’s accuracy against its calibration error and against its size on disk.

  1. jevk5:9b 0.8658
  2. qwen3.8:27b 0.8658
  3. gemma-4:26b-a4b 0.8615
  4. gemma-4:12b 0.8571
  5. qwen3.6:35b-a3b 0.8528
  6. jevk5:4b 0.8442
  7. qwen3.5:9b 0.8398
  8. decider:4b 0.8009
  9. ornith-1.5:9b 0.7879
  10. phi-4:14b 0.7835
  11. gemma-4:e4b 0.7706
  12. qwen3:14b 0.7706
  13. decider:2b 0.7662
  14. qwen3.5:4b 0.7662
  15. gemma-3:12b 0.7576
  16. jevk5:2b 0.7359
  17. qwen3:8b 0.7229
  18. qwen3.5:2b 0.6926
  19. llama-3.1:8b 0.6797
  20. gemma-4:e2b 0.6710
  21. decider:0.8b 0.6667
  22. ling-3.0:tiny 0.6667
  23. phi-4:mini 0.6537
  24. llama-3.2:3b 0.6494
  25. qwen3:4b 0.6364
  26. olmo-3:7b 0.6234
  27. smollm3:3b 0.6234
  28. lfm2.5:2.6b 0.6061
  29. gemma-3:4b 0.6017
  30. minicpm5:1b 0.5887
  31. minicpm5:2b 0.5844
  32. qwen3:1.7b 0.5714
  33. qwen3.5:0.8b 0.5671
  34. lfm2.5:8b-a1b 0.5584
  35. qwen3:0.6b 0.5498
  36. lfm2.5:1.2b 0.5281
  37. gemma-3:1b 0.5065
  38. llama-3.2:1b 0.5065
  39. gemma-4:31b 0.0000
Most attractive quadrant0.600.800.000.200.40ECE (left is better calibrated)accuracy (up is better)gemma-4:e2bgemma-4:e4bgemma-4:12bgemma-4:26b-a4bdecider:0.8bdecider:2bdecider:4bgemma-3:1bgemma-3:4bgemma-3:12bjevk5:2blfm2.5:1.2blfm2.5:2.6blfm2.5:8b-a1bling-3.0:tinyllama-3.1:8bllama-3.2:1bllama-3.2:3bminicpm5:1bminicpm5:2bolmo-3:7bphi-4:14bqwen3:0.6bqwen3:1.7bqwen3:4bqwen3:8bqwen3:14bqwen3.5:0.8bqwen3.5:2b

1 run is left out: a stopped run covers fewer items, and a run without a valid distribution has no ECE.

Most attractive quadrant0.600.800.5 GB1 GB2 GB5 GB10 GB20 GBsize on disk, log scale (left is smaller)accuracy (up is better)gemma-4:e2bgemma-4:e4bgemma-4:12bgemma-4:26b-a4bdecider:0.8bdecider:2bjevk5:4bjevk5:9bqwen3.5:0.8bdecider:4bgemma-3:1bgemma-3:4bgemma-3:12bjevk5:2blfm2.5:1.2blfm2.5:2.6blfm2.5:8b-a1bling-3.0:tinyllama-3.1:8bllama-3.2:1bllama-3.2:3bminicpm5:1bminicpm5:2bolmo-3:7bphi-4:miniphi-4:14bqwen3:0.6bqwen3:1.7bqwen3:8bqwen3:14bqwen3.6:35b-a3bqwen3.8:27b

1 run is left out: a stopped run covers fewer items, and a run without a valid distribution has no accuracy.

gemma-4:e2b

hf.co/ggml-org/gemma-4-E2B-it-GGUF at b4243c15, profile gemma-4:e2b, chat layout

ardana pull gemma-4:e2b
All 231 items
TagAttemptedAccuracy, 95% intervalBrierECE (calibration_n)Timeouts
gemma-4:e2b default gemma-4-E2B-it-Q4_0.gguf · 2.8 GB231 / 231155 / 231 0.6710 0.6035–0.73140.47740.2055 (231)0
0.000.250.500.751.00gemma-4:e2b
By tier
TagTier, itemsAccuracyBrierECE (calibration_n)
gemma-4:e2b Easy 4848 / 48 1.00000.00020.0045 (48)
gemma-4:e2b Standard 7266 / 72 0.91670.15220.0518 (72)
gemma-4:e2b Hard 11141 / 111 0.36940.89470.4267 (111)

gemma-4:e4b

hf.co/ggml-org/gemma-4-E4B-it-GGUF at b8093469, profile gemma-4:e4b, chat layout

ardana pull gemma-4:e4b
All 231 items
TagAttemptedAccuracy, 95% intervalBrierECE (calibration_n)Timeouts
gemma-4:e4b default gemma-4-E4B-it-Q4_0.gguf · 4.6 GB231 / 231178 / 231 0.7706 0.7124–0.82520.32770.1287 (231)0
0.000.250.500.751.00gemma-4:e4b
By tier
TagTier, itemsAccuracyBrierECE (calibration_n)
gemma-4:e4b Easy 4848 / 48 1.00000.00000.0006 (48)
gemma-4:e4b Standard 7267 / 72 0.93060.10360.0509 (72)
gemma-4:e4b Hard 11163 / 111 0.56760.61470.2523 (111)

gemma-4:12b

hf.co/ggml-org/gemma-4-12B-it-GGUF at e3e68173, profile gemma-4:12b, chat layout

ardana pull gemma-4:12b
All 231 items
TagAttemptedAccuracy, 95% intervalBrierECE (calibration_n)Timeouts
gemma-4:12b default gemma-4-12B-it-Q4_0.gguf · 7.2 GB231 / 231198 / 231 0.8571 0.8072–0.90090.20650.0886 (231)0
0.000.250.500.751.00gemma-4:12b
By tier
TagTier, itemsAccuracyBrierECE (calibration_n)
gemma-4:12b Easy 4848 / 48 1.00000.00000.0000 (48)
gemma-4:12b Standard 7270 / 72 0.97220.03610.0164 (72)
gemma-4:12b Hard 11180 / 111 0.72070.40630.1737 (111)

gemma-4:26b-a4b

hf.co/ggml-org/gemma-4-26B-A4B-it-GGUF at bb4531cd, profile gemma-4:26b-a4b, chat layout

ardana pull gemma-4:26b-a4b
All 231 items
TagAttemptedAccuracy, 95% intervalBrierECE (calibration_n)Timeouts
gemma-4:26b-a4b default gemma-4-26B-A4B-it-Q4_0.gguf · 14.6 GB231 / 231199 / 231 0.8615 0.8142–0.90420.20290.0790 (231)0
0.000.250.500.751.00gemma-4:26b-a4b
By tier
TagTier, itemsAccuracyBrierECE (calibration_n)
gemma-4:26b-a4b Easy 4848 / 48 1.00000.00000.0001 (48)
gemma-4:26b-a4b Standard 7270 / 72 0.97220.03810.0233 (72)
gemma-4:26b-a4b Hard 11181 / 111 0.72970.39760.1531 (111)

gemma-4:31b

hf.co/ggml-org/gemma-4-31B-it-GGUF at 4fa4fdf3, profile gemma-4:31b, chat layout

ardana pull gemma-4:31b
All 231 items
TagAttemptedAccuracy, 95% intervalBrierECE (calibration_n)Timeouts
gemma-4:31b default stopped gemma-4-31B-it-Q4_0.gguf · 18.0 GB4 / 2310 / 4 0.0000 0.0000–0.00000.80000.2000 (1)0

A stopped run covers only the items it reached before three consecutive failures, the easier ones, so its row is not comparable with a complete one.

Requests that failed for a reason other than a timeout count as wrong: gemma-4:31b 3.

0.000.250.500.751.00gemma-4:31b
By tier
TagTier, itemsAccuracyBrierECE (calibration_n)
gemma-4:31b stoppedEasy 4of 480 / 4 0.00000.80000.2000 (1)
gemma-4:31b stoppedStandard 0of 720 / 0 ––– (0)
gemma-4:31b stoppedHard 0of 1110 / 0 ––– (0)

Accuracy is correct answers over attempted items, and a request that failed or timed out counts as wrong; the interval is a 95% bootstrap over items, resampling paraphrase pairs together. Brier and ECE (top-label, ten equal bins) run over the items with a valid probability distribution, whose count sits beside every ECE. Every figure is computed by JevBench’s own scorer.

Only JevBench’s public items are used (only the public half of the hard tier), so these numbers are not comparable with its published leaderboard, which includes held-out items.