Benchmarks

smollm3

SmolLM3, a stock instruct model read zero-shot in the chat layout.

Model page
ardana pull smollm3
Benchmark
JevBench bb05a335, dataset dc3995d8, 231 public items, 600 s client timeout
Measured
2026-10-13, 1 run per quant.

Against the other models

Accuracy of every measured size at its default quant, best first, with smollm3’s sizes in ink. Then every measured quant’s accuracy against its calibration error and against its size on disk.

  1. jevk5:9b 0.8658
  2. qwen3.8:27b 0.8658
  3. gemma-4:26b-a4b 0.8615
  4. gemma-4:12b 0.8571
  5. qwen3.6:35b-a3b 0.8528
  6. jevk5:4b 0.8442
  7. qwen3.5:9b 0.8398
  8. decider:4b 0.8009
  9. ornith-1.5:9b 0.7879
  10. phi-4:14b 0.7835
  11. gemma-4:e4b 0.7706
  12. qwen3:14b 0.7706
  13. decider:2b 0.7662
  14. qwen3.5:4b 0.7662
  15. gemma-3:12b 0.7576
  16. jevk5:2b 0.7359
  17. qwen3:8b 0.7229
  18. qwen3.5:2b 0.6926
  19. llama-3.1:8b 0.6797
  20. gemma-4:e2b 0.6710
  21. decider:0.8b 0.6667
  22. ling-3.0:tiny 0.6667
  23. phi-4:mini 0.6537
  24. llama-3.2:3b 0.6494
  25. qwen3:4b 0.6364
  26. olmo-3:7b 0.6234
  27. smollm3:3b 0.6234
  28. lfm2.5:2.6b 0.6061
  29. gemma-3:4b 0.6017
  30. minicpm5:1b 0.5887
  31. minicpm5:2b 0.5844
  32. qwen3:1.7b 0.5714
  33. qwen3.5:0.8b 0.5671
  34. lfm2.5:8b-a1b 0.5584
  35. qwen3:0.6b 0.5498
  36. lfm2.5:1.2b 0.5281
  37. gemma-3:1b 0.5065
  38. llama-3.2:1b 0.5065
  39. gemma-4:31b 0.0000
Most attractive quadrant0.600.800.000.200.40ECE (left is better calibrated)accuracy (up is better)smollm3:3bqwen3.8:27bdecider:0.8bdecider:2bdecider:4bgemma-3:1bgemma-3:4bgemma-3:12bgemma-4:e2bgemma-4:e4bgemma-4:12bgemma-4:26b-a4bjevk5:2blfm2.5:1.2blfm2.5:2.6blfm2.5:8b-a1bling-3.0:tinyllama-3.1:8bllama-3.2:1bllama-3.2:3bminicpm5:1bminicpm5:2bolmo-3:7bphi-4:14bqwen3:0.6bqwen3:1.7bqwen3:4bqwen3:8bqwen3:14bqwen3.5:0.8bqwen3.5:2bqwen3.5:4b

1 run is left out: a stopped run covers fewer items, and a run without a valid distribution has no ECE.

Most attractive quadrant0.600.800.5 GB1 GB2 GB5 GB10 GB20 GBsize on disk, log scale (left is smaller)accuracy (up is better)smollm3:3bdecider:0.8bdecider:2bjevk5:4bjevk5:9bqwen3.5:0.8bdecider:4bgemma-3:1bgemma-3:4bgemma-3:12bgemma-4:e2bgemma-4:e4bgemma-4:12bgemma-4:26b-a4bjevk5:2blfm2.5:1.2blfm2.5:2.6blfm2.5:8b-a1bling-3.0:tinyllama-3.1:8bllama-3.2:1bllama-3.2:3bminicpm5:1bminicpm5:2bolmo-3:7bphi-4:miniphi-4:14bqwen3:0.6bqwen3:8bqwen3:14bqwen3.6:35b-a3bqwen3.8:27b

1 run is left out: a stopped run covers fewer items, and a run without a valid distribution has no accuracy.

smollm3:3b

hf.co/ggml-org/SmolLM3-3B-GGUF at 4965cb60, profile smollm3:3b, chat layout

ardana pull smollm3:3b
All 231 items
TagAttemptedAccuracy, 95% intervalBrierECE (calibration_n)Timeouts
smollm3:3b default SmolLM3-Q4_K_M.gguf · 1.9 GB231 / 231144 / 231 0.6234 0.5541–0.69040.55790.2363 (231)0
0.000.250.500.751.00smollm3:3b
By tier
TagTier, itemsAccuracyBrierECE (calibration_n)
smollm3:3b Easy 4847 / 48 0.97920.01140.0152 (48)
smollm3:3b Standard 7248 / 72 0.66670.49450.2110 (72)
smollm3:3b Hard 11149 / 111 0.44140.83530.3524 (111)

Accuracy is correct answers over attempted items, and a request that failed or timed out counts as wrong; the interval is a 95% bootstrap over items, resampling paraphrase pairs together. Brier and ECE (top-label, ten equal bins) run over the items with a valid probability distribution, whose count sits beside every ECE. Every figure is computed by JevBench’s own scorer.

Only JevBench’s public items are used (only the public half of the hard tier), so these numbers are not comparable with its published leaderboard, which includes held-out items.