spark-x2.5

Accuracy is correct answers over attempted items, and a request that failed or timed out counts as wrong; the interval is a 95% bootstrap over items, resampling paraphrase pairs together. Brier and ECE (top-label, ten equal bins) run over the items with a valid probability distribution, whose count sits beside every ECE. Every figure is computed by JevBench’s own scorer.

Only JevBench’s public items are used (only the public half of the hard tier), so these numbers are not comparable with its published leaderboard, which includes held-out items.