Decision benchmarks leaderboard

Accuracy and calibration of Jev, other hosted decision models and open-weight ones on the decision benchmarks their makers use, broken down by multiple-choice, yes/no and rating questions.

Twenty-three models, fifteen hosted and eight with open weights, all asked the same questions (the exam leaderboard lists them). The benchmarks are the ones AutoTrust and NeoHorse report for their models, scored for calibration as well as accuracy: Kev’s decision-v7, transfer-v4 and transfer-v9 suites, VitaminC and MASSIVE, and the original and easy tiers of JevBench. Every reference answer in the summary comes from people or from rules. Two more benchmarks, JevBench’s hard tier and Nimble, have answers written by language models; they are shown under “Each benchmark” but left out of the summary.

Questions come in three kinds. Multiple choice picks one of several options, yes/no asks whether a statement holds, and rating places something on an ordered scale. Pick a kind to see how each model handles it.

Rank	Model	Accuracy ↑	ECE ↓	Brier score ↓	Over/underconfidence
11 ⊸ 4	Perplexity Decider v1.2Perplexity · API only	92.7% ±0.9%	0.021 ±0.006	0.047 ±0.006	+0.004
21 ⊸ 4	Drex MaxNace AI · API only	92.3% ±0.9%	0.029 ±0.008	0.052 ±0.007	+0.021
31 ⊸ 4	Perplexity Decider Flash v1.2Perplexity · API only	91.0% ±1.0%	0.034 ±0.008	0.059 ±0.007	+0.008
41 ⊸ 4	Mercury DecideInception · API only	91.4% ±1.1%	0.053 ±0.009	0.062 ±0.009	+0.050
55 ⊸ 10	d1-ultraLiquid AI · API only	87.2% ±1.4%	0.034 ±0.009	0.082 ±0.006	−0.010
65 ⊸ 12	ClefCloudflare · API only	87.9% ±1.2%	0.048 ±0.010	0.083 ±0.008	+0.025
75 ⊸ 13	Clef OmniCloudflare · API only	87.9% ±1.3%	0.054 ±0.010	0.086 ±0.007	+0.004
85 ⊸ 14	Clef FlashCloudflare · API only	86.6% ±1.4%	0.048 ±0.010	0.088 ±0.008	+0.007
95 ⊸ 14	Jev 1.13TypeSafe · API only	87.4% ±1.5%	0.061 ±0.011	0.088 ±0.010	+0.051
105 ⊸ 14	Jev 1.14TypeSafe · API only	86.9% ±1.3%	0.061 ±0.011	0.089 ±0.009	+0.050
116 ⊸ 16	JEV-27BAutoTrust AI · Apache-2.0	86.2% ±1.3%	0.066 ±0.011	0.097 ±0.009	+0.044
126 ⊸ 17	JEV-9BAutoTrust AI · Apache-2.0	83.8% ±1.6%	0.055 ±0.012	0.100 ±0.009	+0.039
137 ⊸ 17	JPT-4Bkirp · CC BY-NC 4.0	84.4% ±1.5%	0.065 ±0.012	0.102 ±0.010	+0.037
148 ⊸ 17	Drex v2.0Nace AI · API only	84.0% ±1.6%	0.073 ±0.012	0.103 ±0.008	+0.033
1512 ⊸ 18	Strands Decider 2BAmazon Strands Labs · Apache-2.0	82.4% ±1.6%	0.067 ±0.012	0.115 ±0.009	+0.004
1611 ⊸ 18	GPT-6 Luna DecisionsOpenAI · API only	84.3% ±1.5%	0.096 ±0.013	0.116 ±0.010	+0.082
1711 ⊸ 18	Drex v1.5Nace AI · API only	83.1% ±1.5%	0.092 ±0.013	0.117 ±0.011	+0.080
1815 ⊸ 18	Kev-4BJared Palmer · Apache-2.0	83.8% ±1.6%	0.107 ±0.013	0.124 ±0.009	−0.025
1919 ⊸ 23	Solar DecideUpstage · API only	81.3% ±1.7%	0.153 ±0.016	0.165 ±0.015	+0.149
2019 ⊸ 23	Open-Jev-2BZefan Cai · Apache-2.0	73.9% ±2.1%	0.109 ±0.016	0.165 ±0.009	−0.054
2119 ⊸ 23	Solar Decide FlashUpstage · API only	75.7% ±2.1%	0.149 ±0.017	0.176 ±0.015	+0.134
2219 ⊸ 23	Qwen3.5-9B, untrainedAlibaba Qwen · Apache-2.0	72.0% ±2.2%	0.112 ±0.017	0.178 ±0.009	−0.096
2319 ⊸ 23	LayaConvai Innovations · Apache-2.0	71.6% ±2.3%	0.108 ±0.016	0.183 ±0.013	+0.051

All questions: means over the 4 benchmarks with human or rule-based labels that contain this kind of question. JPT-4B would not answer 89 records across these benchmarks, which its scores leave out. GPT-6 Luna Decisions would not answer 11 records across these benchmarks, which its scores leave out. Solar Decide would not answer 116 records across these benchmarks, which its scores leave out. Solar Decide Flash would not answer 116 records across these benchmarks, which its scores leave out. Ranked by Brier score. Each ± and error bar is a 95% interval from resampling questions, and the small numbers under each rank are the range of ranks those intervals allow. Over/underconfidence is average confidence minus accuracy.

Rank	Model	Accuracy ↑	ECE ↓	Brier score ↓	Over/underconfidence
11 ⊸ 3	Perplexity Decider v1.2Perplexity · API only	93.0% ±0.9%	0.018 ±0.006	0.044 ±0.006	+0.004
21 ⊸ 4	Drex MaxNace AI · API only	92.7% ±0.9%	0.030 ±0.007	0.050 ±0.006	+0.022
31 ⊸ 4	Mercury DecideInception · API only	92.4% ±1.0%	0.048 ±0.008	0.054 ±0.007	+0.046
42 ⊸ 4	Perplexity Decider Flash v1.2Perplexity · API only	91.0% ±1.0%	0.034 ±0.008	0.058 ±0.006	+0.012
55 ⊸ 11	d1-ultraLiquid AI · API only	87.6% ±1.2%	0.039 ±0.009	0.078 ±0.006	−0.015
65 ⊸ 11	ClefCloudflare · API only	88.1% ±1.4%	0.044 ±0.010	0.080 ±0.008	+0.023
75 ⊸ 12	Jev 1.13TypeSafe · API only	88.7% ±1.2%	0.057 ±0.009	0.082 ±0.008	+0.052
85 ⊸ 14	Clef OmniCloudflare · API only	88.3% ±1.5%	0.053 ±0.011	0.084 ±0.008	−0.010
95 ⊸ 13	Clef FlashCloudflare · API only	86.6% ±1.5%	0.046 ±0.010	0.084 ±0.008	+0.003
105 ⊸ 14	Jev 1.14TypeSafe · API only	87.8% ±1.2%	0.059 ±0.010	0.084 ±0.008	+0.054
115 ⊸ 14	JEV-27BAutoTrust AI · Apache-2.0	86.5% ±1.3%	0.066 ±0.011	0.090 ±0.009	+0.058
127 ⊸ 17	JEV-9BAutoTrust AI · Apache-2.0	83.7% ±1.7%	0.055 ±0.012	0.097 ±0.009	+0.046
138 ⊸ 17	JPT-4Bkirp · CC BY-NC 4.0	84.1% ±1.7%	0.068 ±0.011	0.100 ±0.010	+0.034
149 ⊸ 17	Drex v2.0Nace AI · API only	84.3% ±1.6%	0.079 ±0.012	0.101 ±0.008	+0.031
1512 ⊸ 18	Strands Decider 2BAmazon Strands Labs · Apache-2.0	81.9% ±1.7%	0.063 ±0.012	0.108 ±0.008	+0.025
1612 ⊸ 18	Drex v1.5Nace AI · API only	83.2% ±1.8%	0.095 ±0.013	0.113 ±0.010	+0.080
1712 ⊸ 18	GPT-6 Luna DecisionsOpenAI · API only	84.5% ±1.3%	0.092 ±0.013	0.114 ±0.011	+0.080
1815 ⊸ 19	Kev-4BJared Palmer · Apache-2.0	84.1% ±1.5%	0.125 ±0.016	0.125 ±0.010	−0.040
1918 ⊸ 23	Solar DecideUpstage · API only	82.6% ±1.8%	0.147 ±0.017	0.151 ±0.017	+0.138
2019 ⊸ 23	Solar Decide FlashUpstage · API only	75.3% ±2.3%	0.146 ±0.019	0.169 ±0.016	+0.128
2119 ⊸ 23	Open-Jev-2BZefan Cai · Apache-2.0	72.2% ±2.8%	0.127 ±0.018	0.170 ±0.011	−0.070
2219 ⊸ 23	Qwen3.5-9B, untrainedAlibaba Qwen · Apache-2.0	72.1% ±2.5%	0.127 ±0.017	0.175 ±0.011	−0.113
2319 ⊸ 23	LayaConvai Innovations · Apache-2.0	70.3% ±2.6%	0.126 ±0.019	0.177 ±0.015	+0.062

Multiple choice: means over the 4 benchmarks with human or rule-based labels that contain this kind of question. JPT-4B would not answer 89 records across these benchmarks, which its scores leave out. GPT-6 Luna Decisions would not answer 11 records across these benchmarks, which its scores leave out. Solar Decide would not answer 116 records across these benchmarks, which its scores leave out. Solar Decide Flash would not answer 116 records across these benchmarks, which its scores leave out. Ranked by Brier score. Each ± and error bar is a 95% interval from resampling questions, and the small numbers under each rank are the range of ranks those intervals allow. Over/underconfidence is average confidence minus accuracy.

Rank	Model	Accuracy ↑	ECE ↓	Brier score ↓	Over/underconfidence
11 ⊸ 12	Perplexity Decider v1.2Perplexity · API only	95.2% ±0.8%	0.025 ±0.015	0.037 ±0.008	+0.001
21 ⊸ 8	Drex MaxNace AI · API only	94.7% ±0.8%	0.029 ±0.009	0.037 ±0.005	+0.014
31 ⊸ 10	d1-ultraLiquid AI · API only	94.7% ±0.8%	0.020 ±0.008	0.039 ±0.005	−0.002
41 ⊸ 15	Perplexity Decider Flash v1.2Perplexity · API only	94.0% ±0.9%	0.040 ±0.016	0.046 ±0.008	−0.010
51 ⊸ 15	ClefCloudflare · API only	94.5% ±0.9%	0.046 ±0.016	0.046 ±0.008	−0.002
61 ⊸ 16	Clef OmniCloudflare · API only	94.0% ±1.0%	0.058 ±0.012	0.050 ±0.008	+0.011
71 ⊸ 17	Drex v2.0Nace AI · API only	93.1% ±2.5%	0.037 ±0.019	0.050 ±0.011	−0.023
82 ⊸ 18	Jev 1.14TypeSafe · API only	91.8% ±2.7%	0.042 ±0.017	0.054 ±0.014	−0.012
91 ⊸ 19	Mercury DecideInception · API only	93.2% ±2.7%	0.046 ±0.025	0.057 ±0.024	+0.038
104 ⊸ 18	GPT-6 Luna DecisionsOpenAI · API only	90.9% ±3.4%	0.066 ±0.022	0.059 ±0.013	+0.037
114 ⊸ 19	Jev 1.13TypeSafe · API only	90.1% ±3.9%	0.052 ±0.022	0.060 ±0.015	−0.000
123 ⊸ 19	Clef FlashCloudflare · API only	93.1% ±2.6%	0.073 ±0.026	0.061 ±0.020	−0.018
132 ⊸ 21	Drex v1.5Nace AI · API only	91.0% ±3.7%	0.062 ±0.032	0.061 ±0.023	+0.035
143 ⊸ 21	JPT-4Bkirp · CC BY-NC 4.0	91.1% ±3.9%	0.046 ±0.026	0.064 ±0.023	+0.016
154 ⊸ 21	Kev-4BJared Palmer · Apache-2.0	88.8% ±5.5%	0.060 ±0.036	0.066 ±0.023	+0.005
166 ⊸ 19	JEV-9BAutoTrust AI · Apache-2.0	89.9% ±3.7%	0.058 ±0.027	0.067 ±0.013	−0.024
177 ⊸ 19	JEV-27BAutoTrust AI · Apache-2.0	93.1% ±0.9%	0.075 ±0.023	0.069 ±0.009	−0.075
188 ⊸ 22	Strands Decider 2BAmazon Strands Labs · Apache-2.0	89.0% ±4.5%	0.075 ±0.030	0.088 ±0.024	−0.057
1910 ⊸ 22	Solar DecideUpstage · API only	86.9% ±4.8%	0.087 ±0.041	0.112 ±0.041	+0.086
2015 ⊸ 22	Open-Jev-2BZefan Cai · Apache-2.0	84.4% ±5.6%	0.064 ±0.043	0.118 ±0.036	+0.018
2115 ⊸ 23	Solar Decide FlashUpstage · API only	84.3% ±5.6%	0.097 ±0.049	0.130 ±0.044	+0.063
2218 ⊸ 23	Qwen3.5-9B, untrainedAlibaba Qwen · Apache-2.0	79.0% ±6.8%	0.062 ±0.041	0.143 ±0.034	−0.007
2321 ⊸ 23	LayaConvai Innovations · Apache-2.0	74.3% ±6.9%	0.122 ±0.054	0.194 ±0.034	+0.033

Yes/no: means over the 2 benchmarks with human or rule-based labels that contain this kind of question. GPT-6 Luna Decisions would not answer 11 records across these benchmarks, which its scores leave out. JPT-4B would not answer 89 records across these benchmarks, which its scores leave out. Solar Decide would not answer 116 records across these benchmarks, which its scores leave out. Solar Decide Flash would not answer 116 records across these benchmarks, which its scores leave out. Ranked by Brier score. Each ± and error bar is a 95% interval from resampling questions, and the small numbers under each rank are the range of ranks those intervals allow. Over/underconfidence is average confidence minus accuracy.

Rank	Model	Accuracy ↑	ECE ↓	Brier score ↓	Over/underconfidence
11 ⊸ 12	Drex MaxNace AI · API only	84.4% ±2.2%	0.024 ±0.013	0.082 ±0.008	+0.016
21 ⊸ 13	Drex v2.0Nace AI · API only	83.0% ±2.2%	0.042 ±0.014	0.084 ±0.007	−0.028
31 ⊸ 15	Perplexity Decider v1.2Perplexity · API only	84.0% ±2.1%	0.040 ±0.014	0.084 ±0.008	+0.020
41 ⊸ 16	d1-ultraLiquid AI · API only	85.3% ±2.1%	0.035 ±0.016	0.091 ±0.009	+0.016
51 ⊸ 16	Mercury DecideInception · API only	83.6% ±2.1%	0.058 ±0.017	0.092 ±0.011	+0.052
61 ⊸ 16	JEV-27BAutoTrust AI · Apache-2.0	83.0% ±2.2%	0.059 ±0.018	0.092 ±0.010	+0.054
71 ⊸ 16	ClefCloudflare · API only	83.8% ±2.2%	0.058 ±0.018	0.094 ±0.010	+0.018
81 ⊸ 16	Jev 1.14TypeSafe · API only	84.3% ±2.2%	0.055 ±0.018	0.094 ±0.012	+0.055
91 ⊸ 16	Perplexity Decider Flash v1.2Perplexity · API only	82.0% ±2.2%	0.036 ±0.015	0.095 ±0.009	+0.019
101 ⊸ 16	Jev 1.13TypeSafe · API only	84.5% ±2.1%	0.055 ±0.019	0.097 ±0.012	+0.051
111 ⊸ 16	Drex v1.5Nace AI · API only	82.5% ±2.2%	0.044 ±0.021	0.097 ±0.010	+0.024
123 ⊸ 16	Clef FlashCloudflare · API only	82.8% ±2.2%	0.092 ±0.021	0.101 ±0.010	+0.029
131 ⊸ 16	Clef OmniCloudflare · API only	84.1% ±2.1%	0.084 ±0.019	0.101 ±0.012	+0.033
142 ⊸ 16	Kev-4BJared Palmer · Apache-2.0	82.6% ±2.2%	0.093 ±0.028	0.103 ±0.013	−0.048
153 ⊸ 16	GPT-6 Luna DecisionsOpenAI · API only	84.0% ±2.0%	0.077 ±0.025	0.103 ±0.012	+0.032
164 ⊸ 16	JEV-9BAutoTrust AI · Apache-2.0	79.9% ±2.2%	0.050 ±0.019	0.107 ±0.010	+0.043
1717 ⊸ 18	JPT-4Bkirp · CC BY-NC 4.0	79.1% ±2.3%	0.112 ±0.025	0.130 ±0.012	+0.012
1817 ⊸ 18	Open-Jev-2BZefan Cai · Apache-2.0	74.2% ±2.1%	0.113 ±0.045	0.140 ±0.023	−0.014
1919 ⊸ 23	Qwen3.5-9B, untrainedAlibaba Qwen · Apache-2.0	65.6% ±10.5%	0.221 ±0.064	0.204 ±0.023	−0.061
2019 ⊸ 23	Solar Decide FlashUpstage · API only	72.0% ±2.2%	0.209 ±0.026	0.206 ±0.018	+0.169
2119 ⊸ 23	Solar DecideUpstage · API only	75.7% ±2.3%	0.230 ±0.025	0.212 ±0.020	+0.189
2219 ⊸ 23	Strands Decider 2BAmazon Strands Labs · Apache-2.0	77.7% ±2.1%	0.268 ±0.030	0.212 ±0.022	−0.244
2319 ⊸ 23	LayaConvai Innovations · Apache-2.0	64.8% ±7.4%	0.251 ±0.061	0.223 ±0.038	+0.035

Rating: means over the 2 benchmarks with human or rule-based labels that contain this kind of question. GPT-6 Luna Decisions would not answer 11 records across these benchmarks, which its scores leave out. JPT-4B would not answer 89 records across these benchmarks, which its scores leave out. Solar Decide Flash would not answer 116 records across these benchmarks, which its scores leave out. Solar Decide would not answer 116 records across these benchmarks, which its scores leave out. Ranked by Brier score. Each ± and error bar is a 95% interval from resampling questions, and the small numbers under each rank are the range of ranks those intervals allow. Over/underconfidence is average confidence minus accuracy.

By question type

The same models on multiple choice, yes/no and rating questions. A long line means a model behaves very differently depending on how the question is asked.

Each benchmark

Benchmark Question type

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
Perplexity Decider v1.2	3,012	85.3%	0.027	0.013	0.086	+0.025	0.386
d1-ultra	3,012	85.8%	0.011	0.014	0.090	+0.003	0.381
Drex Max	3,012	85.1%	0.041	0.012	0.091	+0.038	0.379
Drex v2.0	3,012	84.0%	0.034	0.017	0.093	−0.020	0.412
Mercury Decide	3,012	85.8%	0.067	0.011	0.094	+0.063	0.418
Clef	3,012	84.4%	0.035	0.013	0.100	+0.033	0.427
Jev 1.14	3,012	83.7%	0.046	0.012	0.104	+0.044	0.707
Kev-4B	3,012	80.9%	0.011	0.017	0.104	−0.003	0.479
Jev 1.13	3,012	84.1%	0.049	0.013	0.105	+0.044	0.830
Drex v1.5	3,012	81.1%	0.063	0.013	0.106	+0.063	0.488
Clef Flash	3,012	82.9%	0.030	0.015	0.107	+0.027	0.470
Perplexity Decider Flash v1.2	3,012	81.5%	0.044	0.014	0.108	+0.040	0.478
Clef Omni	3,012	84.2%	0.044	0.014	0.112	+0.032	0.483
JEV-27B	3,012	81.6%	0.031	0.016	0.117	+0.002	0.526
GPT-6 Luna Decisions	3,001	81.2%	0.079	0.013	0.121	+0.073	0.944
JEV-9B	3,012	77.7%	0.029	0.017	0.126	+0.024	0.570
JPT-4B	2,910	78.2%	0.049	0.017	0.129	+0.045	0.566
Strands Decider 2B	3,012	74.6%	0.034	0.019	0.144	−0.011	0.639
Qwen3.5-9B, untrained	3,012	71.1%	0.044	0.021	0.166	−0.031	0.736
Open-Jev-2B	3,012	69.0%	0.039	0.020	0.172	+0.004	0.795
Solar Decide	2,932	78.3%	0.162	0.009	0.177	+0.162	0.916
Solar Decide Flash	2,932	69.5%	0.155	0.015	0.201	+0.152	0.874
Laya	3,012	60.2%	0.133	0.020	0.219	+0.133	1.466

GPT-6 Luna Decisions would not answer 11 records here, so its scores cover the rest.

Solar Decide would not answer 116 records here, so its scores cover the rest.

Solar Decide Flash would not answer 116 records here, so its scores cover the rest.

JPT-4B would not answer 89 records here, so its scores cover the rest.

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
Mercury Decide	1,412	88.7%	0.056	0.013	0.073	+0.056	0.397
d1-ultra	1,412	87.4%	0.030	0.021	0.074	−0.016	0.385
Perplexity Decider v1.2	1,412	86.5%	0.023	0.019	0.076	+0.018	0.411
Drex v2.0	1,412	85.6%	0.050	0.023	0.081	−0.032	0.412
Drex Max	1,412	86.6%	0.046	0.017	0.082	+0.037	0.394
Clef	1,412	85.8%	0.021	0.021	0.086	+0.012	0.452
Jev 1.13	1,412	87.4%	0.055	0.017	0.089	+0.041	1.068
Jev 1.14	1,412	86.2%	0.057	0.016	0.090	+0.050	0.875
Clef Flash	1,412	83.4%	0.029	0.024	0.094	−0.001	0.500
JEV-27B	1,412	82.8%	0.046	0.020	0.094	+0.042	0.559
Drex v1.5	1,412	80.8%	0.071	0.020	0.096	+0.067	0.554
Clef Omni	1,412	86.0%	0.032	0.023	0.100	−0.024	0.500
Kev-4B	1,412	79.3%	0.034	0.025	0.101	−0.007	0.582
Perplexity Decider Flash v1.2	1,412	81.5%	0.052	0.021	0.105	+0.046	0.558
JEV-9B	1,412	76.9%	0.041	0.023	0.117	+0.041	0.645
GPT-6 Luna Decisions	1,410	80.1%	0.077	0.019	0.117	+0.069	1.033
JPT-4B	1,386	76.6%	0.057	0.026	0.126	+0.033	0.645
Solar Decide	1,332	81.5%	0.133	0.013	0.143	+0.133	0.858
Strands Decider 2B	1,412	71.0%	0.047	0.028	0.145	+0.025	0.762
Qwen3.5-9B, untrained	1,412	68.6%	0.080	0.032	0.173	−0.077	0.894
Open-Jev-2B	1,412	64.1%	0.080	0.032	0.190	−0.048	1.020
Solar Decide Flash	1,332	66.5%	0.143	0.024	0.194	+0.137	0.929
Laya	1,412	53.8%	0.149	0.028	0.221	+0.147	1.888

GPT-6 Luna Decisions would not answer 11 records here, so its scores cover the rest.

Solar Decide would not answer 116 records here, so its scores cover the rest.

Solar Decide Flash would not answer 116 records here, so its scores cover the rest.

JPT-4B would not answer 89 records here, so its scores cover the rest.

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
Perplexity Decider v1.2	1,160	90.4%	0.027	0.015	0.067	+0.025	0.227
Drex Max	1,160	89.5%	0.044	0.014	0.074	+0.041	0.242
d1-ultra	1,160	89.5%	0.024	0.017	0.076	+0.012	0.245
Kev-4B	1,160	88.7%	0.018	0.019	0.078	−0.011	0.250
Drex v2.0	1,160	88.9%	0.027	0.019	0.079	−0.001	0.270
Perplexity Decider Flash v1.2	1,160	88.0%	0.030	0.017	0.080	+0.030	0.257
Clef	1,160	89.1%	0.045	0.015	0.083	+0.043	0.271
Drex v1.5	1,160	87.5%	0.056	0.015	0.087	+0.056	0.279
Mercury Decide	1,160	89.2%	0.063	0.013	0.087	+0.056	0.316
Clef Flash	1,160	88.9%	0.037	0.016	0.088	+0.027	0.296
Jev 1.14	1,160	86.4%	0.027	0.019	0.089	+0.012	0.286
JPT-4B	1,107	87.7%	0.042	0.017	0.089	+0.037	0.285
Jev 1.13	1,160	85.8%	0.028	0.019	0.093	+0.025	0.295
Clef Omni	1,160	88.0%	0.069	0.013	0.094	+0.069	0.328
GPT-6 Luna Decisions	1,160	87.4%	0.076	0.013	0.096	+0.067	0.806
Strands Decider 2B	1,160	86.4%	0.033	0.023	0.098	−0.022	0.329
JEV-9B	1,160	85.3%	0.025	0.022	0.103	−0.021	0.325
JEV-27B	1,160	86.1%	0.088	0.024	0.119	−0.088	0.365
Open-Jev-2B	1,160	82.8%	0.036	0.021	0.121	+0.035	0.385
Solar Decide	1,160	84.9%	0.101	0.012	0.125	+0.098	0.499
Qwen3.5-9B, untrained	1,160	83.0%	0.035	0.024	0.125	−0.024	0.403
Solar Decide Flash	1,160	82.6%	0.084	0.018	0.130	+0.083	0.427
Laya	1,160	76.4%	0.067	0.023	0.174	+0.059	0.792

GPT-6 Luna Decisions would not answer 11 records here, so its scores cover the rest.

Solar Decide would not answer 116 records here, so its scores cover the rest.

Solar Decide Flash would not answer 116 records here, so its scores cover the rest.

JPT-4B would not answer 89 records here, so its scores cover the rest.

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
Drex Max	440	68.9%	0.047	0.044	0.163	+0.033	0.689
Drex v2.0	440	65.9%	0.062	0.053	0.167	−0.034	0.786
Perplexity Decider v1.2	440	68.0%	0.073	0.045	0.168	+0.046	0.723
Kev-4B	440	65.2%	0.058	0.047	0.181	+0.033	0.749
d1-ultra	440	70.7%	0.064	0.043	0.182	+0.040	0.724
Mercury Decide	440	67.3%	0.115	0.042	0.183	+0.105	0.754
JEV-27B	440	65.9%	0.114	0.044	0.185	+0.114	0.841
Clef	440	67.5%	0.077	0.045	0.186	+0.075	0.756
Jev 1.14	440	68.6%	0.110	0.044	0.188	+0.110	1.278
Perplexity Decider Flash v1.2	440	64.1%	0.062	0.048	0.189	+0.047	0.804
Drex v1.5	440	65.0%	0.069	0.047	0.192	+0.067	0.827
Jev 1.13	440	69.1%	0.108	0.042	0.193	+0.104	1.476
Clef Flash	440	65.7%	0.122	0.047	0.197	+0.119	0.835
Clef Omni	440	68.2%	0.119	0.043	0.199	+0.114	0.837
GPT-6 Luna Decisions	431	68.0%	0.119	0.044	0.203	+0.100	1.029
JEV-9B	440	59.8%	0.095	0.050	0.215	+0.091	0.976
Open-Jev-2B	440	48.4%	0.110	0.057	0.245	+0.088	1.151
JPT-4B	417	58.3%	0.142	0.049	0.249	+0.107	1.052
Qwen3.5-9B, untrained	440	48.0%	0.149	0.053	0.253	+0.095	1.107
Strands Decider 2B	440	55.5%	0.142	0.052	0.263	−0.095	1.063
Laya	440	38.0%	0.288	0.053	0.332	+0.283	1.891
Solar Decide Flash	440	44.1%	0.381	0.041	0.410	+0.375	1.884
Solar Decide	440	51.4%	0.422	0.024	0.421	+0.417	2.195

GPT-6 Luna Decisions would not answer 11 records here, so its scores cover the rest.

Solar Decide would not answer 116 records here, so its scores cover the rest.

Solar Decide Flash would not answer 116 records here, so its scores cover the rest.

JPT-4B would not answer 89 records here, so its scores cover the rest.

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
Perplexity Decider v1.2	599	92.8%	0.013	0.023	0.053	−0.008	0.220
Drex Max	599	92.7%	0.027	0.021	0.060	+0.007	0.241
Perplexity Decider Flash v1.2	599	90.7%	0.036	0.026	0.068	−0.007	0.298
Mercury Decide	599	89.6%	0.077	0.015	0.084	+0.072	0.377
Jev 1.13	599	79.8%	0.101	0.026	0.149	+0.099	1.229
Clef Omni	599	78.5%	0.056	0.034	0.151	+0.054	0.626
Clef	599	77.3%	0.075	0.033	0.153	+0.067	0.624
d1-ultra	599	75.6%	0.060	0.038	0.155	−0.000	0.565
JEV-9B	599	76.1%	0.073	0.033	0.156	+0.072	0.664
Strands Decider 2B	599	76.1%	0.098	0.031	0.156	+0.094	0.598
Clef Flash	599	74.8%	0.064	0.036	0.157	+0.042	0.627
Jev 1.14	599	78.0%	0.103	0.030	0.163	+0.098	1.201
JPT-4B	599	76.8%	0.139	0.026	0.164	+0.139	0.682
JEV-27B	599	77.6%	0.130	0.026	0.170	+0.121	0.841
Laya	599	79.0%	0.129	0.041	0.183	−0.051	0.667
Open-Jev-2B	599	64.1%	0.084	0.042	0.190	+0.077	0.830
Qwen3.5-9B, untrained	599	65.4%	0.082	0.045	0.196	−0.037	0.795
Kev-4B	599	72.1%	0.156	0.028	0.210	+0.156	0.941
Drex v2.0	599	67.1%	0.157	0.034	0.222	+0.150	0.790
GPT-6 Luna Decisions	599	72.0%	0.181	0.027	0.224	+0.172	2.668
Drex v1.5	599	67.6%	0.206	0.030	0.247	+0.198	1.020
Solar Decide Flash	599	68.6%	0.219	0.028	0.248	+0.204	1.045
Solar Decide	599	71.8%	0.246	0.013	0.256	+0.245	1.365

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
Perplexity Decider v1.2	599	92.8%	0.013	0.023	0.053	−0.008	0.220
Drex Max	599	92.7%	0.027	0.021	0.060	+0.007	0.241
Perplexity Decider Flash v1.2	599	90.7%	0.036	0.026	0.068	−0.007	0.298
Mercury Decide	599	89.6%	0.077	0.015	0.084	+0.072	0.377
Jev 1.13	599	79.8%	0.101	0.026	0.149	+0.099	1.229
Clef Omni	599	78.5%	0.056	0.034	0.151	+0.054	0.626
Clef	599	77.3%	0.075	0.033	0.153	+0.067	0.624
d1-ultra	599	75.6%	0.060	0.038	0.155	−0.000	0.565
JEV-9B	599	76.1%	0.073	0.033	0.156	+0.072	0.664
Strands Decider 2B	599	76.1%	0.098	0.031	0.156	+0.094	0.598
Clef Flash	599	74.8%	0.064	0.036	0.157	+0.042	0.627
Jev 1.14	599	78.0%	0.103	0.030	0.163	+0.098	1.201
JPT-4B	599	76.8%	0.139	0.026	0.164	+0.139	0.682
JEV-27B	599	77.6%	0.130	0.026	0.170	+0.121	0.841
Laya	599	79.0%	0.129	0.041	0.183	−0.051	0.667
Open-Jev-2B	599	64.1%	0.084	0.042	0.190	+0.077	0.830
Qwen3.5-9B, untrained	599	65.4%	0.082	0.045	0.196	−0.037	0.795
Kev-4B	599	72.1%	0.156	0.028	0.210	+0.156	0.941
Drex v2.0	599	67.1%	0.157	0.034	0.222	+0.150	0.790
GPT-6 Luna Decisions	599	72.0%	0.181	0.027	0.224	+0.172	2.668
Drex v1.5	599	67.6%	0.206	0.030	0.247	+0.198	1.020
Solar Decide Flash	599	68.6%	0.219	0.028	0.248	+0.204	1.045
Solar Decide	599	71.8%	0.246	0.013	0.256	+0.245	1.365

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
Perplexity Decider v1.2	350	92.9%	0.033	0.028	0.048	+0.010	0.225
Perplexity Decider Flash v1.2	350	91.7%	0.031	0.027	0.056	+0.027	0.283
Drex Max	350	91.4%	0.044	0.022	0.058	+0.044	0.283
Mercury Decide	350	91.1%	0.058	0.019	0.060	+0.058	0.362
Clef Flash	350	91.1%	0.030	0.033	0.069	−0.001	0.393
Clef Omni	350	90.0%	0.055	0.038	0.070	−0.017	0.404
Clef	350	90.9%	0.045	0.027	0.075	+0.031	0.412
d1-ultra	350	87.4%	0.050	0.046	0.083	−0.026	0.418
Drex v2.0	350	87.4%	0.050	0.031	0.084	+0.035	0.440
Jev 1.14	350	86.9%	0.076	0.024	0.084	+0.072	0.953
JPT-4B	350	86.0%	0.040	0.045	0.085	−0.009	0.464
Jev 1.13	350	87.4%	0.071	0.026	0.091	+0.070	1.289
Drex v1.5	350	87.1%	0.066	0.027	0.096	+0.058	0.518
JEV-27B	350	85.7%	0.081	0.027	0.097	+0.077	0.589
JEV-9B	350	84.6%	0.073	0.030	0.103	+0.069	0.585
Strands Decider 2B	350	83.1%	0.041	0.043	0.109	+0.027	0.553
GPT-6 Luna Decisions	350	85.7%	0.098	0.026	0.113	+0.091	1.815
Kev-4B	350	86.3%	0.161	0.060	0.123	−0.157	0.587
Solar Decide	350	80.0%	0.170	0.012	0.179	+0.170	1.318
Laya	350	68.0%	0.131	0.047	0.183	+0.120	1.714
Solar Decide Flash	350	73.1%	0.169	0.035	0.184	+0.166	1.134
Open-Jev-2B	350	75.7%	0.248	0.064	0.201	−0.248	0.975
Qwen3.5-9B, untrained	350	69.7%	0.239	0.058	0.213	−0.238	1.285

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
Perplexity Decider v1.2	350	92.9%	0.033	0.028	0.048	+0.010	0.225
Perplexity Decider Flash v1.2	350	91.7%	0.031	0.027	0.056	+0.027	0.283
Drex Max	350	91.4%	0.044	0.022	0.058	+0.044	0.283
Mercury Decide	350	91.1%	0.058	0.019	0.060	+0.058	0.362
Clef Flash	350	91.1%	0.030	0.033	0.069	−0.001	0.393
Clef Omni	350	90.0%	0.055	0.038	0.070	−0.017	0.404
Clef	350	90.9%	0.045	0.027	0.075	+0.031	0.412
d1-ultra	350	87.4%	0.050	0.046	0.083	−0.026	0.418
Drex v2.0	350	87.4%	0.050	0.031	0.084	+0.035	0.440
Jev 1.14	350	86.9%	0.076	0.024	0.084	+0.072	0.953
JPT-4B	350	86.0%	0.040	0.045	0.085	−0.009	0.464
Jev 1.13	350	87.4%	0.071	0.026	0.091	+0.070	1.289
Drex v1.5	350	87.1%	0.066	0.027	0.096	+0.058	0.518
JEV-27B	350	85.7%	0.081	0.027	0.097	+0.077	0.589
JEV-9B	350	84.6%	0.073	0.030	0.103	+0.069	0.585
Strands Decider 2B	350	83.1%	0.041	0.043	0.109	+0.027	0.553
GPT-6 Luna Decisions	350	85.7%	0.098	0.026	0.113	+0.091	1.815
Kev-4B	350	86.3%	0.161	0.060	0.123	−0.157	0.587
Solar Decide	350	80.0%	0.170	0.012	0.179	+0.170	1.318
Laya	350	68.0%	0.131	0.047	0.183	+0.120	1.714
Solar Decide Flash	350	73.1%	0.169	0.035	0.184	+0.166	1.134
Open-Jev-2B	350	75.7%	0.248	0.064	0.201	−0.248	0.975
Qwen3.5-9B, untrained	350	69.7%	0.239	0.058	0.213	−0.238	1.285

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
Drex Max	120	100.0%	0.006	0.009	0.000	−0.006	0.006
d1-ultra	120	100.0%	0.016	0.016	0.001	−0.016	0.017
Perplexity Decider v1.2	120	100.0%	0.009	0.013	0.002	−0.009	0.011
Perplexity Decider Flash v1.2	120	100.0%	0.025	0.025	0.005	−0.025	0.028
Jev 1.14	120	99.2%	0.018	0.021	0.006	−0.012	0.028
JEV-27B	120	100.0%	0.024	0.026	0.006	−0.024	0.028
Clef	120	99.2%	0.039	0.031	0.006	−0.030	0.044
GPT-6 Luna Decisions	120	98.3%	0.027	0.027	0.008	−0.008	0.033
Mercury Decide	120	99.2%	0.008	0.005	0.008	+0.005	0.031
Jev 1.13	120	98.3%	0.024	0.026	0.008	−0.009	0.036
Clef Omni	120	99.2%	0.061	0.040	0.010	−0.051	0.069
Drex v2.0	120	97.5%	0.051	0.044	0.016	−0.033	0.074
JEV-9B	120	96.7%	0.047	0.033	0.017	−0.008	0.060
Drex v1.5	120	96.7%	0.032	0.032	0.019	+0.001	0.066
Clef Flash	120	97.5%	0.068	0.045	0.020	−0.042	0.096
JPT-4B	120	96.7%	0.030	0.042	0.029	−0.026	0.112
Solar Decide	120	95.0%	0.035	0.020	0.047	+0.021	0.194
Strands Decider 2B	120	95.8%	0.094	0.066	0.053	−0.094	0.195
Kev-4B	120	95.8%	0.100	0.070	0.058	−0.097	0.197
Solar Decide Flash	120	91.7%	0.055	0.039	0.070	+0.013	0.284
Open-Jev-2B	120	86.7%	0.066	0.080	0.097	−0.048	0.405
Qwen3.5-9B, untrained	120	81.7%	0.084	0.092	0.135	−0.077	0.504
Laya	120	79.2%	0.041	0.088	0.147	+0.001	0.545

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
Mercury Decide	72	100.0%	0.001	0.001	0.000	−0.001	0.001
Perplexity Decider v1.2	72	100.0%	0.003	0.005	0.000	−0.003	0.003
Jev 1.14	72	100.0%	0.002	0.003	0.000	−0.002	0.002
Jev 1.13	72	100.0%	0.002	0.003	0.000	−0.002	0.002
Drex Max	72	100.0%	0.002	0.005	0.000	−0.002	0.002
JEV-27B	72	100.0%	0.008	0.011	0.001	−0.008	0.008
GPT-6 Luna Decisions	72	100.0%	0.011	0.016	0.001	−0.011	0.012
d1-ultra	72	100.0%	0.018	0.018	0.002	−0.018	0.019
Perplexity Decider Flash v1.2	72	100.0%	0.016	0.022	0.002	−0.016	0.017
Clef	72	98.6%	0.034	0.032	0.006	−0.020	0.040
JEV-9B	72	97.2%	0.032	0.036	0.012	+0.001	0.043
Clef Omni	72	98.6%	0.070	0.049	0.013	−0.053	0.081
Drex v1.5	72	97.2%	0.036	0.038	0.014	−0.001	0.049
Clef Flash	72	97.2%	0.061	0.048	0.015	−0.028	0.077
Drex v2.0	72	97.2%	0.057	0.055	0.016	−0.028	0.075
Strands Decider 2B	72	97.2%	0.065	0.057	0.022	−0.046	0.102
JPT-4B	72	97.2%	0.037	0.051	0.026	−0.028	0.106
Solar Decide	72	97.2%	0.038	0.026	0.029	+0.004	0.143
Solar Decide Flash	72	93.1%	0.052	0.051	0.051	+0.006	0.233
Kev-4B	72	98.6%	0.151	0.103	0.065	−0.151	0.220
Open-Jev-2B	72	84.7%	0.095	0.114	0.098	−0.061	0.477
Laya	72	80.6%	0.095	0.102	0.119	+0.034	0.523
Qwen3.5-9B, untrained	72	84.7%	0.107	0.115	0.119	−0.097	0.496

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
Drex Max	36	100.0%	0.014	0.024	0.001	−0.014	0.015
d1-ultra	36	100.0%	0.016	0.023	0.001	−0.016	0.017
Clef Omni	36	100.0%	0.047	0.053	0.006	−0.047	0.050
Perplexity Decider v1.2	36	100.0%	0.023	0.031	0.007	−0.023	0.028
Clef	36	100.0%	0.047	0.061	0.009	−0.047	0.053
Perplexity Decider Flash v1.2	36	100.0%	0.050	0.062	0.011	−0.050	0.057
JEV-27B	36	100.0%	0.063	0.078	0.018	−0.063	0.076
Jev 1.14	36	97.2%	0.056	0.067	0.020	−0.036	0.091
Drex v2.0	36	97.2%	0.048	0.079	0.021	−0.046	0.089
GPT-6 Luna Decisions	36	94.4%	0.056	0.065	0.021	+0.006	0.072
Mercury Decide	36	97.2%	0.030	0.013	0.026	+0.019	0.102
Jev 1.13	36	94.4%	0.076	0.082	0.027	−0.026	0.114
JEV-9B	36	94.4%	0.090	0.083	0.030	−0.026	0.113
Clef Flash	36	97.2%	0.109	0.088	0.034	−0.063	0.144
Drex v1.5	36	94.4%	0.068	0.063	0.036	+0.013	0.116
JPT-4B	36	94.4%	0.050	0.069	0.040	−0.005	0.133
Kev-4B	36	88.9%	0.102	0.086	0.054	+0.021	0.167
Strands Decider 2B	36	91.7%	0.116	0.129	0.078	−0.092	0.274
Solar Decide	36	88.9%	0.074	0.036	0.099	+0.074	0.348
Open-Jev-2B	36	86.1%	0.092	0.099	0.115	+0.002	0.349
Solar Decide Flash	36	86.1%	0.110	0.071	0.130	+0.042	0.470
Qwen3.5-9B, untrained	36	75.0%	0.089	0.141	0.161	+0.009	0.497
Laya	36	72.2%	0.176	0.162	0.214	+0.006	0.620

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
Jev 1.14	12	100.0%	0.000	0.000	0.000	−0.000	0.000
Drex Max	12	100.0%	0.000	0.001	0.000	−0.000	0.000
Mercury Decide	12	100.0%	0.000	0.000	0.000	−0.000	0.000
Jev 1.13	12	100.0%	0.003	0.005	0.000	−0.003	0.003
JEV-9B	12	100.0%	0.005	0.010	0.000	−0.005	0.005
Perplexity Decider v1.2	12	100.0%	0.006	0.010	0.000	−0.006	0.006
d1-ultra	12	100.0%	0.007	0.010	0.000	−0.007	0.007
JEV-27B	12	100.0%	0.005	0.011	0.000	−0.005	0.005
Perplexity Decider Flash v1.2	12	100.0%	0.009	0.014	0.000	−0.009	0.009
Drex v2.0	12	100.0%	0.022	0.033	0.001	−0.022	0.022
Clef	12	100.0%	0.038	0.061	0.002	−0.038	0.039
Drex v1.5	12	100.0%	0.019	0.035	0.002	−0.019	0.021
Solar Decide Flash	12	100.0%	0.036	0.052	0.003	−0.036	0.038
Clef Omni	12	100.0%	0.049	0.071	0.003	−0.049	0.050
Solar Decide	12	100.0%	0.038	0.062	0.003	−0.038	0.040
GPT-6 Luna Decisions	12	100.0%	0.036	0.064	0.004	−0.036	0.038
Clef Flash	12	100.0%	0.062	0.093	0.005	−0.062	0.064
JPT-4B	12	100.0%	0.082	0.104	0.010	−0.082	0.088
Kev-4B	12	100.0%	0.129	0.174	0.024	−0.129	0.143
Open-Jev-2B	12	100.0%	0.116	0.154	0.036	−0.116	0.143
Laya	12	91.7%	0.214	0.288	0.113	−0.214	0.451
Qwen3.5-9B, untrained	12	83.3%	0.294	0.273	0.154	−0.217	0.580
Strands Decider 2B	12	100.0%	0.394	0.230	0.162	−0.394	0.511

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
d1-ultra	111	83.8%	0.066	0.079	0.091	−0.001	0.384
Drex Max	111	83.8%	0.109	0.053	0.098	+0.075	0.510
Perplexity Decider v1.2	111	79.3%	0.054	0.081	0.108	+0.035	0.459
Jev 1.14	111	77.5%	0.075	0.092	0.110	−0.010	0.501
GPT-6 Luna Decisions	111	77.5%	0.085	0.081	0.115	+0.036	0.926
Drex v2.0	111	73.9%	0.068	0.094	0.127	+0.013	0.552
JPT-4B	103	79.6%	0.111	0.094	0.134	−0.018	0.582
Mercury Decide	111	79.3%	0.131	0.058	0.135	+0.121	0.682
Clef	111	75.7%	0.075	0.091	0.142	+0.021	0.604
Jev 1.13	111	72.1%	0.121	0.089	0.144	+0.056	0.609
Solar Decide	111	72.1%	0.153	0.073	0.170	+0.140	0.871
Drex v1.5	111	71.2%	0.154	0.072	0.183	+0.154	0.831
Strands Decider 2B	93	60.2%	0.104	0.112	0.192	+0.034	0.827
Clef Flash	111	61.3%	0.117	0.098	0.205	+0.117	0.819
Clef Omni	111	58.6%	0.137	0.105	0.213	+0.135	0.896
Kev-4B	111	54.1%	0.145	0.112	0.232	+0.096	0.962
Perplexity Decider Flash v1.2	111	64.0%	0.173	0.077	0.232	+0.154	0.993
Solar Decide Flash	111	56.8%	0.174	0.090	0.232	+0.166	0.964
JEV-9B	111	54.1%	0.214	0.094	0.239	+0.200	1.007
Qwen3.5-9B, untrained	111	58.6%	0.190	0.113	0.243	+0.069	0.968
Laya	111	32.4%	0.286	0.112	0.306	+0.215	1.400
Open-Jev-2B	111	42.3%	0.281	0.099	0.307	+0.262	1.434

This benchmark's reference answers were written by language models without human review, so it is left out of the summary above.

JPT-4B would not answer 8 records here, so its scores cover the rest.

Strands Decider 2B would not answer 18 records here, so its scores cover the rest.

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
d1-ultra	67	83.6%	0.083	0.103	0.073	−0.007	0.360
Perplexity Decider v1.2	67	83.6%	0.085	0.104	0.074	−0.022	0.407
Jev 1.14	67	80.6%	0.118	0.124	0.093	−0.044	0.506
Drex Max	67	82.1%	0.126	0.063	0.110	+0.093	0.626
GPT-6 Luna Decisions	67	76.1%	0.080	0.113	0.111	−0.004	0.538
Drex v2.0	67	79.1%	0.120	0.120	0.113	−0.058	0.556
Mercury Decide	67	79.1%	0.119	0.067	0.118	+0.116	0.719
Jev 1.13	67	74.6%	0.132	0.114	0.123	+0.029	0.612
JPT-4B	63	79.4%	0.121	0.124	0.127	−0.046	0.645
Clef	67	76.1%	0.105	0.124	0.136	+0.019	0.686
Drex v1.5	67	73.1%	0.136	0.092	0.168	+0.132	0.965
Solar Decide	67	68.7%	0.179	0.093	0.170	+0.165	1.060
Strands Decider 2B	54	57.4%	0.125	0.149	0.174	+0.049	0.944
Clef Flash	67	65.7%	0.117	0.129	0.179	+0.050	0.858
JEV-9B	67	53.7%	0.161	0.132	0.193	+0.147	0.987
Clef Omni	67	58.2%	0.160	0.125	0.205	+0.107	1.032
Kev-4B	67	58.2%	0.146	0.146	0.208	−0.002	1.045
Qwen3.5-9B, untrained	67	62.7%	0.160	0.144	0.209	−0.033	0.958
Solar Decide Flash	67	56.7%	0.179	0.119	0.210	+0.150	1.032
Perplexity Decider Flash v1.2	67	61.2%	0.194	0.106	0.233	+0.177	1.115
Open-Jev-2B	67	41.8%	0.212	0.141	0.252	+0.176	1.498
Laya	67	29.9%	0.242	0.138	0.261	+0.148	1.565

This benchmark's reference answers were written by language models without human review, so it is left out of the summary above.

JPT-4B would not answer 8 records here, so its scores cover the rest.

Strands Decider 2B would not answer 18 records here, so its scores cover the rest.

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
Drex Max	38	92.1%	0.051	0.072	0.064	+0.010	0.236
d1-ultra	38	86.8%	0.087	0.099	0.111	+0.023	0.380
GPT-6 Luna Decisions	38	84.2%	0.137	0.069	0.117	+0.094	1.634
Mercury Decide	38	86.8%	0.138	0.071	0.120	+0.068	0.417
JPT-4B	36	80.6%	0.094	0.133	0.135	+0.038	0.437
Jev 1.14	38	78.9%	0.106	0.133	0.141	+0.011	0.414
Solar Decide	38	78.9%	0.100	0.104	0.147	+0.087	0.476
Clef	38	78.9%	0.081	0.130	0.148	+0.011	0.441
Drex v2.0	38	71.1%	0.171	0.116	0.150	+0.114	0.490
Perplexity Decider v1.2	38	76.3%	0.166	0.102	0.155	+0.114	0.481
Jev 1.13	38	73.7%	0.135	0.135	0.178	+0.068	0.524
Drex v1.5	38	71.1%	0.191	0.112	0.187	+0.162	0.536
Strands Decider 2B	35	65.7%	0.088	0.160	0.213	+0.031	0.607
Clef Omni	38	63.2%	0.228	0.145	0.228	+0.169	0.645
Clef Flash	38	57.9%	0.254	0.140	0.244	+0.211	0.689
Solar Decide Flash	38	57.9%	0.187	0.146	0.246	+0.187	0.740
Perplexity Decider Flash v1.2	38	71.1%	0.299	0.132	0.248	+0.121	0.776
Kev-4B	38	50.0%	0.260	0.148	0.284	+0.249	0.778
Qwen3.5-9B, untrained	38	57.9%	0.305	0.146	0.302	+0.202	0.889
JEV-9B	38	57.9%	0.292	0.111	0.305	+0.269	0.941
Laya	38	42.1%	0.353	0.149	0.382	+0.284	1.029
Open-Jev-2B	38	47.4%	0.379	0.105	0.389	+0.379	1.252

This benchmark's reference answers were written by language models without human review, so it is left out of the summary above.

JPT-4B would not answer 8 records here, so its scores cover the rest.

Strands Decider 2B would not answer 18 records here, so its scores cover the rest.

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
Jev 1.14	6	33.3%	0.273	0.284	0.105	+0.243	0.990
Perplexity Decider Flash v1.2	6	50.0%	0.135	0.252	0.120	+0.106	1.011
Drex v2.0	6	33.3%	0.348	0.313	0.127	+0.165	0.901
GPT-6 Luna Decisions	6	50.0%	0.318	0.279	0.137	+0.122	0.771
Jev 1.13	6	33.3%	0.337	0.303	0.160	+0.283	1.124
Kev-4B	6	33.3%	0.382	0.369	0.162	+0.226	1.194
d1-ultra	6	66.7%	0.125	0.211	0.171	−0.084	0.676
Drex Max	6	50.0%	0.295	0.183	0.172	+0.287	0.951
Clef	6	50.0%	0.217	0.298	0.174	+0.111	0.725
Perplexity Decider v1.2	6	50.0%	0.350	0.249	0.182	+0.164	0.904
Clef Omni	6	33.3%	0.401	0.331	0.196	+0.235	0.975
JPT-4B	4	75.0%	0.392	0.369	0.216	−0.083	0.893
Clef Flash	6	33.3%	0.324	0.313	0.238	+0.281	1.210
Strands Decider 2B	4	50.0%	0.483	0.375	0.253	−0.137	1.168
Qwen3.5-9B, untrained	6	16.7%	0.359	0.297	0.257	+0.359	1.578
Solar Decide	6	66.7%	0.385	0.155	0.325	+0.197	1.266
Laya	6	0.0%	0.537	0.262	0.330	+0.537	1.914
JEV-9B	6	33.3%	0.378	0.321	0.332	+0.353	1.645
Drex v1.5	6	50.0%	0.374	0.199	0.335	+0.363	1.205
Solar Decide Flash	6	50.0%	0.392	0.263	0.395	+0.212	1.627
Open-Jev-2B	6	16.7%	0.593	0.332	0.401	+0.491	1.875
Mercury Decide	6	33.3%	0.529	0.242	0.413	+0.505	1.952

This benchmark's reference answers were written by language models without human review, so it is left out of the summary above.

JPT-4B would not answer 8 records here, so its scores cover the rest.

Strands Decider 2B would not answer 18 records here, so its scores cover the rest.

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
Perplexity Decider v1.2	282	97.5%	0.026	0.020	0.024	−0.007	0.115
d1-ultra	282	97.2%	0.032	0.026	0.025	−0.021	0.115
Drex Max	282	96.8%	0.036	0.012	0.033	+0.020	0.200
Jev 1.14	282	94.7%	0.039	0.035	0.052	−0.034	0.364
Jev 1.13	282	92.6%	0.047	0.038	0.066	−0.033	0.501
Clef	282	90.8%	0.021	0.039	0.074	−0.013	0.281
Mercury Decide	282	91.5%	0.080	0.008	0.078	+0.078	0.450
JEV-9B	282	87.2%	0.046	0.041	0.099	+0.000	0.337
GPT-6 Luna Decisions	282	85.5%	0.047	0.042	0.103	+0.023	0.560
Perplexity Decider Flash v1.2	282	85.8%	0.082	0.031	0.110	+0.073	0.433
Clef Omni	282	83.7%	0.066	0.049	0.116	−0.004	0.424
Solar Decide	282	86.5%	0.095	0.025	0.121	+0.081	0.592
Drex v2.0	282	84.0%	0.080	0.036	0.126	+0.072	0.540
Kev-4B	282	77.0%	0.054	0.057	0.143	+0.019	0.503
Clef Flash	282	75.5%	0.097	0.052	0.150	+0.047	0.593
Drex v1.5	282	82.3%	0.127	0.031	0.153	+0.097	0.730
JPT-4B	282	78.4%	0.105	0.043	0.160	+0.083	0.585
Open-Jev-2B	282	74.5%	0.090	0.051	0.169	+0.055	0.728
Solar Decide Flash	282	69.9%	0.108	0.053	0.209	+0.105	0.904
Strands Decider 2B	282	57.1%	0.120	0.067	0.220	+0.102	0.876
Qwen3.5-9B, untrained	282	61.0%	0.107	0.064	0.230	+0.083	0.822
Laya	282	45.0%	0.114	0.071	0.265	+0.107	1.115

This benchmark's reference answers were written by language models without human review, so it is left out of the summary above.

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
Drex Max	116	96.6%	0.041	0.014	0.037	+0.023	0.282
Perplexity Decider v1.2	116	96.6%	0.050	0.026	0.038	+0.007	0.200
d1-ultra	116	94.0%	0.036	0.042	0.042	+0.001	0.198
Jev 1.14	116	92.2%	0.084	0.054	0.075	−0.016	0.695
Jev 1.13	116	87.9%	0.118	0.058	0.102	+0.013	0.985
Clef	116	87.1%	0.063	0.069	0.115	−0.012	0.428
JEV-9B	116	83.6%	0.068	0.071	0.124	+0.018	0.446
GPT-6 Luna Decisions	116	78.4%	0.077	0.076	0.128	+0.041	0.530
Perplexity Decider Flash v1.2	116	83.6%	0.122	0.040	0.129	+0.118	0.550
Clef Omni	116	82.8%	0.076	0.077	0.139	−0.031	0.561
Solar Decide	116	83.6%	0.131	0.021	0.145	+0.131	0.797
Drex v2.0	116	78.4%	0.169	0.054	0.175	+0.131	0.851
Mercury Decide	116	81.0%	0.180	0.010	0.176	+0.180	1.042
Open-Jev-2B	116	69.8%	0.119	0.094	0.177	+0.054	0.844
Kev-4B	116	73.3%	0.106	0.082	0.177	−0.001	0.664
Clef Flash	116	68.1%	0.135	0.095	0.200	+0.083	0.841
JPT-4B	116	74.1%	0.157	0.071	0.202	+0.107	0.791
Drex v1.5	116	75.9%	0.202	0.055	0.214	+0.151	1.103
Laya	116	37.9%	0.090	0.097	0.247	+0.073	1.326
Qwen3.5-9B, untrained	116	57.8%	0.167	0.094	0.248	+0.135	0.957
Strands Decider 2B	116	54.3%	0.182	0.088	0.254	+0.181	1.084
Solar Decide Flash	116	54.3%	0.254	0.078	0.279	+0.254	1.340

This benchmark's reference answers were written by language models without human review, so it is left out of the summary above.

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
d1-ultra	112	99.1%	0.021	0.022	0.010	−0.009	0.035
Perplexity Decider v1.2	112	98.2%	0.023	0.019	0.013	+0.004	0.038
Mercury Decide	112	98.2%	0.015	0.004	0.015	+0.015	0.049
Clef	112	96.4%	0.024	0.033	0.027	−0.009	0.105
Jev 1.14	112	97.3%	0.053	0.038	0.027	−0.023	0.101
Drex Max	112	97.3%	0.031	0.011	0.028	+0.019	0.145
Jev 1.13	112	95.5%	0.074	0.045	0.029	−0.044	0.132
JEV-9B	112	95.5%	0.031	0.042	0.036	−0.007	0.115
Clef Omni	112	93.8%	0.053	0.045	0.036	−0.006	0.128
JPT-4B	112	93.8%	0.049	0.043	0.053	+0.007	0.197
Perplexity Decider Flash v1.2	112	93.8%	0.063	0.027	0.056	+0.038	0.225
Kev-4B	112	90.2%	0.062	0.048	0.060	+0.015	0.190
Drex v2.0	112	92.0%	0.064	0.036	0.061	+0.038	0.222
GPT-6 Luna Decisions	112	92.9%	0.069	0.046	0.064	+0.003	0.419
Clef Flash	112	90.2%	0.087	0.056	0.065	−0.009	0.222
Solar Decide	112	91.1%	0.090	0.040	0.087	+0.026	0.385
Drex v1.5	112	87.5%	0.095	0.030	0.107	+0.090	0.493
Open-Jev-2B	112	86.6%	0.069	0.058	0.114	+0.022	0.369
Solar Decide Flash	112	83.9%	0.127	0.065	0.133	+0.007	0.472
Strands Decider 2B	112	67.9%	0.126	0.083	0.185	+0.074	0.540
Qwen3.5-9B, untrained	112	67.0%	0.091	0.085	0.214	+0.071	0.617
Laya	112	50.0%	0.181	0.086	0.289	+0.181	0.812

This benchmark's reference answers were written by language models without human review, so it is left out of the summary above.

Model	Questions	Accuracy	ECE	Noise floor	Brier score	Over/underconfidence	Log loss
Mercury Decide	54	100.0%	0.009	0.015	0.002	−0.009	0.010
Perplexity Decider v1.2	54	98.1%	0.080	0.072	0.018	−0.060	0.093
d1-ultra	54	100.0%	0.093	0.072	0.018	−0.093	0.104
Drex Max	54	96.3%	0.040	0.024	0.036	+0.017	0.139
Jev 1.14	54	94.4%	0.108	0.112	0.053	−0.095	0.199
Jev 1.13	54	96.3%	0.141	0.094	0.066	−0.110	0.229
Clef	54	87.0%	0.102	0.099	0.085	−0.024	0.334
Drex v1.5	54	85.2%	0.093	0.098	0.120	−0.005	0.421
GPT-6 Luna Decisions	54	85.2%	0.137	0.094	0.128	+0.028	0.917
Solar Decide	54	83.3%	0.114	0.075	0.137	+0.087	0.583
Drex v2.0	54	79.6%	0.077	0.111	0.153	+0.016	0.533
JEV-9B	54	77.8%	0.126	0.116	0.174	−0.024	0.564
Perplexity Decider Flash v1.2	54	74.1%	0.118	0.111	0.183	+0.050	0.612
Clef Flash	54	61.1%	0.182	0.142	0.217	+0.085	0.828
Solar Decide Flash	54	74.1%	0.195	0.132	0.218	−0.013	0.862
Strands Decider 2B	54	40.7%	0.127	0.130	0.218	−0.009	1.125
Qwen3.5-9B, untrained	54	55.6%	0.122	0.156	0.224	−0.006	0.959
Clef Omni	54	64.8%	0.217	0.136	0.230	+0.056	0.745
Kev-4B	54	57.4%	0.187	0.157	0.241	+0.071	0.804
Laya	54	50.0%	0.144	0.127	0.252	+0.027	1.290
Open-Jev-2B	54	59.3%	0.217	0.138	0.269	+0.122	1.224
JPT-4B	54	55.6%	0.209	0.128	0.290	+0.188	0.948

This benchmark's reference answers were written by language models without human review, so it is left out of the summary above.

Checking our runs against published scores

Clean accuracy on Kev's six partitions, and their mean, the "Kev" score in AutoTrust's and NeoHorse's model cards.

Model	decision-v7 dev	decision-v7 test	transfer-v4 dev	transfer-v4 test	transfer-v9 dev	transfer-v9 test	Kev score	Published
JEV-27B	84.2	82.1	83.7	86.9	77.0	78.1	81.99	83.75
JEV-9B	83.3	79.7	81.4	82.3	74.0	72.9	78.93	—
Qwen3.5-9B, untrained	75.6	73.3	73.0	77.3	62.7	65.5	71.23	—
Kev-4B	87.2	86.5	81.7	83.8	73.4	73.4	80.99	81.47
JPT-4B	83.2	80.8	80.2	83.9	71.2	72.4	78.61	—
Strands Decider 2B	82.5	80.8	72.3	77.6	64.0	66.6	73.96	—
Open-Jev-2B	74.4	72.9	68.0	74.2	58.7	61.9	68.36	—
Laya	66.5	65.2	64.8	65.7	50.9	51.9	60.81	61.30

Reliability diagram

Model Benchmark Question type

What we found

Perplexity’s Decider v1.2 leads, with Drex Max close behind. On these benchmarks it is right more than nine times out of ten, and its confidence tracks its accuracy almost exactly. Jev sits in the middle of the pack here, behind several hosted models it beats on exam questions.

Most models are surer than they should be on fact checks. VitaminC asks whether evidence supports a claim, and nearly every model, Jev included, is overconfident on it. Among hosted models, only Perplexity’s two and d1-ultra stay calibrated on it.

Question type matters. Many models handle yes/no questions well and struggle more with ratings, so a model’s overall rank can hide a weak spot. The chart above shows each model by question type.

How the models were asked

Hosted models were asked through OpenRouter in October 2026, each pinned to its maker’s own servers, with every question in its native form. Open models were run through the servers their authors ship, also with native questions. JEV models answered multiple-choice and yes/no questions through their own decision slots. Their rating slots are a fixed 0 to 5 scale without level descriptions, so rating questions were asked as multiple choice over the level descriptions. We also tried JEV’s rating slots with the descriptions written into the question; that was less accurate for both JEV models (62.1% against 64.9% for JEV-27B, 54.0% against 61.5% for JEV-9B).

What to keep in mind

Our scores reproduce the published ones closely, with one exception. Kev-4B and Laya land within half a point of the Kev scores NeoHorse reports. JEV-27B lands 1.8 points below AutoTrust’s figure; the rating format does not explain it, and the likely causes are half precision on our older GPUs and how structured inputs are turned into text.
Kev-4B was trained on decision-v7’s training split, so the decision-v7 partitions are in distribution for it.
Some hosted models skipped records. Solar Decide and Solar Decide Flash take at most 26 options and could not answer 116 of Kev’s records. GPT-6 Luna Decisions refused 11. Their scores leave those out.
JPT-4B’s server could not answer some inputs, mostly conversations stored as lists of messages with a “customer” role. Those records are left out of its scores, and the per-benchmark table says how many.
JEV-27B has no results on JevBench’s hard tier or Nimble. Their longest prompts do not fit on our 11 GB GPUs; neither benchmark is in the summary.
Only clean questions are scored. Kev’s suites also contain shuffled and “none of these” variants, which are not used here.

Decision benchmarks leaderboard

By question type

Each benchmark

Checking our runs against published scores

Reliability diagram

What we found

How the models were asked

What to keep in mind

Data