JEV calibration leaderboard

Whether Jev, thirteen other hosted decision models and eight open-weight ones are right as often as they say they are, all asked the same exam-style questions.

Twenty-three decision models, scored on accuracy, expected calibration error (ECE) and the Brier score. Every model was asked exactly the same questions. Pick a dataset to see every model on it, or a model to see its reliability diagram.

Dataset

Rank	Model	Accuracy ↑	ECE ↓	Brier score ↓
11 ⊸ 5	Perplexity Decider v1.2Perplexity · API only	86.1% ±1.0%	0.046 ±0.008	0.080 ±0.005
21 ⊸ 8	d1-ultraLiquid AI · API only	85.7% ±1.0%	0.045 ±0.007	0.084 ±0.004
31 ⊸ 8	Jev 1.13TypeSafe · API only	86.9% ±1.0%	0.046 ±0.007	0.085 ±0.005
41 ⊸ 8	Jev 1.14TypeSafe · API only	86.7% ±1.0%	0.047 ±0.007	0.087 ±0.005
51 ⊸ 8	Clef OmniCloudflare · API only	84.9% ±1.1%	0.068 ±0.008	0.088 ±0.005
62 ⊸ 8	ClefCloudflare · API only	84.7% ±1.1%	0.048 ±0.007	0.090 ±0.005
72 ⊸ 8	Clef FlashCloudflare · API only	83.5% ±1.1%	0.057 ±0.007	0.090 ±0.004
82 ⊸ 8	Drex MaxNace AI · API only	85.9% ±1.1%	0.062 ±0.009	0.090 ±0.006
99	Drex v2.0Nace AI · API only	81.4% ±1.1%	0.058 ±0.007	0.102 ±0.004
1010 ⊸ 13	JEV-27BAutoTrust AI · Apache-2.0	79.5% ±1.1%	0.068 ±0.008	0.112 ±0.004
1110 ⊸ 14	Perplexity Decider Flash v1.2Perplexity · API only	80.2% ±1.1%	0.070 ±0.009	0.115 ±0.006
1210 ⊸ 14	GPT-6 Luna DecisionsOpenAI · API only	80.1% ±1.1%	0.057 ±0.008	0.115 ±0.005
1310 ⊸ 15	Mercury DecideInception · API only	80.5% ±1.0%	0.104 ±0.009	0.120 ±0.007
1411 ⊸ 16	Drex v1.5Nace AI · API only	77.8% ±1.1%	0.087 ±0.011	0.126 ±0.006
1513 ⊸ 17	JPT-4Bkirp · CC BY-NC 4.0	75.2% ±1.3%	0.064 ±0.009	0.131 ±0.005
1614 ⊸ 17	JEV-9BAutoTrust AI · Apache-2.0	74.0% ±1.1%	0.070 ±0.009	0.136 ±0.005
1715 ⊸ 19	Solar DecideUpstage · API only	80.2% ±1.2%	0.113 ±0.010	0.141 ±0.008
1817 ⊸ 19	Kev-4BJared Palmer · Apache-2.0	72.1% ±1.2%	0.087 ±0.009	0.147 ±0.004
1917 ⊸ 21	Solar Decide FlashUpstage · API only	71.5% ±1.2%	0.070 ±0.010	0.155 ±0.005
2019 ⊸ 21	Qwen3.5-9B, untrainedAlibaba Qwen · Apache-2.0	70.5% ±1.2%	0.124 ±0.010	0.162 ±0.003
2119 ⊸ 21	Strands Decider 2BAmazon Strands Labs · Apache-2.0	66.8% ±1.2%	0.089 ±0.010	0.164 ±0.005
2222	Open-Jev-2BZefan Cai · Apache-2.0	58.3% ±1.3%	0.142 ±0.010	0.213 ±0.004
2323	LayaConvai Innovations · Apache-2.0	40.8% ±1.1%	0.164 ±0.011	0.233 ±0.005

Means over the eight datasets every model could be asked: Under Pressure with 4 and with 16 options, MMLU, ARC-Challenge, CommonsenseQA, OpenBookQA, TruthfulQA and GPQA. GPT-6 Luna Decisions would not answer 5 of these questions, which its scores leave out. Ranked by Brier score. Each ± and error bar is the 95% interval from resampling questions, and the small numbers under each rank are the range of ranks those intervals allow.

Rank	Model	Questions	Accuracy ↑	ECE ↓	Brier score ↓	Over/underconfidence	90% set size ↓
1	Clef OmniCloudflare	800	87.8%	0.038	0.082	+0.005	1.06
2	ClefCloudflare	800	87.6%	0.039	0.089	+0.033	1.07
3	Perplexity Decider v1.2Perplexity	800	87.8%	0.057	0.092	+0.049	1.08
4	Clef FlashCloudflare	800	86.8%	0.035	0.092	+0.035	1.07
5	Perplexity Decider Flash v1.2Perplexity	800	87.3%	0.049	0.093	+0.034	1.08
6	Drex MaxNace AI	800	87.5%	0.072	0.098	+0.063	1.08
7	JPT-4Bkirp	800	82.6%	0.048	0.113	+0.023	1.19
8	d1-ultraLiquid AI	800	81.8%	0.040	0.115	+0.040	1.28
9	Drex v2.0Nace AI	800	81.4%	0.040	0.116	+0.032	1.26
10	Drex v1.5Nace AI	800	84.1%	0.078	0.116	+0.065	1.17
11	Strands Decider 2BAmazon Strands Labs	800	81.5%	0.055	0.121	+0.041	1.25
12	Kev-4BJared Palmer	800	78.1%	0.043	0.130	−0.017	1.32
13	GPT-6 Luna DecisionsOpenAI	800	81.3%	0.085	0.135	+0.081	1.32
14	JEV-27BAutoTrust AI	800	78.4%	0.084	0.135	+0.080	1.35
15	Mercury DecideInception	800	81.6%	0.117	0.136	+0.116	1.24
16	Jev 1.14TypeSafe	800	80.0%	0.097	0.136	+0.094	1.33
17	Jev 1.13TypeSafe	800	79.6%	0.111	0.142	+0.105	1.35
18	JEV-9BAutoTrust AI	800	76.5%	0.074	0.145	+0.063	1.41
19	Qwen3.5-9B, untrainedAlibaba Qwen	800	74.4%	0.085	0.150	−0.076	1.67
20	LayaConvai Innovations	800	76.4%	0.038	0.153	−0.008	1.57
21	Solar Decide FlashUpstage	800	76.0%	0.085	0.160	+0.064	1.64
22	Open-Jev-2BZefan Cai	800	71.9%	0.107	0.175	−0.107	1.85
23	Solar DecideUpstage	800	78.1%	0.163	0.187	+0.163	2.31

Ranked by Brier score on Under Pressure, 4 options. Over/underconfidence is average confidence minus accuracy: negative means the model undersells itself. Set sizes come from 5-fold conformal prediction at 90% coverage, on the same questions as the other columns.

Rank	Model	Questions	Accuracy ↑	ECE ↓	Brier score ↓	Over/underconfidence	90% set size ↓
1	Perplexity Decider v1.2Perplexity	800	85.6%	0.050	0.099	+0.033	1.12
2	Perplexity Decider Flash v1.2Perplexity	800	84.9%	0.045	0.104	+0.031	1.16
3	Clef OmniCloudflare	800	86.6%	0.102	0.105	−0.094	1.12
4	ClefCloudflare	800	84.6%	0.045	0.108	+0.007	1.18
5	Clef FlashCloudflare	800	84.0%	0.044	0.113	−0.024	1.22
6	Drex MaxNace AI	800	84.8%	0.092	0.117	+0.075	1.19
7	Drex v2.0Nace AI	800	79.1%	0.047	0.131	+0.020	1.44
8	Drex v1.5Nace AI	800	79.9%	0.079	0.135	+0.075	1.38
9	JPT-4Bkirp	800	77.5%	0.039	0.138	+0.011	1.42
10	d1-ultraLiquid AI	800	77.5%	0.061	0.138	+0.010	1.43
11	GPT-6 Luna DecisionsOpenAI	798	77.2%	0.061	0.146	+0.024	1.68
12	Strands Decider 2BAmazon Strands Labs	800	78.0%	0.083	0.146	+0.048	1.52
13	Jev 1.14TypeSafe	800	76.3%	0.092	0.147	+0.091	1.56
14	Jev 1.13TypeSafe	800	76.6%	0.101	0.151	+0.094	1.56
15	Kev-4BJared Palmer	800	73.9%	0.107	0.158	−0.107	1.67
16	Mercury DecideInception	800	78.9%	0.145	0.159	+0.137	1.57
17	JEV-27BAutoTrust AI	800	73.9%	0.110	0.164	+0.102	1.74
18	JEV-9BAutoTrust AI	800	70.8%	0.110	0.167	+0.097	2.34
19	Qwen3.5-9B, untrainedAlibaba Qwen	800	63.6%	0.112	0.169	−0.102	4.02
20	Solar Decide FlashUpstage	800	71.8%	0.086	0.169	+0.054	2.41
21	Solar DecideUpstage	800	73.8%	0.187	0.209	+0.180	5.53
22	Open-Jev-2BZefan Cai	800	61.3%	0.240	0.246	−0.240	3.18
23	LayaConvai Innovations	800	64.9%	0.237	0.249	+0.236	3.63

Ranked by Brier score on Under Pressure, 16 options. Over/underconfidence is average confidence minus accuracy: negative means the model undersells itself. Set sizes come from 5-fold conformal prediction at 90% coverage, on the same questions as the other columns. GPT-6 Luna Decisions would not answer 2 questions here.

Rank	Model	Questions	Accuracy ↑	ECE ↓	Brier score ↓	Over/underconfidence	90% set size ↓
1	Perplexity Decider v1.2Perplexity	800	76.4%	0.056	0.135	+0.049	1.51
2	Perplexity Decider Flash v1.2Perplexity	800	75.1%	0.082	0.150	+0.065	1.88
3	d1-ultraLiquid AI	800	67.9%	0.042	0.151	+0.027	2.50
4	ClefCloudflare	800	74.9%	0.057	0.158	−0.003	1.59
5	Jev 1.14TypeSafe	800	66.8%	0.099	0.161	+0.099	2.44
6	Drex v2.0Nace AI	800	65.3%	0.047	0.162	+0.032	2.85
7	Clef FlashCloudflare	800	73.6%	0.078	0.169	−0.040	1.70
8	Drex MaxNace AI	800	73.1%	0.129	0.170	+0.124	2.18
9	Jev 1.13TypeSafe	800	65.9%	0.131	0.174	+0.129	2.43
10	Clef OmniCloudflare	800	75.9%	0.172	0.181	−0.169	1.52
11	JEV-9BAutoTrust AI	800	60.3%	0.116	0.186	+0.109	5.21
12	Drex v1.5Nace AI	800	69.1%	0.132	0.187	+0.130	2.37
13	GPT-6 Luna DecisionsOpenAI	794	61.8%	0.084	0.189	+0.058	20.83
14	Mercury DecideInception	800	69.4%	0.219	0.229	+0.207	6.82

Ranked by Brier score on Under Pressure, 64 options. Over/underconfidence is average confidence minus accuracy: negative means the model undersells itself. Set sizes come from 5-fold conformal prediction at 90% coverage, on the same questions as the other columns. GPT-6 Luna Decisions would not answer 6 questions here.

Rank	Model	Questions	Accuracy ↑	ECE ↓	Brier score ↓	Over/underconfidence	90% set size ↓
1	Clef OmniCloudflare	2,000	92.3%	0.069	0.059	−0.069	0.95
2	Jev 1.13TypeSafe	2,000	92.3%	0.024	0.059	+0.012	0.96
3	Clef FlashCloudflare	2,000	91.3%	0.053	0.060	−0.048	0.97
4	Jev 1.14TypeSafe	2,000	92.0%	0.027	0.061	+0.003	0.96
5	d1-ultraLiquid AI	2,000	90.3%	0.024	0.063	−0.019	0.99
6	ClefCloudflare	2,000	89.3%	0.030	0.070	−0.021	1.02
7	Perplexity Decider v1.2Perplexity	2,000	88.9%	0.027	0.076	+0.019	1.03
8	Drex v2.0Nace AI	2,000	85.3%	0.030	0.089	−0.019	1.12
9	Drex MaxNace AI	2,000	87.6%	0.056	0.089	+0.051	1.07
10	GPT-6 Luna DecisionsOpenAI	2,000	83.8%	0.042	0.103	+0.040	1.21
11	JEV-27BAutoTrust AI	2,000	83.5%	0.030	0.105	−0.014	1.19
12	Mercury DecideInception	2,000	83.0%	0.087	0.121	+0.086	1.23
13	Solar DecideUpstage	2,000	83.4%	0.095	0.129	+0.093	1.35
14	Perplexity Decider Flash v1.2Perplexity	2,000	79.2%	0.092	0.135	+0.091	1.43
15	JEV-9BAutoTrust AI	2,000	75.6%	0.014	0.136	+0.006	1.55
16	Drex v1.5Nace AI	2,000	77.8%	0.087	0.139	+0.083	1.45
17	JPT-4Bkirp	2,000	73.7%	0.054	0.152	+0.054	1.68
18	Kev-4BJared Palmer	2,000	70.9%	0.051	0.153	−0.045	1.82
19	Solar Decide FlashUpstage	2,000	72.2%	0.058	0.159	+0.052	1.75
20	Qwen3.5-9B, untrainedAlibaba Qwen	2,000	72.8%	0.101	0.163	−0.096	1.77
21	Strands Decider 2BAmazon Strands Labs	2,000	59.9%	0.025	0.193	+0.017	2.37
22	Open-Jev-2BZefan Cai	2,000	52.3%	0.056	0.210	−0.033	2.78
23	LayaConvai Innovations	2,000	32.6%	0.131	0.238	+0.131	3.45

Ranked by Brier score on MMLU. Over/underconfidence is average confidence minus accuracy: negative means the model undersells itself. Set sizes come from 5-fold conformal prediction at 90% coverage, on the same questions as the other columns.

Rank	Model	Questions	Accuracy ↑	ECE ↓	Brier score ↓	Over/underconfidence	90% set size ↓
1	Jev 1.14TypeSafe	2,000	80.8%	0.048	0.123	−0.017	1.44
2	Jev 1.13TypeSafe	2,000	81.7%	0.061	0.127	−0.011	1.38
3	d1-ultraLiquid AI	2,000	73.9%	0.063	0.139	−0.062	1.99
4	Drex v2.0Nace AI	2,000	72.0%	0.043	0.140	−0.040	2.04
5	Perplexity Decider v1.2Perplexity	2,000	70.5%	0.042	0.143	+0.028	2.12
6	ClefCloudflare	2,000	63.4%	0.037	0.155	+0.020	2.97
7	Drex MaxNace AI	2,000	70.5%	0.106	0.156	+0.106	2.20
8	Clef OmniCloudflare	2,000	67.4%	0.063	0.158	−0.063	2.63
9	Clef FlashCloudflare	2,000	62.5%	0.036	0.165	−0.036	3.23
10	Solar DecideUpstage	2,000	72.3%	0.147	0.171	+0.145	3.13
11	GPT-6 Luna DecisionsOpenAI	1,992	58.0%	0.067	0.182	+0.033	4.03
12	JEV-9BAutoTrust AI	2,000	53.7%	0.033	0.188	+0.008	4.29
13	Perplexity Decider Flash v1.2Perplexity	2,000	56.7%	0.116	0.196	+0.116	3.79
14	Drex v1.5Nace AI	2,000	56.6%	0.145	0.202	+0.145	3.51
15	Solar Decide FlashUpstage	2,000	44.3%	0.134	0.211	+0.134	5.30
16	Mercury DecideInception	2,000	61.9%	0.167	0.212	+0.166	2.99

Ranked by Brier score on MMLU-Pro. Over/underconfidence is average confidence minus accuracy: negative means the model undersells itself. Set sizes come from 5-fold conformal prediction at 90% coverage, on the same questions as the other columns. GPT-6 Luna Decisions would not answer 8 questions here.

Rank	Model	Questions	Accuracy ↑	ECE ↓	Brier score ↓	Over/underconfidence	90% set size ↓
1	Clef FlashCloudflare	1,172	98.9%	0.053	0.013	−0.052	0.90
2	Clef OmniCloudflare	1,172	98.0%	0.064	0.017	−0.057	0.90
3	d1-ultraLiquid AI	1,172	97.8%	0.018	0.018	−0.016	0.91
4	ClefCloudflare	1,172	97.4%	0.027	0.018	−0.017	0.91
5	Jev 1.14TypeSafe	1,172	97.7%	0.009	0.018	+0.003	0.91
6	Jev 1.13TypeSafe	1,172	97.8%	0.009	0.018	+0.007	0.92
7	Drex MaxNace AI	1,172	97.4%	0.011	0.020	+0.009	0.91
8	Perplexity Decider v1.2Perplexity	1,172	97.0%	0.010	0.021	+0.007	0.91
9	Mercury DecideInception	1,172	96.8%	0.017	0.025	+0.013	0.91
10	JEV-27BAutoTrust AI	1,172	96.9%	0.025	0.026	−0.024	0.91
11	Drex v2.0Nace AI	1,172	96.6%	0.046	0.027	−0.041	0.91
12	GPT-6 Luna DecisionsOpenAI	1,171	96.0%	0.023	0.032	+0.004	0.92
13	Drex v1.5Nace AI	1,172	95.6%	0.015	0.033	−0.005	0.92
14	Perplexity Decider Flash v1.2Perplexity	1,172	95.6%	0.015	0.034	+0.011	0.91
15	Solar DecideUpstage	1,172	95.1%	0.013	0.042	+0.012	0.94
16	JEV-9BAutoTrust AI	1,172	93.8%	0.047	0.043	−0.038	0.92
17	JPT-4Bkirp	1,172	92.2%	0.032	0.054	−0.027	0.96
18	Kev-4BJared Palmer	1,172	92.2%	0.107	0.067	−0.104	0.96
19	Solar Decide FlashUpstage	1,172	90.5%	0.024	0.071	−0.020	0.99
20	Qwen3.5-9B, untrainedAlibaba Qwen	1,172	91.2%	0.175	0.092	−0.170	0.97
21	Strands Decider 2BAmazon Strands Labs	1,172	84.1%	0.094	0.106	−0.080	1.20
22	Open-Jev-2BZefan Cai	1,172	75.5%	0.206	0.186	−0.205	1.65
23	LayaConvai Innovations	1,172	30.5%	0.174	0.249	+0.174	3.38

Ranked by Brier score on ARC-Challenge. Over/underconfidence is average confidence minus accuracy: negative means the model undersells itself. Set sizes come from 5-fold conformal prediction at 90% coverage, on the same questions as the other columns. GPT-6 Luna Decisions would not answer 1 questions here.

Rank	Model	Questions	Accuracy ↑	ECE ↓	Brier score ↓	Over/underconfidence	90% set size ↓
1	Clef OmniCloudflare	1,221	94.1%	0.077	0.048	−0.071	0.93
2	Clef FlashCloudflare	1,221	93.0%	0.076	0.054	−0.064	0.95
3	ClefCloudflare	1,221	90.5%	0.043	0.066	−0.032	1.00
4	d1-ultraLiquid AI	1,221	90.3%	0.051	0.069	−0.046	1.00
5	Perplexity Decider v1.2Perplexity	1,221	90.0%	0.030	0.072	+0.018	1.01
6	Drex MaxNace AI	1,221	89.4%	0.018	0.073	+0.013	1.01
7	Drex v2.0Nace AI	1,221	87.2%	0.046	0.085	−0.035	1.08
8	Jev 1.13TypeSafe	1,221	87.8%	0.028	0.086	+0.023	1.07
9	Jev 1.14TypeSafe	1,221	87.4%	0.028	0.087	+0.017	1.07
10	Perplexity Decider Flash v1.2Perplexity	1,221	85.4%	0.044	0.096	+0.038	1.13
11	Drex v1.5Nace AI	1,221	84.0%	0.045	0.103	+0.031	1.16
12	Mercury DecideInception	1,221	85.0%	0.064	0.104	+0.062	1.15
13	JEV-27BAutoTrust AI	1,221	84.0%	0.034	0.106	−0.034	1.17
14	JEV-9BAutoTrust AI	1,221	80.2%	0.044	0.128	−0.040	1.35
15	GPT-6 Luna DecisionsOpenAI	1,219	81.6%	0.062	0.129	+0.053	1.30
16	JPT-4Bkirp	1,221	76.7%	0.030	0.142	−0.023	1.46
17	Kev-4BJared Palmer	1,221	78.2%	0.104	0.147	−0.102	1.45
18	Strands Decider 2BAmazon Strands Labs	1,221	77.0%	0.086	0.151	−0.084	1.56
19	Solar DecideUpstage	1,221	79.9%	0.142	0.164	+0.142	1.90
20	Qwen3.5-9B, untrainedAlibaba Qwen	1,221	75.3%	0.135	0.172	−0.135	1.68
21	Solar Decide FlashUpstage	1,221	68.6%	0.061	0.183	+0.055	2.05
22	LayaConvai Innovations	1,221	45.3%	0.077	0.246	+0.067	3.57
23	Open-Jev-2BZefan Cai	1,221	66.2%	0.248	0.258	−0.248	2.23

Ranked by Brier score on CommonsenseQA. Over/underconfidence is average confidence minus accuracy: negative means the model undersells itself. Set sizes come from 5-fold conformal prediction at 90% coverage, on the same questions as the other columns. GPT-6 Luna Decisions would not answer 2 questions here.

Rank	Model	Questions	Accuracy ↑	ECE ↓	Brier score ↓	Over/underconfidence	90% set size ↓
1	Perplexity Decider v1.2Perplexity	500	97.8%	0.019	0.016	−0.012	0.91
2	d1-ultraLiquid AI	500	98.2%	0.047	0.020	−0.047	0.90
3	Drex MaxNace AI	500	96.0%	0.019	0.022	+0.011	0.91
4	Jev 1.13TypeSafe	500	95.6%	0.024	0.028	−0.014	0.91
5	Jev 1.14TypeSafe	500	95.2%	0.025	0.030	−0.013	0.92
6	Clef OmniCloudflare	500	95.6%	0.069	0.035	−0.050	0.91
7	Mercury DecideInception	500	96.2%	0.028	0.036	+0.002	0.93
8	ClefCloudflare	500	94.6%	0.040	0.040	−0.025	0.92
9	Clef FlashCloudflare	500	94.6%	0.056	0.043	−0.048	0.93
10	JEV-27BAutoTrust AI	500	95.8%	0.070	0.044	−0.070	0.92
11	Drex v1.5Nace AI	500	92.4%	0.030	0.049	−0.014	0.95
12	Perplexity Decider Flash v1.2Perplexity	500	93.2%	0.030	0.052	+0.007	0.95
13	Drex v2.0Nace AI	500	93.8%	0.078	0.054	−0.076	0.93
14	Solar DecideUpstage	500	92.8%	0.039	0.064	+0.019	0.97
15	GPT-6 Luna DecisionsOpenAI	500	86.6%	0.029	0.080	+0.004	1.11
16	JEV-9BAutoTrust AI	500	88.4%	0.087	0.089	−0.084	1.05
17	JPT-4Bkirp	500	81.6%	0.053	0.108	−0.031	1.31
18	Kev-4BJared Palmer	500	82.4%	0.116	0.131	−0.116	1.34
19	Solar Decide FlashUpstage	500	77.2%	0.045	0.139	−0.015	1.61
20	Strands Decider 2BAmazon Strands Labs	500	77.6%	0.095	0.141	−0.081	1.49
21	Qwen3.5-9B, untrainedAlibaba Qwen	500	80.6%	0.167	0.143	−0.164	1.41
22	Open-Jev-2BZefan Cai	500	63.8%	0.165	0.212	−0.160	2.16
23	LayaConvai Innovations	500	39.0%	0.124	0.256	+0.123	3.34

Ranked by Brier score on OpenBookQA. Over/underconfidence is average confidence minus accuracy: negative means the model undersells itself. Set sizes come from 5-fold conformal prediction at 90% coverage, on the same questions as the other columns.

Rank	Model	Questions	Accuracy ↑	ECE ↓	Brier score ↓	Over/underconfidence	90% set size ↓
1	Jev 1.13TypeSafe	817	91.8%	0.017	0.052	+0.000	0.95
2	d1-ultraLiquid AI	817	91.7%	0.058	0.054	−0.046	0.96
3	Jev 1.14TypeSafe	817	92.0%	0.021	0.058	−0.017	0.96
4	Perplexity Decider v1.2Perplexity	817	86.0%	0.038	0.077	+0.036	1.13
5	Drex MaxNace AI	817	86.0%	0.051	0.083	+0.050	1.17
6	GPT-6 Luna DecisionsOpenAI	817	85.1%	0.035	0.093	+0.021	1.29
7	ClefCloudflare	817	83.6%	0.035	0.101	+0.023	1.35
8	Mercury DecideInception	817	82.4%	0.087	0.101	+0.087	1.30
9	JEV-27BAutoTrust AI	817	81.4%	0.056	0.106	−0.038	1.41
10	JPT-4Bkirp	817	79.7%	0.044	0.112	−0.016	1.80
11	Drex v2.0Nace AI	817	80.8%	0.072	0.117	−0.062	1.55
12	Solar Decide FlashUpstage	817	78.9%	0.035	0.126	+0.010	1.78
13	Clef OmniCloudflare	817	76.1%	0.064	0.128	+0.044	1.87
14	Clef FlashCloudflare	817	75.3%	0.056	0.129	+0.011	1.98
15	Solar DecideUpstage	817	83.4%	0.097	0.136	+0.095	1.84
16	JEV-9BAutoTrust AI	817	70.5%	0.028	0.161	−0.021	2.41
17	Perplexity Decider Flash v1.2Perplexity	817	70.1%	0.108	0.162	+0.106	2.13
18	Drex v1.5Nace AI	817	68.5%	0.130	0.165	+0.130	2.46
19	Kev-4BJared Palmer	817	66.0%	0.073	0.174	−0.062	2.95
20	Qwen3.5-9B, untrainedAlibaba Qwen	817	67.3%	0.110	0.181	−0.104	2.68
21	Open-Jev-2BZefan Cai	817	46.4%	0.046	0.201	−0.013	4.40
22	Strands Decider 2BAmazon Strands Labs	817	44.9%	0.132	0.215	+0.126	4.20
23	LayaConvai Innovations	817	15.5%	0.333	0.262	+0.333	4.64

Ranked by Brier score on TruthfulQA MC1. Over/underconfidence is average confidence minus accuracy: negative means the model undersells itself. Set sizes come from 5-fold conformal prediction at 90% coverage, on the same questions as the other columns.

Rank	Model	Questions	Accuracy ↑	ECE ↓	Brier score ↓	Over/underconfidence	90% set size ↓
1	Jev 1.13TypeSafe	198	73.7%	0.053	0.147	−0.041	1.73
2	Jev 1.14TypeSafe	198	72.7%	0.079	0.159	−0.079	1.71
3	Perplexity Decider v1.2Perplexity	198	55.6%	0.136	0.183	+0.058	2.30
4	d1-ultraLiquid AI	198	58.1%	0.061	0.194	+0.004	2.32
5	Drex v2.0Nace AI	198	47.0%	0.105	0.195	+0.086	2.58
6	Solar DecideUpstage	198	55.1%	0.170	0.197	+0.170	2.57
7	GPT-6 Luna DecisionsOpenAI	198	49.5%	0.117	0.205	+0.091	2.83
8	JEV-27BAutoTrust AI	197	42.1%	0.134	0.213	+0.115	2.93
9	Open-Jev-2BZefan Cai	198	28.8%	0.066	0.213	+0.066	3.44
10	Kev-4BJared Palmer	198	35.4%	0.096	0.214	+0.092	3.21
11	LayaConvai Innovations	198	22.2%	0.198	0.216	+0.198	3.67
12	Clef FlashCloudflare	198	44.4%	0.080	0.217	+0.062	3.01
13	Drex MaxNace AI	198	58.1%	0.172	0.220	+0.122	2.42
14	JEV-9BAutoTrust AI	198	36.4%	0.157	0.222	+0.123	3.32
15	Qwen3.5-9B, untrainedAlibaba Qwen	198	38.9%	0.110	0.223	+0.067	3.39
16	ClefCloudflare	198	50.0%	0.125	0.227	+0.057	3.03
17	Clef OmniCloudflare	198	49.0%	0.058	0.228	+0.040	3.13
18	Solar Decide FlashUpstage	198	36.4%	0.163	0.230	+0.161	3.26
19	JPT-4Bkirp	198	37.4%	0.211	0.231	+0.167	3.21
20	Strands Decider 2BAmazon Strands Labs	198	31.3%	0.141	0.235	+0.121	3.46
21	Perplexity Decider Flash v1.2Perplexity	198	46.0%	0.178	0.245	+0.141	2.92
22	Drex v1.5Nace AI	198	40.4%	0.234	0.265	+0.234	2.84
23	Mercury DecideInception	198	40.4%	0.288	0.281	+0.288	3.03

Ranked by Brier score on GPQA Diamond. Over/underconfidence is average confidence minus accuracy: negative means the model undersells itself. Set sizes come from 5-fold conformal prediction at 90% coverage, on the same questions as the other columns.

Rank	Model	Questions	Accuracy ↑	ECE ↓	Brier score ↓	Over/underconfidence	90% set size ↓
1	Clef FlashCloudflare	1,000	95.8%	0.083	0.042	−0.083	0.92
2	ClefCloudflare	1,000	94.7%	0.061	0.044	−0.059	0.93
3	Drex MaxNace AI	1,000	89.2%	0.029	0.065	+0.019	1.01
4	Perplexity Decider v1.2Perplexity	1,000	89.0%	0.039	0.071	−0.037	1.03
5	Perplexity Decider Flash v1.2Perplexity	1,000	86.8%	0.058	0.083	−0.051	1.10
6	Mercury DecideInception	1,000	87.8%	0.077	0.088	+0.072	1.07
7	d1-ultraLiquid AI	1,000	83.5%	0.051	0.100	−0.046	1.23
8	Drex v1.5Nace AI	1,000	83.5%	0.075	0.106	+0.070	1.27
9	Clef OmniCloudflare	1,000	94.5%	0.249	0.114	−0.248	0.94
10	Drex v2.0Nace AI	1,000	79.2%	0.049	0.121	+0.021	1.75
11	Jev 1.14TypeSafe	1,000	78.8%	0.076	0.124	+0.076	1.70
12	Jev 1.13TypeSafe	1,000	79.5%	0.085	0.129	+0.081	1.76
13	GPT-6 Luna DecisionsOpenAI	986	72.8%	0.071	0.155	+0.041	14.23
14	JEV-9BAutoTrust AI	1,000	73.3%	0.095	0.175	+0.070	3.17

Ranked by Brier score on BANKING77. Over/underconfidence is average confidence minus accuracy: negative means the model undersells itself. Set sizes come from 5-fold conformal prediction at 90% coverage, on the same questions as the other columns. GPT-6 Luna Decisions would not answer 14 questions here.

Reliability diagram

Among questions where a model stated a given confidence, how often it was right. Bars show the share answered correctly, the dark tick on each bar is the confidence the model stated, and the dashed diagonal is perfect calibration.

Model Dataset

What we found

Perplexity’s Decider is the one to beat. It tops this board and the decision benchmarks. Jev is near the top here, on exam questions, but lands in the middle of the pack on decision questions.

Jev knows when it is right. Its open copies are less sure of that. Jev’s confidence matches its accuracy on every standard exam set, from easy ones like ARC to the very hard GPQA. JEV-9B and JEV-27B were trained to copy Jev’s answers, and they undersell themselves on easy questions and oversell on hard ones. Our guess is that they learned how confident Jev is, and that much confidence is too much for a smaller model that knows less.

Training on Jev’s answers makes a model bolder everywhere. The untrained Qwen model is too timid almost across the board. After training, the middle of the range looks right, easy sets stay a little timid, and the hardest set tips into overconfidence. Kev, JPT, Strands Decider and Open-Jev were trained on their own data and lean the same way. Laya, a small encoder, is overconfident almost everywhere.

Some blind spots are shared, Jev included. When the right answer is buried among many options, as in Under Pressure, most models are surer than they should be. d1-ultra is the steadiest model we tested: no dataset pushes its confidence far from its accuracy.

Solar Decide sounds the most certain without earning it, with Mercury Decide close behind.

What it means in practice. If you act on every answer above a fixed confidence, you will let through more mistakes on hard questions than on easy ones, and how many depends on the model. Check calibration on your own task, or use conformal prediction, which keeps its guarantee even when the probabilities are off.

Models

Fifteen models are hosted: TypeSafe’s Jev 1.13 and 1.14 and thirteen decision models from Perplexity, Nace AI, Liquid AI, OpenAI, Cloudflare, Inception and Upstage. We asked them through OpenRouter in October 2026, each pinned to its maker’s own servers. Each name links to the model’s OpenRouter page.

Eight models have open weights. JEV-9B and JEV-27B are AutoTrust’s open copies of Jev, trained on Jev 1.13’s answers. Qwen3.5-9B, untrained, is the same base model read out the same way without that training. The rest are Jev-style models from other teams, each run through the server its authors ship. Each name links to its model card. Open models were asked with lettered options, the format Jev’s documentation uses, so a model tuned to another format may score a little lower here than in its authors’ own tests.

Same questions for everyone

MMLU and MMLU-Pro are cut to 2,000 questions each and BANKING77 to 1,000, sampled in proportion to subject. Every other dataset is used in full. A few models could not answer everything:

Solar Decide and Solar Decide Flash take at most 26 options, so they skipped Under Pressure at 64 options and BANKING77.
GPT-6 Luna Decisions refused 5 questions in the summary datasets.
JEV-27B ran out of GPU memory on MMLU-Pro, BANKING77 and Under Pressure at 64 options.

None of those datasets are in the summary. Jev and GPT-6 Luna round their probabilities to two decimals, which barely moves accuracy, ECE or the Brier score.