Configuration comparison

Each row covers all 120 cases for one configuration and passage condition; 18 of those cases are public. Rows are grouped by total-parameter size class, then ordered by total parameters. A few answers couldn't be graded as clearly right or wrong; those are left out, so some counts are out of fewer than 120. Times are in seconds. They include retries and leave out model loading and grading. An answer is fully correct when it satisfies every required claim and contains no material error. The p95 is the slower end of the distribution, where 1 request in 20 is slower. Modes off and on mean thinking off and on; on (low) means thinking on at low effort. Models labelled low or native cannot turn thinking off: low ran at low effort, and native ran with thinking on.

Size class	Model	Mode	Passages	Mean claim score	Fully correct	Material errors	Mean s	Median s	p25 s	p75 s	p95 s	>30 s
small	LFM2.5 2.6B	native	Without oracle passages	28.4	1/119	118/119	9.82	8.31	5.72	11.72	23.25	1/120
small	LFM2.5 2.6B	native	With oracle passages	74.0	32/120	87/120	7.19	4.99	3.59	7.96	21.39	3/120
small	Granite 4.2 3B	off	Without oracle passages	33.9	2/120	118/120	17.54	11.63	5.96	20.27	52.12	22/120
small	Granite 4.2 3B	off	With oracle passages	84.6	49/119	69/119	11.30	6.87	2.48	12.98	37.39	8/120
small	Granite 4.2 3B	on	Without oracle passages	38.9	9/120	111/120	34.58	26.37	15.91	44.14	86.69	54/120
small	Granite 4.2 3B	on	With oracle passages	90.6	76/119	43/119	15.56	8.51	5.16	18.35	45.17	12/120
small	Qwen3.5 4B	off	Without oracle passages	36.7	0/120	120/120	22.13	6.11	4.26	23.15	82.86	30/120
small	Qwen3.5 4B	off	With oracle passages	87.4	51/120	68/120	7.44	3.57	2.56	5.13	16.17	6/120
small	Qwen3.5 4B	on	Without oracle passages	40.6	2/119	116/119	90.43	83.78	65.06	105.08	163.27	117/120
small	Qwen3.5 4B	on	With oracle passages	95.4	90/120	28/120	38.76	32.70	17.91	47.11	94.70	67/120
small	K2 Horizon 3.7B	low	Without oracle passages	22.1	1/120	115/120	106.87	5.07	2.95	12.67	525.51	27/120
small	K2 Horizon 3.7B	low	With oracle passages	70.7	33/119	81/118	31.19	3.27	2.15	4.70	89.81	8/120
small	Gemma 4 E2B	off	Without oracle passages	29.6	8/120	109/120	2.31	2.09	1.65	2.76	3.95	0/120
small	Gemma 4 E2B	off	With oracle passages	79.8	58/120	57/120	1.92	1.76	1.14	2.37	4.36	0/120
small	Gemma 4 E2B	on	Without oracle passages	28.8	10/120	107/120	4.71	4.52	3.93	5.37	6.93	0/120
small	Gemma 4 E2B	on	With oracle passages	86.8	76/120	33/120	4.14	3.86	3.11	4.67	6.57	0/120
medium	Gemma 4 E4B	off	Without oracle passages	35.9	7/120	111/120	5.60	5.01	3.93	6.70	10.61	0/120
medium	Gemma 4 E4B	off	With oracle passages	87.1	78/119	36/119	4.34	3.99	2.96	5.11	8.77	0/120
medium	Gemma 4 E4B	on	Without oracle passages	39.3	11/120	105/120	9.41	9.31	7.98	10.96	13.03	0/120
medium	Gemma 4 E4B	on	With oracle passages	93.7	91/120	23/120	7.35	7.02	6.00	8.45	11.06	0/120
medium	LFM2.5-8B-A1B	native	Without oracle passages	27.6	0/120	119/120	13.34	9.91	6.38	17.54	29.35	5/120
medium	LFM2.5-8B-A1B	native	With oracle passages	71.4	32/120	86/120	5.72	4.07	2.98	5.65	14.34	1/120
medium	Granite 4.2 8B	off	Without oracle passages	41.1	2/120	118/120	33.19	22.38	13.04	37.70	86.88	46/120
medium	Granite 4.2 8B	off	With oracle passages	88.8	63/120	57/120	16.95	9.41	6.15	19.54	46.69	18/120
medium	Granite 4.2 8B	on	Without oracle passages	44.6	2/119	117/119	48.04	37.74	25.80	64.16	96.42	81/120
medium	Granite 4.2 8B	on	With oracle passages	94.0	63/117	54/117	30.40	18.47	13.29	30.70	97.16	31/120
medium	Qwen3.5 9B	off	Without oracle passages	51.9	5/120	115/120	16.73	7.62	5.22	11.78	76.45	17/120
medium	Qwen3.5 9B	off	With oracle passages	89.2	49/116	63/116	7.47	4.86	3.74	6.77	19.40	4/120
medium	Qwen3.5 9B	on	Without oracle passages	45.3	8/119	110/119	89.90	83.02	67.03	99.00	158.43	116/120
medium	Qwen3.5 9B	on	With oracle passages	96.6	99/120	19/120	35.11	26.66	19.14	40.52	77.80	53/120
large	LFM2.5-24B-A2B	native	Without oracle passages	41.9	7/120	113/120	28.16	23.84	15.85	34.35	65.95	40/120
large	LFM2.5-24B-A2B	native	With oracle passages	95.1	73/119	45/119	13.50	10.76	7.90	14.98	29.95	6/120
large	Gemma 4 26B-A4B	off	Without oracle passages	48.7	6/118	111/118	3.30	3.14	2.37	3.83	5.98	0/120
large	Gemma 4 26B-A4B	off	With oracle passages	94.6	96/120	16/120	2.40	2.13	1.65	2.73	4.56	0/120
large	Gemma 4 26B-A4B	on	Without oracle passages	51.2	16/120	102/120	22.20	17.42	9.27	30.60	63.55	32/120
large	Gemma 4 26B-A4B	on	With oracle passages	99.1	113/120	2/120	7.25	6.08	4.82	7.88	15.68	1/120
large	Qwen3.8 27B	off	Without oracle passages	53.7	6/119	112/119	16.35	14.21	11.34	18.45	28.92	6/120
large	Qwen3.8 27B	off	With oracle passages	95.2	90/120	28/120	8.59	7.71	5.68	10.35	14.14	1/120
large	Qwen3.8 27B	on (low)	Without oracle passages	53.9	12/119	106/119	31.43	27.74	20.85	37.70	68.27	52/120
large	Qwen3.8 27B	on (low)	With oracle passages	98.4	102/119	16/119	12.72	11.28	9.82	14.36	22.27	2/120
large	Granite 4.2 30B	off	Without oracle passages	44.1	0/120	120/120	60.31	40.14	26.87	57.84	116.49	80/120
large	Granite 4.2 30B	off	With oracle passages	95.2	62/115	52/115	33.94	27.07	20.24	39.10	72.33	51/120
large	Granite 4.2 30B	on	Without oracle passages	45.6	1/120	119/120	102.98	71.84	55.85	97.84	239.73	119/120
large	Granite 4.2 30B	on	With oracle passages	95.9	69/119	49/119	64.74	52.04	43.51	62.61	97.29	118/120
large	Muse Glimmer	low	Without oracle passages	60.1	11/120	108/120	27.55	27.29	20.60	31.95	43.57	42/120
large	Muse Glimmer	low	With oracle passages	98.9	96/120	24/120	17.81	16.39	13.84	20.65	27.46	4/120
large	GLM-4.7 Flash	off	Without oracle passages	35.9	0/120	120/120	4.35	4.11	3.29	5.09	7.72	0/120
large	GLM-4.7 Flash	off	With oracle passages	70.8	30/120	89/120	3.40	3.11	2.41	4.10	6.07	0/120
large	GLM-4.7 Flash	on	Without oracle passages	45.7	9/120	111/120	34.38	28.83	20.78	40.23	73.58	56/120
large	GLM-4.7 Flash	on	With oracle passages	90.2	82/120	26/120	18.12	13.77	11.03	19.18	44.25	14/120