d20 Bench · Study 1

Explore the data

Erol Can Akbaba · 2026

25 configurations answered 120 questions, with and without oracle passages. That makes 50 rows and 6,000 answer records. I am publishing 900 of the answers in full, covering 18 human-reviewed cases. The other 102 cases are private and count only in the totals.

Study report · Methods · Limitations · CSV table · Generation speed (18 public cases) · Aggregate data · Build hashes

An answer is fully correct when it satisfies every required claim and contains no material error.

Sampling, token budgets, model versions and architectures differ between configurations, so a difference can't be pinned on one cause. All 120 cases, the 18 public ones included, were used while I built the evaluation.

Loading comparison data…

Turn on JavaScript to explore the data. Without it you can still download: full comparison CSV, aggregate JSON, public answer JSON, public cases.

Quality and response time

Passages Size class

All 120 cases per row, grouped by total-parameter size class. Times are in seconds. They include retries and leave out model loading and grading.
Size class	Model / mode	Passages	Mean claim score	Fully correct	Material errors	Mean s	Median s	p25 s	p75 s	p95 s	Max s	>30 s	Mean tokens

Each row has 120 answers. A few answers couldn't be graded as clearly right or wrong; those are left out, so some counts are out of fewer than 120. Modes off and on mean thinking off and on; on (low) means thinking on at low effort. Models labelled low or native cannot turn thinking off: low ran at low effort, and native ran with thinking on. Token means describe the responses. Times include all recorded attempts. p25 and p75 are the 25th and 75th percentiles and p95 is the 95th, all by nearest rank. The p95 is the slower end of the distribution, where 1 request in 20 is slower. With an even count, the median averages the two middle values.

The slow end

A typical response time can hide a tail of slow ones. The report's figure shows the median and p95 time of each configuration with oracle passages, grouped by size class, on a logarithmic time axis.

See the median and p95 figure in the report, or download the full timing table.

Explore the public cases

Pick a case and a configuration to read the question, the reference answer and the grading rubric. Then compare the answers with and without oracle passages.

Cases, sizes and grades

The 18 public cases got a human quality review, including the repaired versions. An AI reviewed the other 102, which appear only in the totals.

Size classes use total parameters, including embeddings. Qwen is version 3.5 at the smaller sizes and 3.8 at the larger size. Gemma 4 E4B and LFM2.5 2.6B ran with a 32K-token context and an 8K-token output ceiling; the other configurations ran with 64K and 32K.

Public answers show the original expert annotations beside the final claim awards and the fully correct and material-error judgments, which include human decisions.

Public answer records · 18 public cases · Data dictionary · Comparison table