d20 Bench · Study 1

Which local AI answers rules questions correctly in 5 seconds?

Baseline study of English SRD 5.2.1 rules questions, by Erol Can Akbaba.


I gave 25 AI setups 120 questions about a game's rulebook, with and without the passage behind each question. The best setup went from 16/120 answers fully correct to 113/120 with the passage, almost three times faster. Two setups scored almost the same, 99.08 and 98.86 out of 100, yet one added a wrong rule in 2 answers and the other in 24. Three of the 25 gave a typical answer within 5 seconds and got more than half fully correct, and all three were Gemma 4, a model family from Google.

A rulebook works like a policy manual: clear rules, plenty of tricky cases, and answers you can check. If you are choosing an AI for that kind of work, this report shows which numbers reveal the risk, what the right documents add, and how much quality a faster answer costs. Look for four things.

  • Fully correct counts before average scores. A fully correct answer meets every required claim and contains no material error, a wrong or invented rule the answer adds, which many people call a hallucination. The claim score sums the points earned for required claims, out of 100, with no deduction for errors, so a perfect score can still hide a wrong rule.
  • The median beside the p95. The median is the middle completion time. The p95 is the slower end of the distribution, where 1 request in 20 is slower. A low median cannot excuse a slow tail.
  • The gap with and without the passage. Oracle passages are the rules passages the task authors used to write that question. The gap shows what feeding an AI the right text is worth.
  • The price of thinking. With thinking on, a model works through the problem before it answers; with thinking off, it answers straight away. Thinking spends tokens, the word fragments a model reads and writes, and each one adds to the wait.

The product question sets the speed target

This study looks at one job in interactive narrative systems such as game master agents and tabletop role-playing assistants: applying precise rules knowledge live, inside a conversation. I use the openly licensed rules of Dungeons & Dragons (SRD 5.2.1) as the test domain; the name d20 refers to the twenty-sided die the game rolls.

A player describes an action. The game master (GM) agent takes 30 seconds just to say “roll Persuasion,” and a live session has many such steps. My target is about 5 seconds per step. Usability research puts the limit for keeping a user's attention on a dialogue at about 10 seconds [1]. I set a 30-second hard cutoff because, in my judgment, a model that takes that long on a simple rules question even 1% of the time invites players to switch off.

What was tested

I tested 15 models in 25 configurations, where a configuration is one model with one thinking setting and its run settings. Each answered the same 120 cases with and without oracle passages, for 6,000 answer records.

All models ran locally, on my own machine, on a single NVIDIA RTX 3090 graphics card (GPU) in October 2026. The card has 24 GB of VRAM, its own memory, which a model has to fit into. I ran each model in LM Studio with its defaults and each provider's recommended settings. VRAM usage was not tracked.

All results below cover the full 120 cases per configuration. The repository exposes 18 human-reviewed public cases and their 900 answer records. The other 102 cases remain private and contribute only aggregates. A few answers couldn't be graded as clearly right or wrong; those are left out, so some counts are out of fewer than 120.

Comparisons are grouped by parameters, the learned numbers that measure a model's size, into three size classes by total count: small, roughly 2.7–5.1B (billion); medium, 8.0–9.7B; large, 24–31B. Dense models use all their parameters for each token. Mixture-of-experts models use only the active experts alongside shared weights: Gemma 4 26B-A4B has about 25.8B total and 3.8B active parameters. The active count is what matters for speed [2, 3], provided the model fits in VRAM. Runtime and configuration differences remain confounds.

Completion time covers every recorded attempt and outcome, including retries, failures and capped outputs, and leaves out model loading and grading.

Oracle passages improved every configuration

All 25 configurations had higher mean claim scores, lower median completion times and fewer output tokens with oracle passages. Gemma 4 26B-A4B with thinking on went from 16/120 fully correct at a 17.42-second median to 113/120 at 6.08 seconds. The lowest count with passages, 30/120, still beats the best count without them. This supports context injection, handing the model the text it needs, and fits the retrieval-augmented generation literature [4].

Oracle passages improved quality and reduced time and output

Each configuration improved on all three measures: claim score, completion time and output tokens. Every marker covers all 120 cases; focus a marker to see the exact values and counts.

off/on: thinking off/on · on (low): on at low effort · low/native: cannot turn thinking off; low effort/thinking on

Static data table and text alternative
Without → with oracle passages
ConfigurationSizeMean claim /100Median sMean tokens
LFM2.5 2.6B · nativesmall28.40 → 74.038.31 → 4.992392.0 → 1734.0
Granite 4.2 3B · offsmall33.86 → 84.5611.63 → 6.872866.7 → 1832.6
Granite 4.2 3B · onsmall38.88 → 90.5826.37 → 8.515508.4 → 2508.2
Qwen3.5 4B · offsmall36.69 → 87.376.11 → 3.572916.6 → 986.5
Qwen3.5 4B · onsmall40.59 → 95.3883.78 → 32.7011715.9 → 5193.2
K2 Horizon 3.7B · lowsmall22.10 → 70.705.07 → 3.276853.4 → 2106.6
Gemma 4 E2B · offsmall29.57 → 79.842.09 → 1.76464.8 → 380.0
Gemma 4 E2B · onsmall28.83 → 86.844.52 → 3.86960.3 → 837.2
Gemma 4 E4B · offmedium35.90 → 87.125.01 → 3.99517.6 → 383.7
Gemma 4 E4B · onmedium39.27 → 93.669.31 → 7.021058.3 → 801.7
LFM2.5-8B-A1B · nativemedium27.63 → 71.419.91 → 4.074046.9 → 1729.9
Granite 4.2 8B · offmedium41.09 → 88.8322.38 → 9.413264.1 → 1685.1
Granite 4.2 8B · onmedium44.58 → 93.9737.74 → 18.474734.8 → 2971.9
Qwen3.5 9B · offmedium51.88 → 89.247.62 → 4.861701.4 → 749.7
Qwen3.5 9B · onmedium45.30 → 96.6383.02 → 26.669077.4 → 3577.9
LFM2.5-24B-A2B · nativelarge41.93 → 95.0823.84 → 10.765959.3 → 2868.5
Gemma 4 26B-A4B · offlarge48.72 → 94.593.14 → 2.13389.2 → 255.3
Gemma 4 26B-A4B · onlarge51.24 → 99.0817.42 → 6.082754.9 → 892.6
Qwen3.8 27B · offlarge53.67 → 95.1614.21 → 7.71836.3 → 456.8
Qwen3.8 27B · on (low)large53.88 → 98.4227.74 → 11.281735.0 → 709.6
Granite 4.2 30B · offlarge44.07 → 95.2440.14 → 27.071406.6 → 825.6
Granite 4.2 30B · onlarge45.55 → 95.9271.84 → 52.042336.3 → 1493.3
Muse Glimmer · lowlarge60.07 → 98.8627.29 → 16.39912.8 → 581.0
GLM-4.7 Flash · offlarge35.87 → 70.814.11 → 3.11473.3 → 356.2
GLM-4.7 Flash · onlarge45.73 → 90.2228.83 → 13.773885.2 → 2061.9

The public case d20-en-pilot-t01-003 asks whether a willing character can fail a saving throw without rolling. Gemma 4 E2B with thinking off insisted on a roll without oracle passages and allowed voluntary failure with them. Its score rose from 0 to 100, while time fell from 1.76 to 0.70 seconds and output from 356 to 130 tokens.

On d20-en-pilot-t11-006, Qwen3.5 4B with thinking off also rose from 0 to 100, dropping fabricated restrictions for Knock and Arcane Lock.

Under the 5-second bar, Gemma 4 leads

With oracle passages, 10/25 configurations had a median at or below 5 seconds, but only three configurations met the 5-second median bar and got more than half the answers fully correct. All three were Gemma 4. Rows are sorted by fully correct answers, then by lower median time. Here, off/on mean thinking off/on; native marks models that cannot turn thinking off, and low marks reduced thinking effort.

Size classConfigurationFully correct / assessedMedian sp95 sMean output tokens
LargeGemma 4 26B-A4B on113/1206.0815.68892.6
LargeQwen3.8 27B on (low)102/11911.2822.27709.6
MediumQwen3.5 9B on99/12026.6677.803,577.9
LargeGemma 4 26B-A4B off96/1202.134.56255.3
LargeMuse Glimmer low96/12016.3927.46581.0
MediumGemma 4 E4B on91/1207.0211.06801.7
LargeQwen3.8 27B off90/1207.7114.14456.8
SmallQwen3.5 4B on90/12032.7094.705,193.2
LargeGLM-4.7 Flash on82/12013.7744.252,061.9
MediumGemma 4 E4B off78/1193.998.77383.7
SmallGemma 4 E2B on76/1203.866.57837.2
SmallGranite 4.2 3B on76/1198.5145.172,508.2
LargeLFM2.5-24B-A2B native73/11910.7629.952,868.5
LargeGranite 4.2 30B on69/11952.0497.291,493.3
MediumGranite 4.2 8B off63/1209.4146.691,685.1
MediumGranite 4.2 8B on63/11718.4797.162,971.9
LargeGranite 4.2 30B off62/11527.0772.33825.6
SmallGemma 4 E2B off58/1201.764.36380.0
SmallQwen3.5 4B off51/1203.5716.17986.5
MediumQwen3.5 9B off49/1164.8619.40749.7
SmallGranite 4.2 3B off49/1196.8737.391,832.6
SmallK2 Horizon 3.7B low33/1193.2789.812,106.6
MediumLFM2.5-8B-A1B native32/1204.0714.341,729.9
SmallLFM2.5 2.6B native32/1204.9921.391,734.0
LargeGLM-4.7 Flash off30/1203.116.07356.2

Bold rows: median at or below 5 seconds.

Only three configurations clear the median bar with a majority fully correct

With oracle passages: fully correct counts versus median completion time. Lines extend to p95, the slower end of the distribution. The preceding report table also gives p95 and mean output tokens.

off/on: thinking off/on · on (low): on at low effort · low/native: cannot turn thinking off; low effort/thinking on

Static values: see the full oracle table immediately above. All 120 requests have timing and token measurements for each configuration. The CSV table also includes counts above 30 seconds.

Among the bold rows, Gemma 4 26B-A4B off had the most fully correct answers and the lowest mean output tokens, and its 4.56-second p95 also sat below the target. That combination is why I picked Gemma 4.

In the public answers, Gemma 4 26B-A4B generated about 130 tokens per second, compared with about 24–33 tokens per second for the dense roughly 30B models without speculative decoding on the same GPU. I see its smaller active workload as the main reason Gemma clears my 5-second bar with thinking off, alongside its short answers.

K2's 3.27-second median accompanies an 89.81-second p95, and it exceeded the 30-second cutoff in 8/120 requests.

Thinking can buy quality, or consume the session

For Gemma 4 26B-A4B with oracle passages, thinking on increased fully correct answers from 96/120 to 113/120, the highest count in the study. Median time rose from 2.13 to 6.08 seconds, just past the target and p95 from 4.56 to 15.68 seconds.

Thinking adds waiting time; the quality return varies

Each thinking-off configuration beside its thinking-on partner, with oracle passages. Every marker covers all 120 cases.

off/on: thinking off/on · on (low): on at low effort · low/native: cannot turn thinking off; low effort/thinking on

Static data table and text alternative
Thinking off → on
ModelSizeFully correctMedian sp95 sMean tokens
Granite 4.2 3Bsmall49/119 → 76/1196.87 → 8.5137.39 → 45.171832.6 → 2508.2
Qwen3.5 4Bsmall51/120 → 90/1203.57 → 32.7016.17 → 94.70986.5 → 5193.2
Gemma 4 E2Bsmall58/120 → 76/1201.76 → 3.864.36 → 6.57380.0 → 837.2
Gemma 4 E4Bmedium78/119 → 91/1203.99 → 7.028.77 → 11.06383.7 → 801.7
Granite 4.2 8Bmedium63/120 → 63/1179.41 → 18.4746.69 → 97.161685.1 → 2971.9
Qwen3.5 9Bmedium49/116 → 99/1204.86 → 26.6619.40 → 77.80749.7 → 3577.9
Gemma 4 26B-A4Blarge96/120 → 113/1202.13 → 6.084.56 → 15.68255.3 → 892.6
Qwen3.8 27B · off → on (low)large90/120 → 102/1197.71 → 11.2814.14 → 22.27456.8 → 709.6
Granite 4.2 30Blarge62/115 → 69/11927.07 → 52.0472.33 → 97.29825.6 → 1493.3
GLM-4.7 Flashlarge30/120 → 82/1203.11 → 13.776.07 → 44.25356.2 → 2061.9

With oracle passages, Qwen3.5 4B went from 51/120 to 90/120, but its median rose from 3.57 to 32.70 seconds, beyond the 30-second cutoff.

The public case d20-en-pilot-t03-009, about damage after stabilization and run on Granite 4.2 30B without oracle passages, makes the cost concrete. Thinking on reduced the score from 20 to 10, increased time from 94 to 795 seconds, and output from 2,256 to 11,321 tokens. The longer answer invented rules; without grounding, thinking turned into excessive rumination, a pattern also reported for other reasoning models [5].

On the spell-slot case d20-en-pilot-t10-008, Qwen3.5 4B fell from 100 to 93 with thinking on, while time rose from 15 to 123 seconds.

For knowledge-heavy questions, the right evidence reliably improves accuracy, while extra thinking brings uneven gains [6, 7]. The advantage of oracle passages in this study fits that pattern.

A high average can conceal wrong rules

With oracle passages, Muse Glimmer with low thinking effort averaged 98.86 claim points, close to Gemma 4 26B-A4B thinking on at 99.08. Their fully correct counts were 96/120 and 113/120. Muse had material errors in 24/120 answers; Gemma had them in 2/120.

Similar claim means can accompany different error counts

With oracle passages: mean required-claim score, fully correct counts and material-error counts (answers that add a wrong or invented rule). Claim scores sum the points awarded across claims and carry no error deductions. The dashed line in the fully correct column marks half of the 120 cases. Focus a marker to see each count's denominator.

off/on: thinking off/on · on (low): on at low effort · low/native: cannot turn thinking off; low effort/thinking on

Static data table and text alternative
With oracle passages
ConfigurationSizeMean claim /100Fully correctMaterial errors
LFM2.5 2.6B · nativesmall74.0332/12087/120
Granite 4.2 3B · offsmall84.5649/11969/119
Granite 4.2 3B · onsmall90.5876/11943/119
Qwen3.5 4B · offsmall87.3751/12068/120
Qwen3.5 4B · onsmall95.3890/12028/120
K2 Horizon 3.7B · lowsmall70.7033/11981/118
Gemma 4 E2B · offsmall79.8458/12057/120
Gemma 4 E2B · onsmall86.8476/12033/120
Gemma 4 E4B · offmedium87.1278/11936/119
Gemma 4 E4B · onmedium93.6691/12023/120
LFM2.5-8B-A1B · nativemedium71.4132/12086/120
Granite 4.2 8B · offmedium88.8363/12057/120
Granite 4.2 8B · onmedium93.9763/11754/117
Qwen3.5 9B · offmedium89.2449/11663/116
Qwen3.5 9B · onmedium96.6399/12019/120
LFM2.5-24B-A2B · nativelarge95.0873/11945/119
Gemma 4 26B-A4B · offlarge94.5996/12016/120
Gemma 4 26B-A4B · onlarge99.08113/1202/120
Qwen3.8 27B · offlarge95.1690/12028/120
Qwen3.8 27B · on (low)large98.42102/11916/119
Granite 4.2 30B · offlarge95.2462/11552/115
Granite 4.2 30B · onlarge95.9269/11949/119
Muse Glimmer · lowlarge98.8696/12024/120
GLM-4.7 Flash · offlarge70.8130/12089/120
GLM-4.7 Flash · onlarge90.2282/12026/120

In d20-en-pilot-t05-010, LFM2.5 2.6B with oracle passages correctly separated negotiating from reading body language and earned 100 claim points. It also invented an Intelligence-versus-Willpower check for Influence, so the answer has a material error and isn't fully correct.

Inspect the evidence

The study index links the offline comparison and public case explorer. The aggregate results contain every condition; public answer records contain the five examples above. Methods, Limitations, the data dictionary and provenance give measurement definitions and the public/private boundary. The source rules retain their SRD attribution.

Next: measure the gap from retrieval to oracle

A live system needs retrieval, software that finds the passages for the model. The next study, in progress, tests BM25, a standard keyword search that ranks passages by how well their words match the question, with Gemma 4 at several sizes.

See what a study on your own AI would give you.

References

  1. Nielsen, J. Response Times: The 3 Important Limits. Excerpt from Usability Engineering, chapter 5. Academic Press, 1993.
  2. Fedus, W., Zoph, B. & Shazeer, N. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. JMLR, 2022.
  3. Jiang, A. Q. et al. Mixtral of Experts. 2024.
  4. Lewis, P. et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS, 2020.
  5. Chen, X. et al. Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs. 2024.
  6. Trivedi, H., Balasubramanian, N., Khot, T. & Sabharwal, A. Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions. ACL, 2023.
  7. Zhao, J. X., Hooi, B. & Ng, S.-K. Test-Time Scaling in Reasoning Models Is Not Effective for Knowledge-Intensive Tasks Yet. COLM, 2026.