Exploratory English SRD 5.2.1 studies
d20 Bench
Erol Can Akbaba · 2026
Which local model can keep a live interaction moving? These studies measure answer quality, tokens and completion time for local language models answering D&D rules questions. That stands in for a broader need: interactive narrative systems have to apply precise rules knowledge live. I start with a baseline, then measure how much retrieving the right context helps.
Studies
Study 1 tested 15 local models in 25 combinations of model and thinking mode on 120 SRD 5.2.1 rules questions, with and without oracle passages, the passages used to write each question. With those passages, only three configurations got more than half the answers fully correct with a median completion time of 5 seconds or less, and all three were Gemma 4. The totals cover all 120 questions; the explorer shows the 18 public cases and their 900 answers. Read the report.
Study 2 is in progress: BM25 retrieval, a standard keyword-ranking method, with Gemma 4, on the same 120 questions. It has no published results yet.