← Back to the report

Limitations

Study 1 is an exploratory comparison of 25 configurations, with no preregistration.

Comparing models

• Models differ in architecture, total and active parameters, sampling, templates and output budgets. Qwen is version 3.5 at the smaller sizes and 3.8 at the larger size, so version and size change together.
• Parameter totals come from the official checkpoint files. Active counts follow each vendor's convention, and where a vendor publishes none, the count is unknown.

The questions

• Eighteen task versions got a human quality review. An AI reviewed the rest. All 120 tasks are development material, and the 18 public cases are a small subset.
• The 72 public Gemma 4 E4B answers (36 thinking off, 36 thinking on) have final scores and fully correct and material-error results but no per-claim awards. The expert annotations that exist for them stay separate.

Grading

• One model, GPT 6.1 Sol at medium effort, graded each new answer. I did not measure its accuracy, and a human reviewed only some cases.
• Each question was answered once per configuration and passage condition, so the data shows no run-to-run variation.
• A few answers couldn't be graded as clearly right or wrong; those are left out, so some counts are out of fewer than 120.
• Mean claim scores sum the points awarded per claim. An answer can earn every claim point and still add a wrong rule.

Timing and runtime

• Context and token ceilings, request deadlines and retries shape the completion times, which come from one run on one machine. Grading and model loading are separate costs outside the times.
• I did not record the exact llama.cpp CUDA 12 backend version I used with LM Studio 0.4.x. The GPU also drove my desktop display. LM Studio's echoed load settings show the configuration I requested; GPU placement and peak VRAM were not measured.
• Generation-speed medians pool both passage conditions from the 18 public cases. They exclude retries and answers shorter than 100 output tokens. Some configurations lack first-token timing, which leaves prompt and client overhead in the rate's denominator.

The data

• The private task set and raw private answers are excluded, so the full-120 aggregates cannot be recomputed from this public release.

VRAM estimates

I did not measure VRAM use. I calculated these estimates after the runs. Each one adds three things:

• the file size of a public Q4_K_M GGUF file, as a stand-in for the weights;
• the KV or recurrent state (the memory a model keeps for the text so far) at the configured context;
• 1 GiB of assumed runtime overhead.

GiB means bytes divided by 2³⁰. Each row combines the thinking modes. Model names link to the weight files I used as proxies, and the cache figures link to the architecture configs. The linked files are public stand-ins for the files I evaluated.

The estimate of 33.50 GiB for Granite 4.2 30B exceeds the card's 24 GB, so part of the model may have run off the GPU, which would slow its times. Every other row is at or below 21.19 GiB.

The calculation assumes one loaded model and one generation sequence, FP16 K/V, FP32 hybrid recurrent/convolution states, and full configured capacity for full-attention caches. Sliding-window caches use compact allocation with round_up(window + 512, 256) slots, assuming a 512-token microbatch. Gemma's cross-layer cache sharing and GLM's compressed MLA cache are included. These assumptions follow llama.cpp cache defaults, sliding-window allocation, hybrid state allocation and cache sharing and MLA cache storage.

Cache precision, allocation mode, per-slot state replication and runtime buffers were not measured, and LM Studio reported four parallel slots. The estimates leave out separate vision/audio projectors and extra speculative or rollback states. The Qwen3.8 MTP allocation is unknown.

Gemma 4 26B-A4B's estimate rises from 18.19 to 30.39 GiB if sliding-window layers get full-context caches. The same alternative gives 7.72 GiB for Gemma 4 E4B, 5.32 for Gemma 4 E2B and 19.86 for Muse Glimmer. Such assumptions can move an estimate across the GPU's 24 GB capacity.