← Back to the report

Data dictionary

The public data files and what their fields mean. All files are UTF-8 JSON with schema_version: 1.

Files

../config.json is an input to the build. It holds the import exclusions, the ID mapping, the model metadata and the integrity checks on the retained data.

Identifiers

Configuration IDs have the form model/mode. Provenance lists every mapping from the old IDs. source_id is the original ID with its batch prefix. batch is the research batch it came from, and suite_id is an older name for batch. Case IDs never changed.

The mode in a configuration ID is off, on, native or low. native marks the LFM2.5 models, which cannot turn thinking off and ran with thinking on. low marks K2 Horizon and Muse Glimmer, which cannot turn thinking off and ran at a low reasoning-effort setting. Qwen3.8 27B on also ran at low reasoning effort.

To match an answer record to its configuration and case, use the three fields configuration, case_id and condition together. condition is no_source (without oracle passages) or supplied_passage (with oracle passages).

Scores and reviews

• score.earned_points is the sum of the final claim awards, out of 100.
• claim_awards holds the per-claim points, including human overrides. It is empty for the 72 public Gemma 4 E4B answers (36 thinking off, 36 thinking on). For these answers the data holds final scores and fully correct and material-error results but no per-claim awards, and historical_review labels this.
• For the 36 thinking-on answers among them, historical_expert_annotations keeps the available history from two experts.
• expert_annotation keeps the original expert's claim judgments and source findings, including earlier uncertainties. Where they are present, original_expert_score and final_review show that the final interpretation changed.
• score.complete_task_success and score.has_material_error hold the final fully correct and material-error judgments. Each is true, false or null (unknown), and null answers are left out of the counts.
• profile_assessed and material_error_assessed in the quality summary give the number of answers that could be graded. They are the denominators of the fully correct and material-error counts. A few answers couldn't be graded as clearly right or wrong; those are left out, so some counts are out of fewer than 120.

Timing and tokens

• request_timing.total_attempt_seconds adds up all recorded attempts for the answer.
• request_timing.has_final means the answer has a non-empty final section.
• operations and the token distributions describe the response. Raw thinking traces are not exported, and reasoning-token counts may be missing.
• Aggregate timing includes the mean, the median, nearest-rank quartiles and tails, coverage, retries and conditional-final statistics.

Request text and hashes

candidate_request holds prompt text only: either the rendered raw-completion prompt, or a native system_prompt and input. Text paths and private addresses are redacted where they appeared.

candidate_input_sha256 and answer_sha256 tie records back to my private workspace. They hash the original inputs before sanitizing.

sources in ../data/results.json holds hashes of the private reports, metrics and timing files. Each entry's selected_records counts the records published from that source. source_selected_records counts the records in the original source report. It exceeds selected_records only for the source that held the two configurations I left out (Provenance). Runtime metadata I could not verify is labeled as unverified. A build manifest holds hashes of the generated files. A release manifest holds the hash of the local release archive.

Parameter counts and size classes

I checked the counts in October 2026 against vendor model cards and the safetensors metadata of each official repository.

total_parameters is the integer safetensors.total that Hugging Face reports for the pinned official artifact. It includes embeddings and any modality components counted in that artifact. It is one consistent convention for totals and can differ from the rounded number in a model's name. The table below rounds to three decimals of a billion; each configuration stores the exact integer.

active_parameters is an active count the vendor explicitly published. Null means I found no separate active count, as for dense models. For Gemma 4 E2B and E4B, the effective counts are stored separately as effective_parameters (2.3B and 4.5B). Active counts follow each vendor's convention, so they may not be comparable with full-artifact totals.

parameter_source_url pins the official artifact metadata. parameter_card_url and parameter_card_sha256 identify the model card I checked. parameter_notes records differences in scope.

The observed clusters are 2.697–5.123B (small), 7.996–9.653B (medium) and 23.972–31.221B (large).

• Gemma 4 E2B (5.123B) and K2 Horizon 3.7B (5.058B) sit at the upper edge of small, nearer my 4B reference than 9B.
• Gemma 4 E4B counts as medium because its total is 7.996B.
• LFM2.5-8B-A1B stays medium despite its 1.5B active count.
• LFM2.5-24B-A2B is at the low end of large.

No total falls in the gaps between the clusters. The classes describe this selection of models. A future model with an in-between total needs an explicit assignment.

Model names and cards often differ from the artifact totals:

• Gemma 4 26B-A4B: the card gives 25.2B for the language model and about 0.55B for the vision encoder. The artifact total is 25.806B.
• Qwen: the cards label the language models 4B, 9B and 27B. The artifact totals are 4.660B, 9.653B and 27.781B.
• LFM2.5: the cards give 2.69B, 8.3B and 24.0B totals. The artifact totals are 2.697B, 8.468B and 23.972B.
• K2 Horizon 3.7B: the 3.7B core/activated count excludes part of its 5.058B total.
• Granite: the cards identify dense 3B, 8B and 30B models. The artifact totals are 3.660B, 8.792B and 29.277B.
• GLM: the nominal 30B-A3B corresponds to a 31.221B artifact.
• Muse Glimmer: the card rounds its dense model, including vision, to about 29.6B. The artifact metadata gives 29.777B.