Files
esh-pfi-infrastructure/scripts
vh 04950c2881 feat(training-probes): re-measure the R49 name pool under the Qwen3 tokenizer
brokkr-smithy flagged that R49 F02's name-pool token splits were measured with the
Qwen3.5-2B tokenizer, so the dense-Qwen3 carrier ruling invalidates them. Measured
rather than left on their critical path; handed over as input to their re-check,
since the dictionary and the adjudication are theirs.

The multi-token property strengthens on the chosen carrier: pool multi-token
88.0% -> 90.3%, mean tokens 2.33 -> 2.46. A smaller vocabulary fragments more, so
Qwen3's 151,936 splits names into more pieces than Qwen3.5's 248,320. The operator's
requirement that names be multi-token, so the drafter reconstructs them from the
prefix instead of recalling one embedding, is better served after the ruling.

Positive control: the Qwen3.5 column reproduces F02's published figure on the same
pool and tokenizer (F02 89% / mean 2.35; here 88.0% / 2.33), so the instrument
recovers a known-true value before being asked about an unknown one. The pool is
deduped across locales, which reconciles male_given and female_given exactly
against the dictionary's own totals block.
2026-09-09 22:59:15 -07:00
..