Two harness defects that would each make a voice number uninterpretable.
build_beat_fixture.py — the fixture is now SAMPLED from the val split rather than
hand-written. The original BabyYarros fixture was five hand-written beats about a
stray dog and a kitten: wrong genre, so 'He licked her clean' came back as
explicit sex from a romantasy adapter, and n=5 had a noise floor of 0.800 that
manufactured a +0.45 result which collapsed to +0.08 at n=120. Sampling from val
makes it in-genre and held out by construction, spread across works so a naive
head(30) is not one novel. Refuses outright if the pairs carry any split but val,
because a fixture drawn from training data makes every downstream number a
memorisation measurement wearing a voice label.
gen_beats_chat_yarros.py --system-from — the SYS constant in this harness is
Yarros's. Driving a Bronte or Hemingway adapter with it measures the arm under a
system prompt it was never trained on and confounds the carrier change with a
prompt change. Rather than duplicate the register table and rely on whoever runs
it to pick the matching one, read the prompt out of the pair build's own
provenance, which is the artefact that records what the adapter actually saw.
Adds the Yarros-side evaluation the training exists to justify: does the adapter
move arbitrary prose toward Yarros, and can the instruct arm still expand a beat
to a paragraph on direction. Yarros-flavoured voice prompts (modern/neutral/
romantasy tiers so any Yarros voice in the modern tier is adapter-attributable,
not prompt-supplied) and a Yarros-register beat SYS on the chat generator.
voice_distance.py is the honest slice of adjudication that needs no seat: Burrows's
Delta over character bigrams against held-out Yarros. Its first cut mis-framed the
noise floor — it used the same-author distance (held-out vs itself) as the
between-arm significance threshold, which is the target, not the threshold. Fixed
to the measured floor: the within-arm seed spread, which is this metric's sampling
variance at this sample size, computed from the two seeds already generated rather
than assumed.
Result on the built corpus, ordering base-125-tuned < instruct-tuned <
base-unadapted, both adapters clearing the 0.046 measured floor (base +0.157,
instruct +0.076), and the ordering corroborating the independent held-out loss
ordering (Base below Instruct). One seed-pair per arm, so it corroborates rather
than settles; the full frozen adjudication still needs a romantasy control panel,
a second seed, and the gen seat for the beat-incumbent leg.
vh
Vuong Hoang