3e7d3a3e3ddd3fbbf71a6f73d06607f428880cd1
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 vh
|
5e6611466c |
audit_pairs_sourcenames: --filter-out, so the detector is also the fix
An already-built pair set cannot be repaired by build_sft_pairs.py --source-entities;
that flag only works at generation time. Hemingway's and Yarros's sets both predate it.
The contamination is in the BEAT, so dropping the row removes it outright. Measured on
the Hemingway train pairs: 7,094 -> 7,024, 70 dropped, 0.99% of the training data. That
is cheaper and cleaner than regenerating 70 beats against a second generator session,
which would leave the set mixed-provenance for the sake of 1% more data.
Verified by read-back rather than by the write succeeding: re-auditing the filtered file
reports 0 of 7,024 on both columns, controls green, GATE PASS.
Two refusals rather than a best-effort write:
- a contaminated RESPONSE column aborts. That is a different fault -- pairs built
against an unrenamed corpus -- and dropping rows would hide it instead of fixing it.
- more than one --pairs input aborts, because the output is a single file and would
silently merge train and val into one.
Also cross-validated the detector against the lv-bronte pair sets on real data, where the
answer is already on the record:
pairs-full + pairs-val (post-fix) 0 of 3,858 matches the recorded "0 leaks across
3,858 pairs" exactly
pairs-full.CONTAMINATED 15 of 792 = 1.89%, Rochester x6, Jane, Brocklehurst
x2, Beck, Fairfax, Burns, Helen, Eyre -- against a
record of "13 of the first 714 beats (1.8%)" with
the same names
An independently written instrument reproducing a documented finding at the right
magnitude, on the right names, is the control that says its zeroes mean absent and not
blind.
|
||
|
vh
|
051b99e063 |
audit_entity_map: the rename can damage the prose and no gate will ever say so
audit_stoplist.py finds surfaces wrongly held OUT of the entity map -- a stoplisted character is an undetectable leak. This is the mirror: surfaces wrongly held IN it. leak_gate.py only ever asks whether the author's names are GONE, never whether non-names were spared, so renaming `the Chinese` into an invented surname passes it perfectly. Found sideways on Hemingway. The pairs audit reported beats naming African, Chinese, X-ray, Republican and Cezanne as leaks -- correctly, those surfaces really were removed from the corpus. Reading why turned up the larger defect: they should never have been renameable in the first place. Measured on the Hemingway map, both controls green: positive `other` 764/1356 article-preceded = 0.56 negative 100 honorific-confirmed people, highest Inglés at 0.26, bulk 0.00-0.06 FLAGGED 130 of 946 surfaces, 1,616 instances = 0.162% of corpus words The signal is an article in front of the surface: you write `the Frenchman` and `a Martini`, never `the Rinaldi`. It is a heuristic and every hit is reported FOR READING, never auto-removed -- `the Widow` and `the Informer` are genuine Hemingway epithet-names that SHOULD be renamed, and the band's own top entry makes the point, since Inglés at 0.26 is an in-world nickname deliberately kept renameable and sits just under the bar. Initials are excluded from the negative-control band rather than admitted to it. `Mr. P.` is an initial, not a person, so letting it in lets a map defect poison the control that validates the detector -- on Hemingway `P` (0.32, every occurrence `the P. O. U. M.`) was the one surface failing a band whose next highest was 0.26. Initials take no article and are invisible to the scan anyway, so every surface of two characters or fewer is now listed unconditionally. Sixteen of them are in this map, C at 274 occurrences; the same class as the `G` that was caught by hand about to be renamed to a surname 248 times. The unresolved count that drives the exit code is computed over every flagged surface, not the --show slice. Tying a gate's verdict to a display flag is the same defect as a log filter that turns a real event into a clean zero. Also corrects a wrong claim in audit_pairs_sourcenames.py's docstring: the Hemingway rename did not HOLD 591 surfaces. Paris, Madrid and Spain survive because the stoplist keeps them out of the entity map before it is built, so the map is exactly the removed set -- 941 surfaces, 941 removed, 0 kept. Measured per run rather than assumed, because a pipeline that carried kept surfaces into the map would report every `Paris` as a leak. |
||
|
vh
|
0bb4938518 |
lv-hemingway: pre-register the v2 gate, and fix the floor rule that decided lv-bronte
The gate design is written before any generation exists, because lv-bronte's
verdict turned on a choice that was only visible after the numbers printed.
THE FLOOR RULE IS NOW PAIRWISE. lv-bronte computed the noise floor as the largest
within-arm seed spread across ALL arms present. Its ckpt475 shipped at +0.193
against a 0.251 floor set entirely by ckpt925 -- a third arm nobody was shipping,
on one outlier seed. Scored against the arm it was actually compared to, the floor
is 0.092 and the same gap clears at 2.1x. A candidate's verdict must not depend on
which other arms happened to be generated. voice_distance.py now prints both floors
and flags any disagreement, so the lv-bronte record stays comparable.
audit_pairs_sourcenames.py closes the blind spot leak_gate.py has by construction:
it reads the corpus and the renamed copies, never the generated beats, so it cannot
see a beat-writing model restoring the author's real character names. Run over the
Hemingway pairs, which predate build_sft_pairs.py --source-entities:
val 0 of 200 -- the eval fixture is clean, the gate is unconfounded
train 70 of 7,094 (0.96%) -- Santiago x16, Catherine x7, Rinaldi x3, Brett,
Harry, Jake, Pablo, Nick, Maria ...
responses 0 of 7,294 -- the lv-bronte beat-only signature exactly
A matched surface is only counted when the rename actually removed it, verified
against the renamed copies, so a beat naming a held real-world place is not a leak.
Controls run every time: 941/941 surfaces found in the unrenamed source, nonce
absent from both trees, and 6 planted canonical names detected 6/6.
voice_distance.py --author is now REQUIRED. It was hardcoded "Yarros" and printed
"reference: held-out Yarros" over Brontë's numbers into a committed artifact. A
default would have moved the silent-wrong-label failure rather than removed it. The
stale "one seed-pair per arm / corroborates Base < Instruct" footer is replaced with
what the run actually carries.
Gate design: three arms (base-unadapted, ckpt1750, ckpt850), 60 beats, 4 seeds.
ckpt850 is present because the loss curve cannot separate it from ckpt1750 -- +0.0040
against a 0.0044 median neighbour jitter, with three checkpoints inside one jitter of
the minimum. adapter/ is excluded: +0.0762 is 17.4x the jitter and is resolved without
a gate.
|