audit_pairs_sourcenames: --filter-out, so the detector is also the fix
An already-built pair set cannot be repaired by build_sft_pairs.py --source-entities;
that flag only works at generation time. Hemingway's and Yarros's sets both predate it.
The contamination is in the BEAT, so dropping the row removes it outright. Measured on
the Hemingway train pairs: 7,094 -> 7,024, 70 dropped, 0.99% of the training data. That
is cheaper and cleaner than regenerating 70 beats against a second generator session,
which would leave the set mixed-provenance for the sake of 1% more data.
Verified by read-back rather than by the write succeeding: re-auditing the filtered file
reports 0 of 7,024 on both columns, controls green, GATE PASS.
Two refusals rather than a best-effort write:
- a contaminated RESPONSE column aborts. That is a different fault -- pairs built
against an unrenamed corpus -- and dropping rows would hide it instead of fixing it.
- more than one --pairs input aborts, because the output is a single file and would
silently merge train and val into one.
Also cross-validated the detector against the lv-bronte pair sets on real data, where the
answer is already on the record:
pairs-full + pairs-val (post-fix) 0 of 3,858 matches the recorded "0 leaks across
3,858 pairs" exactly
pairs-full.CONTAMINATED 15 of 792 = 1.89%, Rochester x6, Jane, Brocklehurst
x2, Beck, Fairfax, Burns, Helen, Eyre -- against a
record of "13 of the first 714 beats (1.8%)" with
the same names
An independently written instrument reproducing a documented finding at the right
magnitude, on the right names, is the control that says its zeroes mean absent and not
blind.
This commit is contained in:
@@ -65,6 +65,16 @@ def main() -> int:
|
||||
help="rename.py's renameable threshold; mirrored from leak_gate.py")
|
||||
ap.add_argument("--report", default=None, help="write the full JSON breakdown here")
|
||||
ap.add_argument("--show", type=int, default=25, help="example beats to print")
|
||||
ap.add_argument("--filter-out", default=None, metavar="PATH",
|
||||
help="write a copy of the pairs with every contaminated ROW removed. "
|
||||
"Turns this detector into the fix for an already-built pair set: "
|
||||
"the contamination is in the beat, so dropping the row removes it "
|
||||
"outright, and on Hemingway that costs 0.96%% of the training data. "
|
||||
"Cheaper and cleaner than regenerating 70 beats against a second "
|
||||
"generator session, which would leave the set mixed-provenance. "
|
||||
"Refuses to write when a RESPONSE is contaminated -- that is a "
|
||||
"different fault (pairs built against an unrenamed corpus) and "
|
||||
"dropping rows would hide it rather than fix it.")
|
||||
a = ap.parse_args()
|
||||
|
||||
ents_all = json.loads(Path(a.entities).read_text())
|
||||
@@ -157,6 +167,40 @@ def main() -> int:
|
||||
}, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(f"\n wrote {a.report}")
|
||||
|
||||
if a.filter_out:
|
||||
if nr:
|
||||
print("\n== REFUSING to write a filtered copy: the RESPONSE column is contaminated.")
|
||||
print(" That is not the beat-generator fault and dropping rows would hide it.")
|
||||
print(" Check the provenance `corpus` path -- the pairs were probably built")
|
||||
print(" against an unrenamed corpus, and the fix is upstream of this file.")
|
||||
return 2
|
||||
if not (pos_ok and neg_ok):
|
||||
print("\n== REFUSING to write a filtered copy: the controls did not pass, so the")
|
||||
print(" set of rows to drop is not trustworthy.")
|
||||
return 2
|
||||
if len({r["_src"] for r in rows}) != 1:
|
||||
print("\n== REFUSING to write a filtered copy from more than one --pairs file:")
|
||||
print(" the output is a single file and would silently merge train and val.")
|
||||
print(" Filter each input separately.")
|
||||
return 2
|
||||
drop = set()
|
||||
for i, r in enumerate(rows):
|
||||
if removed_pat.search(r.get("beat") or ""):
|
||||
drop.add(i)
|
||||
out = Path(a.filter_out)
|
||||
kept = 0
|
||||
with out.open("w", encoding="utf-8") as fh:
|
||||
for i, r in enumerate(rows):
|
||||
if i in drop:
|
||||
continue
|
||||
r = {k: v for k, v in r.items() if k != "_src"}
|
||||
fh.write(json.dumps(r, ensure_ascii=False) + "\n")
|
||||
kept += 1
|
||||
print(f"\n wrote {out}: {kept} pairs kept, {len(drop)} dropped "
|
||||
f"({len(drop) / len(rows):.2%})")
|
||||
print(" ⚠ Re-run this audit against the filtered file before training on it. A fix")
|
||||
print(" that is not read back is a claim, not a result.")
|
||||
|
||||
ok = pos_ok and neg_ok and nb == 0 and nr == 0
|
||||
print(f"\n GATE: {'PASS' if ok else 'FAIL'}")
|
||||
return 0 if ok else 2
|
||||
|
||||
Reference in New Issue
Block a user