diff --git a/docs/pfi/training-throughput-playbook.md b/docs/pfi/training-throughput-playbook.md index 1da17e7..51f4479 100644 --- a/docs/pfi/training-throughput-playbook.md +++ b/docs/pfi/training-throughput-playbook.md @@ -861,6 +861,38 @@ that, because the instrument was dark. If breaking it changes nothing, you have documentation, not a test. +#### ⚠ 4.6.2 ⭐⭐ A NULL RESULT needs a positive control before it counts as a null + +`0.00% / floor 0.00% / max_item 0.0%` across all 72 items is the correct output +of a memorisation probe on a model that has never seen the corpus. **It is also +the exact output of a probe that is not firing at all**, and nothing in the +number distinguishes them. + +The move that separates them costs one minute — drive the metric's own function +with inputs whose answer you already know: + + identical text 100.00% + half-verbatim 65.38% + unrelated English 0.00% + empty string 0.00% + +Now the zero means something: the instrument *can* go red, and did not. + +⚠ **This is §4.5's inert gate wearing a different face.** There it was a check +that could not return "fail"; here it is a measurement that cannot return +non-zero. A clean null is the most reassuring output any instrument produces and +the least self-evidencing, so it is precisely the one that has to be earned. + +Same trap in a metric that reads **identical on both arms**: a diversity battery +whose rp family froze zero markers reported an attractor hit rate of 0.0 for +base *and* tuned. That reads as "no attractors, clean result" and means "this +instrument cannot discriminate on this family." Report it as a bounded +limitation — that family is measured on one axis rather than two — never as a +delta of zero. **A check that returns the same value for every input is not +measuring.** + +(Both from the run-2 gate, brokkr-smithy-dev, 2026-08-26.) + #### ⚠ 4.6.1 …but calibrate the gate against a CORRECT result, not a convenient one The opposite failure, and it costs trust rather than correctness. A coherence @@ -949,6 +981,9 @@ completed, plausible, wrong run above. [ ] log rotates on relaunch one file describes one run [ ] present-and-null, not absent a run that claims nothing must say so explicitly [ ] watchdog tested negative kill something and confirm it fires +[ ] every NULL has a positive drive the metric with known-answer inputs; + control a clean zero is the least self-evidencing + result any instrument produces [ ] stale numbers grepped for the figure, repo-wide, not just in view ```