docs(training-playbook): 4.4.1 sample identity at launch; 4.6.1 calibrate gates against correct input

4.4.1 -- the dirty-tree case was only half of the harness_commit problem. Run 2
launched CLEAN at 1909d86 and recorded 460f372, because three commits landed on
the same checkout during its seven hours and _git_commit() was called at save
time. Commit AHEAD of the code that ran, naming changes it never executed --
including the provenance fields this section prompted. Same defect as run 1's
BEHIND, opposite sign: the identity was sampled at the wrong moment. Sample at
launch, carry it, and record the dirty flag beside the commit rather than
instead of it. Generalises to every run-scoped identity: anything read at save
time describes the world at save time.

4.6.1 -- the inverse of the inert gate, and it costs trust rather than
correctness. A coherence gate false-rejected 'The capital of Portugal is
Lisbon' as degenerate against a global 15-word floor. The floor was calibrated
against the wrong reference, not set too strict. Lowering it globally would
blunt the check where short output genuinely is degeneration; the fix is a floor
per prompt. Write the positive test alongside the negative one.
This commit is contained in:
vh
2026-08-26 01:35:27 -07:00
parent bcf63db527
commit d54f25605f
+62
View File
@@ -742,6 +742,44 @@ Record `git describe --always --dirty` rather than a bare SHA if you want a
softer version, but an outright refusal is better: a run long enough to be worth
provenance is long enough to be worth one commit first.
#### ⚠⚠ 4.4.1 The same field lies in the OTHER direction too — sample at LAUNCH
The dirty-tree case above is only half of it, and the next run demonstrated the
half nobody had thought about.
Run 2 launched from a **clean** tree at commit `1909d86`. Its manifest recorded
`460f372`. Three commits landed on the same checkout during the seven hours it
trained — someone fixing unrelated things on a shared box — and `_git_commit()`
was called while building the provenance dict **at save time**. So it read HEAD
seven hours after the process had loaded its modules.
**The recorded commit was AHEAD of the code that ran**, and named changes the
run never executed — including, with some irony, the very provenance fields
this section prompted.
run 1 commit BEHIND the code (dirty tree, uncommitted work live)
run 2 commit AHEAD of the code (clean tree, HEAD moved during the run)
Same defect, opposite sign: **the identity was sampled at the wrong moment.** A
long run is long enough for the repo to move underneath it, and on a shared box
it will.
```python
# at LAUNCH, right after preflight — not in the provenance dict at save time
harness_identity = {
"harness_commit": _git_commit(),
"harness_dirty_at_launch": _git_is_dirty(),
}
```
Sample once, at start, carry it to the end. Record the dirty flag *beside* the
commit rather than instead of it — "which commit" and "was that commit the whole
story" are two questions and one field cannot answer both.
⚠ Generalises to every run-scoped identity you record: library versions,
config-file shas, dataset shas, the container tag. **Anything read at save time
describes the world at save time, not the world the run happened in.**
**Correcting it after the fact — annotate, never edit.** We left
`provenance.json` untouched and wrote a `PROVENANCE-NOTE.md` beside it. Editing
a shipped artifact so it says something it did not say is the worse failure.
@@ -823,6 +861,30 @@ that, because the instrument was dark.
If breaking it changes nothing, you have documentation, not a test.
#### ⚠ 4.6.1 …but calibrate the gate against a CORRECT result, not a convenient one
The opposite failure, and it costs trust rather than correctness. A coherence
gate written for a freshly-merged tune applied a single global floor — output
must exceed 15 words — and false-rejected on its first real run:
❌ [refusal-probe] only 6 words (min 15)
"The capital of Portugal is **Lisbon**."
A correct and complete answer to a six-word question. The floor was not too
strict; it was **calibrated against the wrong reference**, because a generative
prompt and a closed factual one have different correct lengths.
The fix that is available and wrong is lowering the global floor — that blunts
the check on exactly the prompts where six words genuinely *is* degeneration.
The fix is a floor per prompt, set against what a correct answer to *that*
prompt looks like.
**A gate that cannot fail is useless; a gate that fails on correct input is
worse**, because it spends attention on a false alarm and teaches everyone to
route around it. When you write the negative test (§4.6), also write the
positive one: confirm the gate PASSES a known-good result before you trust it to
reject a bad one.
### ⚠ 4.7 ⭐ Fix a stale measurement at the SOURCE, or the copy carries it forward
A launcher script carried the comment *"the 609 MB encode cache (2.5 min to