From d54f25605f34f17031cdac046028b7659ad534fe Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 26 Aug 2026 01:35:27 -0700 Subject: [PATCH] docs(training-playbook): 4.4.1 sample identity at launch; 4.6.1 calibrate gates against correct input 4.4.1 -- the dirty-tree case was only half of the harness_commit problem. Run 2 launched CLEAN at 1909d86 and recorded 460f372, because three commits landed on the same checkout during its seven hours and _git_commit() was called at save time. Commit AHEAD of the code that ran, naming changes it never executed -- including the provenance fields this section prompted. Same defect as run 1's BEHIND, opposite sign: the identity was sampled at the wrong moment. Sample at launch, carry it, and record the dirty flag beside the commit rather than instead of it. Generalises to every run-scoped identity: anything read at save time describes the world at save time. 4.6.1 -- the inverse of the inert gate, and it costs trust rather than correctness. A coherence gate false-rejected 'The capital of Portugal is Lisbon' as degenerate against a global 15-word floor. The floor was calibrated against the wrong reference, not set too strict. Lowering it globally would blunt the check where short output genuinely is degeneration; the fix is a floor per prompt. Write the positive test alongside the negative one. --- docs/pfi/training-throughput-playbook.md | 62 ++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/docs/pfi/training-throughput-playbook.md b/docs/pfi/training-throughput-playbook.md index 429fb6d..1da17e7 100644 --- a/docs/pfi/training-throughput-playbook.md +++ b/docs/pfi/training-throughput-playbook.md @@ -742,6 +742,44 @@ Record `git describe --always --dirty` rather than a bare SHA if you want a softer version, but an outright refusal is better: a run long enough to be worth provenance is long enough to be worth one commit first. +#### ⚠⚠ 4.4.1 The same field lies in the OTHER direction too — sample at LAUNCH + +The dirty-tree case above is only half of it, and the next run demonstrated the +half nobody had thought about. + +Run 2 launched from a **clean** tree at commit `1909d86`. Its manifest recorded +`460f372`. Three commits landed on the same checkout during the seven hours it +trained — someone fixing unrelated things on a shared box — and `_git_commit()` +was called while building the provenance dict **at save time**. So it read HEAD +seven hours after the process had loaded its modules. + +**The recorded commit was AHEAD of the code that ran**, and named changes the +run never executed — including, with some irony, the very provenance fields +this section prompted. + + run 1 commit BEHIND the code (dirty tree, uncommitted work live) + run 2 commit AHEAD of the code (clean tree, HEAD moved during the run) + +Same defect, opposite sign: **the identity was sampled at the wrong moment.** A +long run is long enough for the repo to move underneath it, and on a shared box +it will. + +```python +# at LAUNCH, right after preflight — not in the provenance dict at save time +harness_identity = { + "harness_commit": _git_commit(), + "harness_dirty_at_launch": _git_is_dirty(), +} +``` + +Sample once, at start, carry it to the end. Record the dirty flag *beside* the +commit rather than instead of it — "which commit" and "was that commit the whole +story" are two questions and one field cannot answer both. + +⚠ Generalises to every run-scoped identity you record: library versions, +config-file shas, dataset shas, the container tag. **Anything read at save time +describes the world at save time, not the world the run happened in.** + **Correcting it after the fact — annotate, never edit.** We left `provenance.json` untouched and wrote a `PROVENANCE-NOTE.md` beside it. Editing a shipped artifact so it says something it did not say is the worse failure. @@ -823,6 +861,30 @@ that, because the instrument was dark. If breaking it changes nothing, you have documentation, not a test. +#### ⚠ 4.6.1 …but calibrate the gate against a CORRECT result, not a convenient one + +The opposite failure, and it costs trust rather than correctness. A coherence +gate written for a freshly-merged tune applied a single global floor — output +must exceed 15 words — and false-rejected on its first real run: + + ❌ [refusal-probe] only 6 words (min 15) + "The capital of Portugal is **Lisbon**." + +A correct and complete answer to a six-word question. The floor was not too +strict; it was **calibrated against the wrong reference**, because a generative +prompt and a closed factual one have different correct lengths. + +The fix that is available and wrong is lowering the global floor — that blunts +the check on exactly the prompts where six words genuinely *is* degeneration. +The fix is a floor per prompt, set against what a correct answer to *that* +prompt looks like. + +**A gate that cannot fail is useless; a gate that fails on correct input is +worse**, because it spends attention on a false alarm and teaches everyone to +route around it. When you write the negative test (§4.6), also write the +positive one: confirm the gate PASSES a known-good result before you trust it to +reject a bad one. + ### ⚠ 4.7 ⭐ Fix a stale measurement at the SOURCE, or the copy carries it forward A launcher script carried the comment *"the 609 MB encode cache (2.5 min to