docs(training-playbook): 4.4.1 sample identity at launch; 4.6.1 calibrate gates against correct input
4.4.1 -- the dirty-tree case was only half of the harness_commit problem. Run 2 launched CLEAN at 1909d86 and recorded 460f372, because three commits landed on the same checkout during its seven hours and _git_commit() was called at save time. Commit AHEAD of the code that ran, naming changes it never executed -- including the provenance fields this section prompted. Same defect as run 1's BEHIND, opposite sign: the identity was sampled at the wrong moment. Sample at launch, carry it, and record the dirty flag beside the commit rather than instead of it. Generalises to every run-scoped identity: anything read at save time describes the world at save time. 4.6.1 -- the inverse of the inert gate, and it costs trust rather than correctness. A coherence gate false-rejected 'The capital of Portugal is Lisbon' as degenerate against a global 15-word floor. The floor was calibrated against the wrong reference, not set too strict. Lowering it globally would blunt the check where short output genuinely is degeneration; the fix is a floor per prompt. Write the positive test alongside the negative one.
This commit is contained in:
@@ -742,6 +742,44 @@ Record `git describe --always --dirty` rather than a bare SHA if you want a
|
|||||||
softer version, but an outright refusal is better: a run long enough to be worth
|
softer version, but an outright refusal is better: a run long enough to be worth
|
||||||
provenance is long enough to be worth one commit first.
|
provenance is long enough to be worth one commit first.
|
||||||
|
|
||||||
|
#### ⚠⚠ 4.4.1 The same field lies in the OTHER direction too — sample at LAUNCH
|
||||||
|
|
||||||
|
The dirty-tree case above is only half of it, and the next run demonstrated the
|
||||||
|
half nobody had thought about.
|
||||||
|
|
||||||
|
Run 2 launched from a **clean** tree at commit `1909d86`. Its manifest recorded
|
||||||
|
`460f372`. Three commits landed on the same checkout during the seven hours it
|
||||||
|
trained — someone fixing unrelated things on a shared box — and `_git_commit()`
|
||||||
|
was called while building the provenance dict **at save time**. So it read HEAD
|
||||||
|
seven hours after the process had loaded its modules.
|
||||||
|
|
||||||
|
**The recorded commit was AHEAD of the code that ran**, and named changes the
|
||||||
|
run never executed — including, with some irony, the very provenance fields
|
||||||
|
this section prompted.
|
||||||
|
|
||||||
|
run 1 commit BEHIND the code (dirty tree, uncommitted work live)
|
||||||
|
run 2 commit AHEAD of the code (clean tree, HEAD moved during the run)
|
||||||
|
|
||||||
|
Same defect, opposite sign: **the identity was sampled at the wrong moment.** A
|
||||||
|
long run is long enough for the repo to move underneath it, and on a shared box
|
||||||
|
it will.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# at LAUNCH, right after preflight — not in the provenance dict at save time
|
||||||
|
harness_identity = {
|
||||||
|
"harness_commit": _git_commit(),
|
||||||
|
"harness_dirty_at_launch": _git_is_dirty(),
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Sample once, at start, carry it to the end. Record the dirty flag *beside* the
|
||||||
|
commit rather than instead of it — "which commit" and "was that commit the whole
|
||||||
|
story" are two questions and one field cannot answer both.
|
||||||
|
|
||||||
|
⚠ Generalises to every run-scoped identity you record: library versions,
|
||||||
|
config-file shas, dataset shas, the container tag. **Anything read at save time
|
||||||
|
describes the world at save time, not the world the run happened in.**
|
||||||
|
|
||||||
**Correcting it after the fact — annotate, never edit.** We left
|
**Correcting it after the fact — annotate, never edit.** We left
|
||||||
`provenance.json` untouched and wrote a `PROVENANCE-NOTE.md` beside it. Editing
|
`provenance.json` untouched and wrote a `PROVENANCE-NOTE.md` beside it. Editing
|
||||||
a shipped artifact so it says something it did not say is the worse failure.
|
a shipped artifact so it says something it did not say is the worse failure.
|
||||||
@@ -823,6 +861,30 @@ that, because the instrument was dark.
|
|||||||
|
|
||||||
If breaking it changes nothing, you have documentation, not a test.
|
If breaking it changes nothing, you have documentation, not a test.
|
||||||
|
|
||||||
|
#### ⚠ 4.6.1 …but calibrate the gate against a CORRECT result, not a convenient one
|
||||||
|
|
||||||
|
The opposite failure, and it costs trust rather than correctness. A coherence
|
||||||
|
gate written for a freshly-merged tune applied a single global floor — output
|
||||||
|
must exceed 15 words — and false-rejected on its first real run:
|
||||||
|
|
||||||
|
❌ [refusal-probe] only 6 words (min 15)
|
||||||
|
"The capital of Portugal is **Lisbon**."
|
||||||
|
|
||||||
|
A correct and complete answer to a six-word question. The floor was not too
|
||||||
|
strict; it was **calibrated against the wrong reference**, because a generative
|
||||||
|
prompt and a closed factual one have different correct lengths.
|
||||||
|
|
||||||
|
The fix that is available and wrong is lowering the global floor — that blunts
|
||||||
|
the check on exactly the prompts where six words genuinely *is* degeneration.
|
||||||
|
The fix is a floor per prompt, set against what a correct answer to *that*
|
||||||
|
prompt looks like.
|
||||||
|
|
||||||
|
**A gate that cannot fail is useless; a gate that fails on correct input is
|
||||||
|
worse**, because it spends attention on a false alarm and teaches everyone to
|
||||||
|
route around it. When you write the negative test (§4.6), also write the
|
||||||
|
positive one: confirm the gate PASSES a known-good result before you trust it to
|
||||||
|
reject a bad one.
|
||||||
|
|
||||||
### ⚠ 4.7 ⭐ Fix a stale measurement at the SOURCE, or the copy carries it forward
|
### ⚠ 4.7 ⭐ Fix a stale measurement at the SOURCE, or the copy carries it forward
|
||||||
|
|
||||||
A launcher script carried the comment *"the 609 MB encode cache (2.5 min to
|
A launcher script carried the comment *"the 609 MB encode cache (2.5 min to
|
||||||
|
|||||||
Reference in New Issue
Block a user