docs(training-playbook): 4.4.1 sample identity at launch; 4.6.1 calibrate gates against correct input
4.4.1 -- the dirty-tree case was only half of the harness_commit problem. Run 2 launched CLEAN at 1909d86 and recorded 460f372, because three commits landed on the same checkout during its seven hours and _git_commit() was called at save time. Commit AHEAD of the code that ran, naming changes it never executed -- including the provenance fields this section prompted. Same defect as run 1's BEHIND, opposite sign: the identity was sampled at the wrong moment. Sample at launch, carry it, and record the dirty flag beside the commit rather than instead of it. Generalises to every run-scoped identity: anything read at save time describes the world at save time. 4.6.1 -- the inverse of the inert gate, and it costs trust rather than correctness. A coherence gate false-rejected 'The capital of Portugal is Lisbon' as degenerate against a global 15-word floor. The floor was calibrated against the wrong reference, not set too strict. Lowering it globally would blunt the check where short output genuinely is degeneration; the fix is a floor per prompt. Write the positive test alongside the negative one.
This commit is contained in:
@@ -742,6 +742,44 @@ Record `git describe --always --dirty` rather than a bare SHA if you want a
|
||||
softer version, but an outright refusal is better: a run long enough to be worth
|
||||
provenance is long enough to be worth one commit first.
|
||||
|
||||
#### ⚠⚠ 4.4.1 The same field lies in the OTHER direction too — sample at LAUNCH
|
||||
|
||||
The dirty-tree case above is only half of it, and the next run demonstrated the
|
||||
half nobody had thought about.
|
||||
|
||||
Run 2 launched from a **clean** tree at commit `1909d86`. Its manifest recorded
|
||||
`460f372`. Three commits landed on the same checkout during the seven hours it
|
||||
trained — someone fixing unrelated things on a shared box — and `_git_commit()`
|
||||
was called while building the provenance dict **at save time**. So it read HEAD
|
||||
seven hours after the process had loaded its modules.
|
||||
|
||||
**The recorded commit was AHEAD of the code that ran**, and named changes the
|
||||
run never executed — including, with some irony, the very provenance fields
|
||||
this section prompted.
|
||||
|
||||
run 1 commit BEHIND the code (dirty tree, uncommitted work live)
|
||||
run 2 commit AHEAD of the code (clean tree, HEAD moved during the run)
|
||||
|
||||
Same defect, opposite sign: **the identity was sampled at the wrong moment.** A
|
||||
long run is long enough for the repo to move underneath it, and on a shared box
|
||||
it will.
|
||||
|
||||
```python
|
||||
# at LAUNCH, right after preflight — not in the provenance dict at save time
|
||||
harness_identity = {
|
||||
"harness_commit": _git_commit(),
|
||||
"harness_dirty_at_launch": _git_is_dirty(),
|
||||
}
|
||||
```
|
||||
|
||||
Sample once, at start, carry it to the end. Record the dirty flag *beside* the
|
||||
commit rather than instead of it — "which commit" and "was that commit the whole
|
||||
story" are two questions and one field cannot answer both.
|
||||
|
||||
⚠ Generalises to every run-scoped identity you record: library versions,
|
||||
config-file shas, dataset shas, the container tag. **Anything read at save time
|
||||
describes the world at save time, not the world the run happened in.**
|
||||
|
||||
**Correcting it after the fact — annotate, never edit.** We left
|
||||
`provenance.json` untouched and wrote a `PROVENANCE-NOTE.md` beside it. Editing
|
||||
a shipped artifact so it says something it did not say is the worse failure.
|
||||
@@ -823,6 +861,30 @@ that, because the instrument was dark.
|
||||
|
||||
If breaking it changes nothing, you have documentation, not a test.
|
||||
|
||||
#### ⚠ 4.6.1 …but calibrate the gate against a CORRECT result, not a convenient one
|
||||
|
||||
The opposite failure, and it costs trust rather than correctness. A coherence
|
||||
gate written for a freshly-merged tune applied a single global floor — output
|
||||
must exceed 15 words — and false-rejected on its first real run:
|
||||
|
||||
❌ [refusal-probe] only 6 words (min 15)
|
||||
"The capital of Portugal is **Lisbon**."
|
||||
|
||||
A correct and complete answer to a six-word question. The floor was not too
|
||||
strict; it was **calibrated against the wrong reference**, because a generative
|
||||
prompt and a closed factual one have different correct lengths.
|
||||
|
||||
The fix that is available and wrong is lowering the global floor — that blunts
|
||||
the check on exactly the prompts where six words genuinely *is* degeneration.
|
||||
The fix is a floor per prompt, set against what a correct answer to *that*
|
||||
prompt looks like.
|
||||
|
||||
**A gate that cannot fail is useless; a gate that fails on correct input is
|
||||
worse**, because it spends attention on a false alarm and teaches everyone to
|
||||
route around it. When you write the negative test (§4.6), also write the
|
||||
positive one: confirm the gate PASSES a known-good result before you trust it to
|
||||
reject a bad one.
|
||||
|
||||
### ⚠ 4.7 ⭐ Fix a stale measurement at the SOURCE, or the copy carries it forward
|
||||
|
||||
A launcher script carried the comment *"the 609 MB encode cache (2.5 min to
|
||||
|
||||
Reference in New Issue
Block a user