docs(training-playbook): audit the whole manifest against the pairing rule
§4.3's generalisation was stated and then not applied to the manifest that prompted it. brokkr-smithy-dev did the audit: most fields are intent-only, and the one pairing that would have caught the §4.1 cache failure -- the mask's sha against the loss-token delta -- existed by accident, because someone had asked for an encode report for unrelated reasons. Adds the audit table, and the rider that matters more than the table: put the observed check where it can actually FAIL. chat_template_sha256's pair is the sha of the string the tokenizer carries, but asserting that in the parent one line after assigning the file to the tokenizer compares a value to itself. It belongs in the encode worker -- a different process, across a pickle boundary, where an unset config key silently leaves every worker rendering through the checkpoint's own template.
This commit is contained in:
@@ -673,6 +673,32 @@ the configuration silently not taking effect, you need a second field recording
|
|||||||
an *observed* consequence — and the pairing is the check. A settings dump alone
|
an *observed* consequence — and the pairing is the check. A settings dump alone
|
||||||
is decorative.
|
is decorative.
|
||||||
|
|
||||||
|
**Audit the whole manifest against that rule once.** Ours came out mostly
|
||||||
|
intent-only, and the pairing that saved us existed by accident:
|
||||||
|
|
||||||
|
| configured | observed pair | |
|
||||||
|
|---|---|---|
|
||||||
|
| `max_seq_len` | truncation report | ✅ |
|
||||||
|
| impersonation mask sha | loss-token delta (−221,712, context identical) | ✅ *by luck* |
|
||||||
|
| `quantized_base` | tensor-level counts | ✅ |
|
||||||
|
| `chat_template_sha256` | sha of what the tokenizer **rendered** | ✅ *added after the audit* |
|
||||||
|
| LoRA rank / alpha / targets | `lora_B` norms, already collected | ⚠ available, unwired |
|
||||||
|
| eligibility override | — | correctly unpaired; its consequence is a decision, not a measurement |
|
||||||
|
|
||||||
|
The mask row is the instructive one. The sha alone would have sat in the
|
||||||
|
manifest reading true while the cache served unmasked encodings (§4.1) — the
|
||||||
|
delta is what makes the sha mean anything, and we only had it because someone
|
||||||
|
asked for an encode report for unrelated reasons.
|
||||||
|
|
||||||
|
**And put the observed check where it can actually fail.** `chat_template_sha256`
|
||||||
|
is a sha of a file; the pair is the sha of the string the tokenizer carries. But
|
||||||
|
asserting that in the parent, one line after assigning the file to the
|
||||||
|
tokenizer, compares a value to itself — inert again. It belongs in the **encode
|
||||||
|
worker**: a different process, reached across a pickle boundary, where
|
||||||
|
`if template:` is a real branch and an unset config key leaves every worker
|
||||||
|
silently rendering through the *checkpoint's own* template. That is the
|
||||||
|
train/serve skew the config key exists to prevent, and it raises nothing.
|
||||||
|
|
||||||
The dynamo counters are the third leg: cheap, in-band, and they name the
|
The dynamo counters are the third leg: cheap, in-band, and they name the
|
||||||
recompile activity directly. Keep them best-effort and nullable — a missing
|
recompile activity directly. Keep them best-effort and nullable — a missing
|
||||||
counter table is not worth failing a seven-hour run over at save time.
|
counter table is not worth failing a seven-hour run over at save time.
|
||||||
|
|||||||
Reference in New Issue
Block a user