docs(training-playbook): audit the whole manifest against the pairing rule
§4.3's generalisation was stated and then not applied to the manifest that prompted it. brokkr-smithy-dev did the audit: most fields are intent-only, and the one pairing that would have caught the §4.1 cache failure -- the mask's sha against the loss-token delta -- existed by accident, because someone had asked for an encode report for unrelated reasons. Adds the audit table, and the rider that matters more than the table: put the observed check where it can actually FAIL. chat_template_sha256's pair is the sha of the string the tokenizer carries, but asserting that in the parent one line after assigning the file to the tokenizer compares a value to itself. It belongs in the encode worker -- a different process, across a pickle boundary, where an unset config key silently leaves every worker rendering through the checkpoint's own template.
This commit is contained in:
@@ -673,6 +673,32 @@ the configuration silently not taking effect, you need a second field recording
|
||||
an *observed* consequence — and the pairing is the check. A settings dump alone
|
||||
is decorative.
|
||||
|
||||
**Audit the whole manifest against that rule once.** Ours came out mostly
|
||||
intent-only, and the pairing that saved us existed by accident:
|
||||
|
||||
| configured | observed pair | |
|
||||
|---|---|---|
|
||||
| `max_seq_len` | truncation report | ✅ |
|
||||
| impersonation mask sha | loss-token delta (−221,712, context identical) | ✅ *by luck* |
|
||||
| `quantized_base` | tensor-level counts | ✅ |
|
||||
| `chat_template_sha256` | sha of what the tokenizer **rendered** | ✅ *added after the audit* |
|
||||
| LoRA rank / alpha / targets | `lora_B` norms, already collected | ⚠ available, unwired |
|
||||
| eligibility override | — | correctly unpaired; its consequence is a decision, not a measurement |
|
||||
|
||||
The mask row is the instructive one. The sha alone would have sat in the
|
||||
manifest reading true while the cache served unmasked encodings (§4.1) — the
|
||||
delta is what makes the sha mean anything, and we only had it because someone
|
||||
asked for an encode report for unrelated reasons.
|
||||
|
||||
**And put the observed check where it can actually fail.** `chat_template_sha256`
|
||||
is a sha of a file; the pair is the sha of the string the tokenizer carries. But
|
||||
asserting that in the parent, one line after assigning the file to the
|
||||
tokenizer, compares a value to itself — inert again. It belongs in the **encode
|
||||
worker**: a different process, reached across a pickle boundary, where
|
||||
`if template:` is a real branch and an unset config key leaves every worker
|
||||
silently rendering through the *checkpoint's own* template. That is the
|
||||
train/serve skew the config key exists to prevent, and it raises nothing.
|
||||
|
||||
The dynamo counters are the third leg: cheap, in-band, and they name the
|
||||
recompile activity directly. Keep them best-effort and nullable — a missing
|
||||
counter table is not worth failing a seven-hour run over at save time.
|
||||
|
||||
Reference in New Issue
Block a user