docs(training-playbook): audit the whole manifest against the pairing rule

§4.3's generalisation was stated and then not applied to the manifest that
prompted it. brokkr-smithy-dev did the audit: most fields are intent-only, and
the one pairing that would have caught the §4.1 cache failure -- the mask's sha
against the loss-token delta -- existed by accident, because someone had asked
for an encode report for unrelated reasons.

Adds the audit table, and the rider that matters more than the table: put the
observed check where it can actually FAIL. chat_template_sha256's pair is the
sha of the string the tokenizer carries, but asserting that in the parent one
line after assigning the file to the tokenizer compares a value to itself. It
belongs in the encode worker -- a different process, across a pickle boundary,
where an unset config key silently leaves every worker rendering through the
checkpoint's own template.
This commit is contained in:
2026-08-25 21:30:45 -07:00
parent c1db188e6a
commit dbca9a3c66
+26
View File
@@ -673,6 +673,32 @@ the configuration silently not taking effect, you need a second field recording
an *observed* consequence — and the pairing is the check. A settings dump alone
is decorative.
**Audit the whole manifest against that rule once.** Ours came out mostly
intent-only, and the pairing that saved us existed by accident:
| configured | observed pair | |
|---|---|---|
| `max_seq_len` | truncation report | ✅ |
| impersonation mask sha | loss-token delta (221,712, context identical) | ✅ *by luck* |
| `quantized_base` | tensor-level counts | ✅ |
| `chat_template_sha256` | sha of what the tokenizer **rendered** | ✅ *added after the audit* |
| LoRA rank / alpha / targets | `lora_B` norms, already collected | ⚠ available, unwired |
| eligibility override | — | correctly unpaired; its consequence is a decision, not a measurement |
The mask row is the instructive one. The sha alone would have sat in the
manifest reading true while the cache served unmasked encodings (§4.1) — the
delta is what makes the sha mean anything, and we only had it because someone
asked for an encode report for unrelated reasons.
**And put the observed check where it can actually fail.** `chat_template_sha256`
is a sha of a file; the pair is the sha of the string the tokenizer carries. But
asserting that in the parent, one line after assigning the file to the
tokenizer, compares a value to itself — inert again. It belongs in the **encode
worker**: a different process, reached across a pickle boundary, where
`if template:` is a real branch and an unset config key leaves every worker
silently rendering through the *checkpoint's own* template. That is the
train/serve skew the config key exists to prevent, and it raises nothing.
The dynamo counters are the third leg: cheap, in-band, and they name the
recompile activity directly. Keep them best-effort and nullable — a missing
counter table is not worth failing a seven-hour run over at save time.