diff --git a/docs/pfi/training-throughput-playbook.md b/docs/pfi/training-throughput-playbook.md index dfd19d2..429fb6d 100644 --- a/docs/pfi/training-throughput-playbook.md +++ b/docs/pfi/training-throughput-playbook.md @@ -673,6 +673,32 @@ the configuration silently not taking effect, you need a second field recording an *observed* consequence — and the pairing is the check. A settings dump alone is decorative. +**Audit the whole manifest against that rule once.** Ours came out mostly +intent-only, and the pairing that saved us existed by accident: + +| configured | observed pair | | +|---|---|---| +| `max_seq_len` | truncation report | ✅ | +| impersonation mask sha | loss-token delta (−221,712, context identical) | ✅ *by luck* | +| `quantized_base` | tensor-level counts | ✅ | +| `chat_template_sha256` | sha of what the tokenizer **rendered** | ✅ *added after the audit* | +| LoRA rank / alpha / targets | `lora_B` norms, already collected | ⚠ available, unwired | +| eligibility override | — | correctly unpaired; its consequence is a decision, not a measurement | + +The mask row is the instructive one. The sha alone would have sat in the +manifest reading true while the cache served unmasked encodings (§4.1) — the +delta is what makes the sha mean anything, and we only had it because someone +asked for an encode report for unrelated reasons. + +**And put the observed check where it can actually fail.** `chat_template_sha256` +is a sha of a file; the pair is the sha of the string the tokenizer carries. But +asserting that in the parent, one line after assigning the file to the +tokenizer, compares a value to itself — inert again. It belongs in the **encode +worker**: a different process, reached across a pickle boundary, where +`if template:` is a real branch and an unset config key leaves every worker +silently rendering through the *checkpoint's own* template. That is the +train/serve skew the config key exists to prevent, and it raises nothing. + The dynamo counters are the third leg: cheap, in-band, and they name the recompile activity directly. Keep them best-effort and nullable — a missing counter table is not worth failing a seven-hour run over at save time.