From dbca9a3c668bcdc8f4fd71d2a7e95d5d9ca03200 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 25 Aug 2026 21:30:45 -0700 Subject: [PATCH] docs(training-playbook): audit the whole manifest against the pairing rule MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §4.3's generalisation was stated and then not applied to the manifest that prompted it. brokkr-smithy-dev did the audit: most fields are intent-only, and the one pairing that would have caught the §4.1 cache failure -- the mask's sha against the loss-token delta -- existed by accident, because someone had asked for an encode report for unrelated reasons. Adds the audit table, and the rider that matters more than the table: put the observed check where it can actually FAIL. chat_template_sha256's pair is the sha of the string the tokenizer carries, but asserting that in the parent one line after assigning the file to the tokenizer compares a value to itself. It belongs in the encode worker -- a different process, across a pickle boundary, where an unset config key silently leaves every worker rendering through the checkpoint's own template. --- docs/pfi/training-throughput-playbook.md | 26 ++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/docs/pfi/training-throughput-playbook.md b/docs/pfi/training-throughput-playbook.md index dfd19d2..429fb6d 100644 --- a/docs/pfi/training-throughput-playbook.md +++ b/docs/pfi/training-throughput-playbook.md @@ -673,6 +673,32 @@ the configuration silently not taking effect, you need a second field recording an *observed* consequence — and the pairing is the check. A settings dump alone is decorative. +**Audit the whole manifest against that rule once.** Ours came out mostly +intent-only, and the pairing that saved us existed by accident: + +| configured | observed pair | | +|---|---|---| +| `max_seq_len` | truncation report | ✅ | +| impersonation mask sha | loss-token delta (−221,712, context identical) | ✅ *by luck* | +| `quantized_base` | tensor-level counts | ✅ | +| `chat_template_sha256` | sha of what the tokenizer **rendered** | ✅ *added after the audit* | +| LoRA rank / alpha / targets | `lora_B` norms, already collected | ⚠ available, unwired | +| eligibility override | — | correctly unpaired; its consequence is a decision, not a measurement | + +The mask row is the instructive one. The sha alone would have sat in the +manifest reading true while the cache served unmasked encodings (§4.1) — the +delta is what makes the sha mean anything, and we only had it because someone +asked for an encode report for unrelated reasons. + +**And put the observed check where it can actually fail.** `chat_template_sha256` +is a sha of a file; the pair is the sha of the string the tokenizer carries. But +asserting that in the parent, one line after assigning the file to the +tokenizer, compares a value to itself — inert again. It belongs in the **encode +worker**: a different process, reached across a pickle boundary, where +`if template:` is a real branch and an unset config key leaves every worker +silently rendering through the *checkpoint's own* template. That is the +train/serve skew the config key exists to prevent, and it raises nothing. + The dynamo counters are the third leg: cheap, in-band, and they name the recompile activity directly. Keep them best-effort and nullable — a missing counter table is not worth failing a seven-hour run over at save time.