diff --git a/docs/pfi/training-throughput-playbook.md b/docs/pfi/training-throughput-playbook.md index 57dd95e..c6f1e0e 100644 --- a/docs/pfi/training-throughput-playbook.md +++ b/docs/pfi/training-throughput-playbook.md @@ -268,6 +268,52 @@ separately rather than folding it into an intensity story. --- +## 3.10 ⭐⭐ Prove the SERVING path before you spend the training window + +Playbook-for-quants §4.1 says prove the quantization targets before spending +GPU time. The same rule applies one step later and is easier to skip: **prove +you can serve the artifact, in the shape you intend to serve it, before you +train it.** + +Worked failure, 2026-08-25. A ~7-hour LoRA run was built on the assumption that +the adapter could be hot-swapped onto a quantized base at serve time. The +sizing doc had flagged this correctly — *"serving the result is not settled… +if it still no-ops, the harness must emit merged weights, and Eitri needs that +requirement while he is early, not after the run"* — and then the check was +deferred rather than run. Tested after the fact: + + AttributeError: To support LoRA for MoE model, + 'get_expert_mapping' must be implemented + +**One grep would have found it.** `vllm/lora/utils.py::process_packed_modules_mapping` +branches on `is_moe_model()`, and the model class in question implements zero +occurrences of `get_expert_mapping`. Static fact about the serving stack, +available months before the run. + +The check is cheap and mechanical: + +```bash +# does the serving engine's model class support what you plan to do? +grep -c "SupportsLoRA\|get_expert_mapping" /model_executor/models/.py +# and: start the engine with the feature flag ONLY (no adapter needed). +# --enable-lora alone forces the machinery to initialise, which is where it fails. +``` + +Two generalisations worth carrying: + +- **Feature support is per-architecture, not per-family.** LoRA worked for the + dense sibling of this exact model family and not for the MoE one. "Model X is + supported" is not a statement about X's variants. +- **A capability gap in the serving engine is not fixable by the training + side.** No harness change, no quantization choice, and no adapter scoping + works around it — the adapter here never touched experts and was refused + anyway, because the refusal keys on the *model* being MoE. + +The recovery is usually fine (merge instead of hot-swap, at ~35 min per tune). +The point is that it should be a *decision* made before the window, not a +discovery made after — because the alternative it forecloses may be an +architecture choice, and by then you have already trained. + ## 4. Panel / consult discipline for perf work Perf investigations are unusually good at generating confident wrong answers,