docs(training-playbook): prove the serving path before spending the window
§3.10. The quantization playbook already says prove your targets before spending GPU time; this is the same rule one step later, and easier to skip. A ~7h LoRA run was built assuming the adapter could be hot-swapped onto a quantized base at serve time. The sizing doc flagged serving as unsettled and said the requirement was needed "while he is early, not after the run" — the concern was identified correctly and then the check was deferred. Tested afterwards, vLLM refuses outright: gemma4's model class implements zero occurrences of get_expert_mapping, which process_packed_modules_mapping requires for any MoE model. One grep, available months earlier. Two generalisations recorded: - Feature support is per-architecture, not per-family. LoRA works for the DENSE sibling of this same model family and not the MoE one, so "model X is supported" says nothing about X's variants. - A capability gap in the serving engine cannot be worked around from the training side. The adapter here never touched experts and was refused anyway, because the refusal keys on the model being MoE, not on what the adapter targets. Includes the mechanical check: grep the engine's model class for the capability, then start the engine with the feature flag alone — no adapter required, since --enable-lora forces the machinery to initialise and that is where it fails. The recovery is cheap here (merge, ~35 min per tune). The cost of finding out late is that it forecloses an architecture choice after the training window has already been spent.
This commit is contained in:
@@ -268,6 +268,52 @@ separately rather than folding it into an intensity story.
|
||||
|
||||
---
|
||||
|
||||
## 3.10 ⭐⭐ Prove the SERVING path before you spend the training window
|
||||
|
||||
Playbook-for-quants §4.1 says prove the quantization targets before spending
|
||||
GPU time. The same rule applies one step later and is easier to skip: **prove
|
||||
you can serve the artifact, in the shape you intend to serve it, before you
|
||||
train it.**
|
||||
|
||||
Worked failure, 2026-08-25. A ~7-hour LoRA run was built on the assumption that
|
||||
the adapter could be hot-swapped onto a quantized base at serve time. The
|
||||
sizing doc had flagged this correctly — *"serving the result is not settled…
|
||||
if it still no-ops, the harness must emit merged weights, and Eitri needs that
|
||||
requirement while he is early, not after the run"* — and then the check was
|
||||
deferred rather than run. Tested after the fact:
|
||||
|
||||
AttributeError: To support LoRA for MoE model,
|
||||
'get_expert_mapping' must be implemented
|
||||
|
||||
**One grep would have found it.** `vllm/lora/utils.py::process_packed_modules_mapping`
|
||||
branches on `is_moe_model()`, and the model class in question implements zero
|
||||
occurrences of `get_expert_mapping`. Static fact about the serving stack,
|
||||
available months before the run.
|
||||
|
||||
The check is cheap and mechanical:
|
||||
|
||||
```bash
|
||||
# does the serving engine's model class support what you plan to do?
|
||||
grep -c "SupportsLoRA\|get_expert_mapping" <engine>/model_executor/models/<arch>.py
|
||||
# and: start the engine with the feature flag ONLY (no adapter needed).
|
||||
# --enable-lora alone forces the machinery to initialise, which is where it fails.
|
||||
```
|
||||
|
||||
Two generalisations worth carrying:
|
||||
|
||||
- **Feature support is per-architecture, not per-family.** LoRA worked for the
|
||||
dense sibling of this exact model family and not for the MoE one. "Model X is
|
||||
supported" is not a statement about X's variants.
|
||||
- **A capability gap in the serving engine is not fixable by the training
|
||||
side.** No harness change, no quantization choice, and no adapter scoping
|
||||
works around it — the adapter here never touched experts and was refused
|
||||
anyway, because the refusal keys on the *model* being MoE.
|
||||
|
||||
The recovery is usually fine (merge instead of hot-swap, at ~35 min per tune).
|
||||
The point is that it should be a *decision* made before the window, not a
|
||||
discovery made after — because the alternative it forecloses may be an
|
||||
architecture choice, and by then you have already trained.
|
||||
|
||||
## 4. Panel / consult discipline for perf work
|
||||
|
||||
Perf investigations are unusually good at generating confident wrong answers,
|
||||
|
||||
Reference in New Issue
Block a user