docs(training-playbook): prove the serving path before spending the window

§3.10. The quantization playbook already says prove your targets before
spending GPU time; this is the same rule one step later, and easier to skip.

A ~7h LoRA run was built assuming the adapter could be hot-swapped onto a
quantized base at serve time. The sizing doc flagged serving as unsettled and
said the requirement was needed "while he is early, not after the run" — the
concern was identified correctly and then the check was deferred. Tested
afterwards, vLLM refuses outright: gemma4's model class implements zero
occurrences of get_expert_mapping, which process_packed_modules_mapping
requires for any MoE model. One grep, available months earlier.

Two generalisations recorded:

- Feature support is per-architecture, not per-family. LoRA works for the DENSE
  sibling of this same model family and not the MoE one, so "model X is
  supported" says nothing about X's variants.
- A capability gap in the serving engine cannot be worked around from the
  training side. The adapter here never touched experts and was refused anyway,
  because the refusal keys on the model being MoE, not on what the adapter
  targets.

Includes the mechanical check: grep the engine's model class for the capability,
then start the engine with the feature flag alone — no adapter required, since
--enable-lora forces the machinery to initialise and that is where it fails.

The recovery is cheap here (merge, ~35 min per tune). The cost of finding out
late is that it forecloses an architecture choice after the training window has
already been spent.
This commit is contained in:
vh
2026-08-25 01:36:01 -07:00
parent 8de5f7a73c
commit 96731bb090
+46
View File
@@ -268,6 +268,52 @@ separately rather than folding it into an intensity story.
---
## 3.10 ⭐⭐ Prove the SERVING path before you spend the training window
Playbook-for-quants §4.1 says prove the quantization targets before spending
GPU time. The same rule applies one step later and is easier to skip: **prove
you can serve the artifact, in the shape you intend to serve it, before you
train it.**
Worked failure, 2026-08-25. A ~7-hour LoRA run was built on the assumption that
the adapter could be hot-swapped onto a quantized base at serve time. The
sizing doc had flagged this correctly — *"serving the result is not settled…
if it still no-ops, the harness must emit merged weights, and Eitri needs that
requirement while he is early, not after the run"* — and then the check was
deferred rather than run. Tested after the fact:
AttributeError: To support LoRA for MoE model,
'get_expert_mapping' must be implemented
**One grep would have found it.** `vllm/lora/utils.py::process_packed_modules_mapping`
branches on `is_moe_model()`, and the model class in question implements zero
occurrences of `get_expert_mapping`. Static fact about the serving stack,
available months before the run.
The check is cheap and mechanical:
```bash
# does the serving engine's model class support what you plan to do?
grep -c "SupportsLoRA\|get_expert_mapping" <engine>/model_executor/models/<arch>.py
# and: start the engine with the feature flag ONLY (no adapter needed).
# --enable-lora alone forces the machinery to initialise, which is where it fails.
```
Two generalisations worth carrying:
- **Feature support is per-architecture, not per-family.** LoRA worked for the
dense sibling of this exact model family and not for the MoE one. "Model X is
supported" is not a statement about X's variants.
- **A capability gap in the serving engine is not fixable by the training
side.** No harness change, no quantization choice, and no adapter scoping
works around it — the adapter here never touched experts and was refused
anyway, because the refusal keys on the *model* being MoE.
The recovery is usually fine (merge instead of hot-swap, at ~35 min per tune).
The point is that it should be a *decision* made before the window, not a
discovery made after — because the alternative it forecloses may be an
architecture choice, and by then you have already trained.
## 4. Panel / consult discipline for perf work
Perf investigations are unusually good at generating confident wrong answers,