diff --git a/docs/pfi/training-throughput-playbook.md b/docs/pfi/training-throughput-playbook.md index c6f1e0e..c96ff58 100644 --- a/docs/pfi/training-throughput-playbook.md +++ b/docs/pfi/training-throughput-playbook.md @@ -314,6 +314,64 @@ The point is that it should be a *decision* made before the window, not a discovery made after — because the alternative it forecloses may be an architecture choice, and by then you have already trained. +## 3.11 Base-viability pre-flight — three greps, before you pick + +Run this on any candidate base BEFORE committing a training window. Each check +is minutes; skipping them cost a night in 2026-08. + +**1. Does it fit for TRAINING?** BF16 weights on one card, with room for the +real peak — not the weight figure. + + ana-ml2 reference: Gemma-4 26B-A4B is 48.1 GiB of weights and peaks at + 79.7 GiB at micro-batch 2 / seq 16,384. So ~48 GB of weights is close to + the practical ceiling for a 97.9 GiB card at that shape. + +⚠ Model-line names lie about size. "Mistral **Small** 4" is 119 B — 238 GB in +BF16, more than both cards combined. Read `params.json` / `config.json`, never +the name. + +⚠ QLoRA is NOT an escape hatch for MoE. `bitsandbytes` walks `nn.Linear`, and +fused 3-D expert parameters are not that — see quantization playbook §3.15. + +**2. If MoE — does the serving engine implement the expert mapping?** + +```bash +grep -c "def get_expert_mapping" /model_executor/models/.py +``` + +Zero means **LoRA cannot be served at all** and merged weights are mandatory. +Measured: `gemma4*.py` → 0 (refuses); `deepseek_v2.py`, `mixtral.py`, +`glm4_moe.py`, `ernie45_moe.py` → present. + +**3. Does the model class support LoRA?** ⚠ **Grep the class, not the file** — +capability is usually INHERITED and a file-level grep misses it entirely: + +```python +from vllm.model_executor.models. import as C +print([c.__name__ for c in C.__mro__]) +print(hasattr(C, "get_expert_mapping"), getattr(C, "supports_lora", None)) +``` + +`mistral.py` greps as `SupportsLoRA=0` and is fully LoRA-capable — it inherits +from `LlamaForCausalLM`. `mistral_large_3.py` greps as 0 for both and inherits +`get_expert_mapping` from `DeepseekV3ForCausalLM`. Both file greps are wrong; +only MRO resolution is right. (Same failure as asserting a substring instead of +an effective value.) + +**Worked results, 2026-08-25:** + +| base | fits (1) | MoE mapping (2) | LoRA (3) | verdict | +|---|---|---|---|---| +| Gemma-4 26B-A4B | ✅ 48 GB | ❌ absent | n/a | trainable, **merge-only** | +| Mistral Small 4 119B | ❌ 238 GB | ✅ via DeepSeek-V3 | ✅ | servable w/ hot-swap, **not trainable here** | +| Ministral 3 14B | ✅ ~28 GB | n/a (dense) | ✅ inherited | **passes all three** | + +**Architecture shape is worth a fourth glance**, because it predicts how much +of this playbook you will need. Uniform `head_dim` ≤ 128 with no sliding window +means flash AND cuDNN are both reachable and §3.1/§3.3 simply do not apply. +Mixed head dims plus a sliding window — Gemma-4's shape — is what forces dense +O(n²) attention on Ampere-generation kernels and costs 65% of the step. + ## 4. Panel / consult discipline for perf work Perf investigations are unusually good at generating confident wrong answers,