From 2ec8f422970deb802e19e33e2acb7e53e5fd350e Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 25 Aug 2026 01:51:06 -0700 Subject: [PATCH] docs(training-playbook): base-viability pre-flight, three greps before you pick MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §3.11. Three consecutive "what about X as a base?" questions in one session, each answerable in minutes, none of which had been asked before a 7-hour training window was committed. Writing the check down so it runs first. 1. does it fit for TRAINING - BF16 weights against the real measured peak, not the weight figure (Gemma-4 is 48.1 GiB of weights and peaks at 79.7 GiB at mb2/seq-16k). Model-line names lie: "Mistral Small 4" is 119 B, 238 GB in BF16, more than both cards combined. QLoRA is not an escape hatch for MoE - bitsandbytes walks nn.Linear and fused 3-D experts are not that. 2. if MoE - does the serving engine implement get_expert_mapping. Zero means LoRA cannot be served at all. gemma4*.py -> 0; deepseek_v2, mixtral, glm4_moe, ernie45_moe -> present. 3. does the model class support LoRA - and GREP THE CLASS, NOT THE FILE. Point 3 has teeth and I nearly got it wrong twice in one turn. mistral.py greps as SupportsLoRA=0 and is fully LoRA-capable via LlamaForCausalLM. mistral_large_3.py greps as 0 for both and inherits get_expert_mapping from DeepseekV3ForCausalLM. Capability is inherited; a file-level grep misses it and only MRO resolution answers it. Same class of error as asserting a substring instead of an effective value. Worked results recorded for the three candidates evaluated: Gemma-4 26B-A4B fits, no expert mapping -> trainable, MERGE-ONLY Mistral Small 4 119B 238 GB, has mapping -> servable, NOT trainable here Ministral 3 14B ~28 GB, dense, inherited -> passes all three Adds a fourth glance at architecture shape, since it predicts how much of this playbook applies at all: uniform head_dim <= 128 with no sliding window keeps both flash and cuDNN reachable and makes §3.1/§3.3 moot, while mixed head dims plus a sliding window is exactly what forces dense O(n^2) attention onto Ampere-generation kernels for 65% of the step. --- docs/pfi/training-throughput-playbook.md | 58 ++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/docs/pfi/training-throughput-playbook.md b/docs/pfi/training-throughput-playbook.md index c6f1e0e..c96ff58 100644 --- a/docs/pfi/training-throughput-playbook.md +++ b/docs/pfi/training-throughput-playbook.md @@ -314,6 +314,64 @@ The point is that it should be a *decision* made before the window, not a discovery made after — because the alternative it forecloses may be an architecture choice, and by then you have already trained. +## 3.11 Base-viability pre-flight — three greps, before you pick + +Run this on any candidate base BEFORE committing a training window. Each check +is minutes; skipping them cost a night in 2026-08. + +**1. Does it fit for TRAINING?** BF16 weights on one card, with room for the +real peak — not the weight figure. + + ana-ml2 reference: Gemma-4 26B-A4B is 48.1 GiB of weights and peaks at + 79.7 GiB at micro-batch 2 / seq 16,384. So ~48 GB of weights is close to + the practical ceiling for a 97.9 GiB card at that shape. + +⚠ Model-line names lie about size. "Mistral **Small** 4" is 119 B — 238 GB in +BF16, more than both cards combined. Read `params.json` / `config.json`, never +the name. + +⚠ QLoRA is NOT an escape hatch for MoE. `bitsandbytes` walks `nn.Linear`, and +fused 3-D expert parameters are not that — see quantization playbook §3.15. + +**2. If MoE — does the serving engine implement the expert mapping?** + +```bash +grep -c "def get_expert_mapping" /model_executor/models/.py +``` + +Zero means **LoRA cannot be served at all** and merged weights are mandatory. +Measured: `gemma4*.py` → 0 (refuses); `deepseek_v2.py`, `mixtral.py`, +`glm4_moe.py`, `ernie45_moe.py` → present. + +**3. Does the model class support LoRA?** ⚠ **Grep the class, not the file** — +capability is usually INHERITED and a file-level grep misses it entirely: + +```python +from vllm.model_executor.models. import as C +print([c.__name__ for c in C.__mro__]) +print(hasattr(C, "get_expert_mapping"), getattr(C, "supports_lora", None)) +``` + +`mistral.py` greps as `SupportsLoRA=0` and is fully LoRA-capable — it inherits +from `LlamaForCausalLM`. `mistral_large_3.py` greps as 0 for both and inherits +`get_expert_mapping` from `DeepseekV3ForCausalLM`. Both file greps are wrong; +only MRO resolution is right. (Same failure as asserting a substring instead of +an effective value.) + +**Worked results, 2026-08-25:** + +| base | fits (1) | MoE mapping (2) | LoRA (3) | verdict | +|---|---|---|---|---| +| Gemma-4 26B-A4B | ✅ 48 GB | ❌ absent | n/a | trainable, **merge-only** | +| Mistral Small 4 119B | ❌ 238 GB | ✅ via DeepSeek-V3 | ✅ | servable w/ hot-swap, **not trainable here** | +| Ministral 3 14B | ✅ ~28 GB | n/a (dense) | ✅ inherited | **passes all three** | + +**Architecture shape is worth a fourth glance**, because it predicts how much +of this playbook you will need. Uniform `head_dim` ≤ 128 with no sliding window +means flash AND cuDNN are both reachable and §3.1/§3.3 simply do not apply. +Mixed head dims plus a sliding window — Gemma-4's shape — is what forces dense +O(n²) attention on Ampere-generation kernels and costs 65% of the step. + ## 4. Panel / consult discipline for perf work Perf investigations are unusually good at generating confident wrong answers,