feat(erp-tune): NVFP4A16 serving pipeline, and the MoE landmine it uncovered
Merge + quantize path for turning the Gemma-4 26B-A4B ERP/RP LoRA into a
servable NVFP4A16 seat, plus a playbook entry for the defect found while
validating it.
The landmine (playbook §3.15): a `targets=["Linear"]` NVFP4 recipe silently
misses every MoE expert on this architecture. Gemma-4 stores each layer's 128
experts as two fused 3-D nn.Parameter tensors, not nn.Linear modules, so the
recipe resolves 205 of 427 modules and ZERO experts — 22.84 B params, 88.5% of
the model, left in BF16 with no warning. This is the same blind spot that
killed QLoRA here via bitsandbytes; the tool changed, the checkpoint layout did
not.
before linearize_moe: 427 Linears, 205 targeted, experts 0
after linearize_moe: 11,947 Linears, 11,725 targeted, experts 11,520
(30 layers x 128 experts x 3 projections)
llmcompressor's linearize_moe unfuses them; no registration needed because
Gemma-4 satisfies FusedExpertsProtocol structurally. Caught by an §4.1 dry run
that asserts the expert count before any GPU spend, which is now the documented
requirement rather than an optional step.
Scheme is NVFP4A16, deviating from the playbook's mixed-W4A4 default on
measured grounds: brokkr-smithy-dev benched the W4A4 quant of this checkpoint
at 12% on contradiction detection with CoT off against gen's 81%, the signature
of 4-bit input activations on a reasoning-dense task, and W4A4 KLD degrades
2-4x past ~10k ctx on sm_120. This is a 16,384-ctx RP seat. Marlin's prefill
cost is accepted.
Two further silent-failure guards, both from prior hard-won lessons:
- the merged model ships the UPSTREAM chat template, not the trainee base's
stale 365-line one, because training rendered through upstream and the
mismatch would present as a tuning failure
- calibration reads the run's own encode cache rather than re-tokenizing, which
sidesteps §3.14 (a fast tokenizer mutated by truncation=True and persisted by
save_pretrained clamps every prompt forever)
Merge-then-quantize rather than LoRA hot-swap, since hot-swap onto NVFP4 was a
silent no-op on vLLM 0.24.0 (#47639). merge_lora.py asserts sampled target
weights actually changed, so an inert adapter cannot ship as a tune.
This commit is contained in:
@@ -226,6 +226,54 @@ bug — it is architectural). The proper upstream fix (vllm#51113) is in `main`
|
||||
Two cross-frontier peers (dvalin/bil-smithy) confirmed the bug class and pointed
|
||||
at the open symptom-twin issue #47087.
|
||||
|
||||
### 3.15 ⭐⭐ Fused 3-D MoE experts are INVISIBLE to a `targets=["Linear"]` recipe
|
||||
|
||||
**Symptom: none.** The quant completes, the artifact loads, and 88.5% of the
|
||||
model is still BF16. Nothing warns you.
|
||||
|
||||
Modern MoE checkpoints store each layer's experts as **two fused 3-D
|
||||
`nn.Parameter` tensors**, not as N `nn.Linear` modules. Gemma-4 26B-A4B:
|
||||
|
||||
model.language_model.layers.N.experts.gate_up_proj BF16 [128, 1408, 2816]
|
||||
model.language_model.layers.N.experts.down_proj BF16 [128, 2816, 704]
|
||||
|
||||
Note the **absent `.weight` suffix** — that is the tell. `mlp.down_proj.weight`
|
||||
is an `nn.Linear`; `experts.down_proj` is a bare parameter.
|
||||
|
||||
Measured on that checkpoint, recipe targeting `["Linear"]`:
|
||||
|
||||
Linear modules 427
|
||||
WILL quantize 205 (experts: 0) <- 22.84 B params untouched
|
||||
|
||||
**This is the same defect that killed QLoRA on this architecture** —
|
||||
`bitsandbytes` 4-bit replacement also walks `nn.Linear` modules and also
|
||||
silently skipped the experts. Two different tools, one blind spot, because the
|
||||
blind spot is in the *checkpoint layout*, not the tool.
|
||||
|
||||
**The fix** (llm-compressor ≥ 0.12):
|
||||
|
||||
```python
|
||||
from llmcompressor.modeling.moe.linearize import linearize_moe
|
||||
model = SomeForConditionalGeneration.from_pretrained(...)
|
||||
linearize_moe(model) # BEFORE building the recipe
|
||||
```
|
||||
|
||||
Linear modules 11947
|
||||
WILL quantize 11725 (experts: 11520) # 30 layers x 128 x 3 proj
|
||||
|
||||
`linearize_moe` unfuses the 3-D parameters into per-expert
|
||||
`experts.N.{gate,up,down}_proj` Linears. **No registration is needed** if the
|
||||
module satisfies `FusedExpertsProtocol` structurally — bare `down_proj` plus
|
||||
`gate_up_proj`/`up_proj` Parameters. `load_quantizable_moe(model_cls)` is the
|
||||
faster variant that linearizes during load rather than after.
|
||||
|
||||
**Always assert the expert count before spending GPU time** (§4.1). The
|
||||
arithmetic is `layers × experts × projections`; if your target list does not
|
||||
hit it exactly, the recipe is wrong and the failure is silent.
|
||||
|
||||
⚠ **Keep routers in `ignore`.** A 4-bit router picks *different experts* — that
|
||||
error does not average out downstream, it changes which weights run at all.
|
||||
|
||||
### 3.4 Toolchain version deadlocks
|
||||
|
||||
Both directions have burned us, so the resolution is: **use llm-compressor / compressed-tensors,
|
||||
|
||||
Reference in New Issue
Block a user