vh
6a8582936e
Merge + quantize path for turning the Gemma-4 26B-A4B ERP/RP LoRA into a
servable NVFP4A16 seat, plus a playbook entry for the defect found while
validating it.
The landmine (playbook §3.15): a `targets=["Linear"]` NVFP4 recipe silently
misses every MoE expert on this architecture. Gemma-4 stores each layer's 128
experts as two fused 3-D nn.Parameter tensors, not nn.Linear modules, so the
recipe resolves 205 of 427 modules and ZERO experts — 22.84 B params, 88.5% of
the model, left in BF16 with no warning. This is the same blind spot that
killed QLoRA here via bitsandbytes; the tool changed, the checkpoint layout did
not.
before linearize_moe: 427 Linears, 205 targeted, experts 0
after linearize_moe: 11,947 Linears, 11,725 targeted, experts 11,520
(30 layers x 128 experts x 3 projections)
llmcompressor's linearize_moe unfuses them; no registration needed because
Gemma-4 satisfies FusedExpertsProtocol structurally. Caught by an §4.1 dry run
that asserts the expert count before any GPU spend, which is now the documented
requirement rather than an optional step.
Scheme is NVFP4A16, deviating from the playbook's mixed-W4A4 default on
measured grounds: brokkr-smithy-dev benched the W4A4 quant of this checkpoint
at 12% on contradiction detection with CoT off against gen's 81%, the signature
of 4-bit input activations on a reasoning-dense task, and W4A4 KLD degrades
2-4x past ~10k ctx on sm_120. This is a 16,384-ctx RP seat. Marlin's prefill
cost is accepted.
Two further silent-failure guards, both from prior hard-won lessons:
- the merged model ships the UPSTREAM chat template, not the trainee base's
stale 365-line one, because training rendered through upstream and the
mismatch would present as a tuning failure
- calibration reads the run's own encode cache rather than re-tokenizing, which
sidesteps §3.14 (a fast tokenizer mutated by truncation=True and persisted by
save_pretrained clamps every prompt forever)
Merge-then-quantize rather than LoRA hot-swap, since hot-swap onto NVFP4 was a
silent no-op on vLLM 0.24.0 (#47639). merge_lora.py asserts sampled target
weights actually changed, so an inert adapter cannot ship as a tune.
81 lines
3.8 KiB
Markdown
81 lines
3.8 KiB
Markdown
# ERP/RP tune → served NVFP4 seat
|
||
|
||
Pipeline for turning the Gemma-4 26B-A4B ERP/RP LoRA into a servable NVFP4A16
|
||
model on `ana-ml2`. Written 2026-08-24 alongside round 2 of the tune.
|
||
|
||
Live copies run from `/tank/erp-tune/serve/` on ana-ml2. Model-agnostic quant
|
||
lessons belong in
|
||
[`docs/pfi/model-quantization-playbook.md`](../../docs/pfi/model-quantization-playbook.md);
|
||
the Gemma-4-specific ones are in
|
||
[`docs/pfi/gemma4-erp-tune-sizing.md`](../../docs/pfi/gemma4-erp-tune-sizing.md).
|
||
|
||
## Order
|
||
|
||
```bash
|
||
Q=/tank/aimodels/quant-work/.venv/bin/python # llmcompressor 0.12, ct 0.17.1
|
||
|
||
# 1. merge the adapter into bf16 (CPU, ~48 GB RAM, no GPU)
|
||
$Q merge_lora.py \
|
||
--base /tank/aimodels/gemma4-26b-a4b-it-heretic-bf16 \
|
||
--adapter /tank/erp-tune/run-01/adapter \
|
||
--out /tank/erp-tune/serve/merged-bf16
|
||
|
||
# 2. PROVE the target set before spending GPU time
|
||
$Q quant_nvfp4a16.py --model /tank/erp-tune/serve/merged-bf16 \
|
||
--out /tmp/x --calib-cache <encode-cache>.jsonl --dry-run
|
||
|
||
# 3. quantize
|
||
$Q quant_nvfp4a16.py --model /tank/erp-tune/serve/merged-bf16 \
|
||
--out /tank/erp-tune/serve/nvfp4a16 \
|
||
--calib-cache /tank/erp-tune/run-01/encode-cache/encoded-*.jsonl
|
||
```
|
||
|
||
## The three things that would silently ruin this
|
||
|
||
**1. `targets=["Linear"]` misses every MoE expert.** Gemma-4 stores 128 experts
|
||
per layer as two fused 3-D `nn.Parameter`s, so a Linear-targeting recipe hits
|
||
205 of 427 modules and **zero** experts — 22.84 B params stay BF16 and nothing
|
||
warns you. `linearize_moe(model)` unfuses them (427 → 11,947 Linears, 11,520
|
||
expert targets). Same blind spot that killed QLoRA here via `bitsandbytes`.
|
||
Playbook §3.15. **The dry run exists to catch this; use it.**
|
||
|
||
**2. Shipping the base's own chat template is train/serve skew.** The trainee
|
||
base carries a *stale* 365-line `chat_template.jinja`; upstream's is 390. The
|
||
harness trained through upstream (config key `chat_template_path`), so the
|
||
merged model must ship upstream's. Wrong template presents as a tuning failure
|
||
with no error. `merge_lora.py` copies it explicitly and refuses if absent.
|
||
|
||
**3. Calibration bakes a truncation cap into the tokenizer.** Playbook §3.14 —
|
||
a fast tokenizer called with `truncation=True` mutates its Rust backend state
|
||
in place, and `save_pretrained` persists it, clamping every prompt forever.
|
||
Sidestepped here by calibrating on the run's **encode cache** (already-tokenized
|
||
records) so the tokenizer is never called with truncation at all. Both scripts
|
||
still assert `tokenizer.json` has no `truncation` block before declaring success.
|
||
|
||
## Why NVFP4**A16** and not the playbook's default mixed W4A4
|
||
|
||
Playbook §1 prefers mixed NVFP4-W4A4 + FP8. This seat deviates deliberately:
|
||
|
||
- brokkr-smithy-dev benched the W4A4 quant of this checkpoint at **12% on
|
||
contradiction detection with CoT off against gen's 81%**, while T1/T3/T4/T5
|
||
sat at 100%. Not general degradation — the signature of 4-bit *input
|
||
activations* on a reasoning-dense task.
|
||
- W4A4 KLD is 2–4× worse past ~10k ctx on sm_120; activation-quant noise
|
||
compounds with KV lookups.
|
||
- This is a 16,384-ctx RP seat. Long sessions **are** the workload.
|
||
|
||
Cost accepted: A16 forces the Marlin kernel, ~half the prefill of native FP4.
|
||
Decode is memory-bound and barely moves.
|
||
|
||
⚠ Several HF repos named `…-NVFP4A16` declare `input_activations num_bits 4` —
|
||
they are W4A4 wearing an A16 label. `quant_nvfp4a16.py` refuses if the emitted
|
||
config says `num_bits: 4`. Verify before substituting any upstream artifact.
|
||
|
||
## Merge, don't hot-swap
|
||
|
||
LoRA-on-NVFP4 hot-swap was a silent no-op on vLLM 0.24.0 (#47639, proven
|
||
quant-agnostic). Merging first means the quantizer sees ordinary bf16 weights
|
||
and the served artifact needs no adapter machinery. `merge_lora.py` asserts the
|
||
merge actually changed sampled target weights — a bit-identical merge would
|
||
otherwise ship the base model wearing the tune's name.
|