Files

T

vh 6a8582936e feat(erp-tune): NVFP4A16 serving pipeline, and the MoE landmine it uncovered

Merge + quantize path for turning the Gemma-4 26B-A4B ERP/RP LoRA into a
servable NVFP4A16 seat, plus a playbook entry for the defect found while
validating it.

The landmine (playbook §3.15): a `targets=["Linear"]` NVFP4 recipe silently
misses every MoE expert on this architecture. Gemma-4 stores each layer's 128
experts as two fused 3-D nn.Parameter tensors, not nn.Linear modules, so the
recipe resolves 205 of 427 modules and ZERO experts — 22.84 B params, 88.5% of
the model, left in BF16 with no warning. This is the same blind spot that
killed QLoRA here via bitsandbytes; the tool changed, the checkpoint layout did
not.

  before linearize_moe:  427 Linears,    205 targeted, experts 0
  after  linearize_moe:  11,947 Linears, 11,725 targeted, experts 11,520
                         (30 layers x 128 experts x 3 projections)

llmcompressor's linearize_moe unfuses them; no registration needed because
Gemma-4 satisfies FusedExpertsProtocol structurally. Caught by an §4.1 dry run
that asserts the expert count before any GPU spend, which is now the documented
requirement rather than an optional step.

Scheme is NVFP4A16, deviating from the playbook's mixed-W4A4 default on
measured grounds: brokkr-smithy-dev benched the W4A4 quant of this checkpoint
at 12% on contradiction detection with CoT off against gen's 81%, the signature
of 4-bit input activations on a reasoning-dense task, and W4A4 KLD degrades
2-4x past ~10k ctx on sm_120. This is a 16,384-ctx RP seat. Marlin's prefill
cost is accepted.

Two further silent-failure guards, both from prior hard-won lessons:
- the merged model ships the UPSTREAM chat template, not the trainee base's
  stale 365-line one, because training rendered through upstream and the
  mismatch would present as a tuning failure
- calibration reads the run's own encode cache rather than re-tokenizing, which
  sidesteps §3.14 (a fast tokenizer mutated by truncation=True and persisted by
  save_pretrained clamps every prompt forever)

Merge-then-quantize rather than LoRA hot-swap, since hot-swap onto NVFP4 was a
silent no-op on vLLM 0.24.0 (#47639). merge_lora.py asserts sampled target
weights actually changed, so an inert adapter cannot ship as a tune.

2026-08-24 23:11:33 -07:00

3.8 KiB

Raw Blame History

ERP/RP tune → served NVFP4 seat

Pipeline for turning the Gemma-4 26B-A4B ERP/RP LoRA into a servable NVFP4A16 model on ana-ml2. Written 2026-08-24 alongside round 2 of the tune.

Live copies run from /tank/erp-tune/serve/ on ana-ml2. Model-agnostic quant lessons belong in docs/pfi/model-quantization-playbook.md; the Gemma-4-specific ones are in docs/pfi/gemma4-erp-tune-sizing.md.

Order

Q=/tank/aimodels/quant-work/.venv/bin/python     # llmcompressor 0.12, ct 0.17.1

# 1. merge the adapter into bf16 (CPU, ~48 GB RAM, no GPU)
$Q merge_lora.py \
    --base    /tank/aimodels/gemma4-26b-a4b-it-heretic-bf16 \
    --adapter /tank/erp-tune/run-01/adapter \
    --out     /tank/erp-tune/serve/merged-bf16

# 2. PROVE the target set before spending GPU time
$Q quant_nvfp4a16.py --model /tank/erp-tune/serve/merged-bf16 \
    --out /tmp/x --calib-cache <encode-cache>.jsonl --dry-run

# 3. quantize
$Q quant_nvfp4a16.py --model /tank/erp-tune/serve/merged-bf16 \
    --out /tank/erp-tune/serve/nvfp4a16 \
    --calib-cache /tank/erp-tune/run-01/encode-cache/encoded-*.jsonl

The three things that would silently ruin this

1. targets=["Linear"] misses every MoE expert. Gemma-4 stores 128 experts per layer as two fused 3-D nn.Parameters, so a Linear-targeting recipe hits 205 of 427 modules and zero experts — 22.84 B params stay BF16 and nothing warns you. linearize_moe(model) unfuses them (427 → 11,947 Linears, 11,520 expert targets). Same blind spot that killed QLoRA here via bitsandbytes. Playbook §3.15. The dry run exists to catch this; use it.

2. Shipping the base's own chat template is train/serve skew. The trainee base carries a stale 365-line chat_template.jinja; upstream's is 390. The harness trained through upstream (config key chat_template_path), so the merged model must ship upstream's. Wrong template presents as a tuning failure with no error. merge_lora.py copies it explicitly and refuses if absent.

3. Calibration bakes a truncation cap into the tokenizer. Playbook §3.14 — a fast tokenizer called with truncation=True mutates its Rust backend state in place, and save_pretrained persists it, clamping every prompt forever. Sidestepped here by calibrating on the run's encode cache (already-tokenized records) so the tokenizer is never called with truncation at all. Both scripts still assert tokenizer.json has no truncation block before declaring success.

Why NVFP4A16 and not the playbook's default mixed W4A4

Playbook §1 prefers mixed NVFP4-W4A4 + FP8. This seat deviates deliberately:

brokkr-smithy-dev benched the W4A4 quant of this checkpoint at 12% on contradiction detection with CoT off against gen's 81%, while T1/T3/T4/T5 sat at 100%. Not general degradation — the signature of 4-bit input activations on a reasoning-dense task.
W4A4 KLD is 2–4× worse past ~10k ctx on sm_120; activation-quant noise compounds with KV lookups.
This is a 16,384-ctx RP seat. Long sessions are the workload.

Cost accepted: A16 forces the Marlin kernel, ~half the prefill of native FP4. Decode is memory-bound and barely moves.

⚠ Several HF repos named …-NVFP4A16 declare input_activations num_bits 4 — they are W4A4 wearing an A16 label. quant_nvfp4a16.py refuses if the emitted config says num_bits: 4. Verify before substituting any upstream artifact.

Merge, don't hot-swap

LoRA-on-NVFP4 hot-swap was a silent no-op on vLLM 0.24.0 (#47639, proven quant-agnostic). Merging first means the quantizer sees ordinary bf16 weights and the served artifact needs no adapter machinery. merge_lora.py asserts the merge actually changed sampled target weights — a bit-identical merge would otherwise ship the base model wearing the tune's name.

3.8 KiB Raw Blame History Unescape Escape