vh
ab980e9345
Validated the full adapter -> merge -> NVFP4A16 -> serve pipeline against
checkpoint-100 of the live run. It works, and it produced a served model
generating coherent prose. Getting there surfaced four failures, none of which
announced itself as the thing it actually was.
1. transformers 5.15 MIGRATES the config schema on save. It drops Gemma-4's
`global_head_dim` / `num_global_key_value_heads` and writes `per_layer_config`
instead. transformers 5.10 (what the llmcompressor venv pins) does not know
the new key and resolves num_key_value_heads to None:
TypeError: unsupported operand type(s) for //: 'int' and 'NoneType'
Every working artifact on the box - bf16 base, served nvfp4 prod seat,
nvfp4a16 build - uses the OLD schema. Merging changes weights, not
architecture, so the merge now downgrades the schema and asserts the result.
2. llmcompressor cannot auto-init a processor for a multimodal checkpoint and
dies with a message that names neither the model nor the cause. Calibration
here is text-only, so the tokenizer is passed explicitly as `processor`.
3. save_pretrained writes tokenizer files only, so `processor_config.json` was
never carried. vLLM then fails at startup with "Can't load feature extractor",
which reads as a vision bug and is actually a missing-file bug. Both scripts
now carry the base's auxiliary configs.
4. The quant needs more than the 32 GiB free on GPU1 alongside the resident
seats. Rather than leave that to a caller, quant_with_gen_down.sh stops
vllm-gen and restores it from a trap on EVERY exit path - crash, OOM, kill,
or success - because the restore must not depend on the calling session
surviving. Uses `docker start`, not `compose up`, so the container comes back
with its exact original config. Measured window: ~15 min, gen healthy after.
Verified on the resulting artifact:
merge 410 adapter tensors, sampled target weights confirmed CHANGED,
upstream 390-line chat template shipped (not the base's stale 365)
quant 49 GB -> 17 GB, format nvfp4-pack-quantized, a=null (genuine A16),
weight_packed 11,725 of which 11,520 expert = 30 x 128 x 3,
tokenizer truncation clean
serve Marlin NVFP4 kernel + Marlin MoE backend, 40,492-token KV cache,
coherent generation with content correctly populated
One quality note: the reference nvfp4a16 artifact triggers a vLLM warning that
parallel layers (q/k/v) carry different weight global scales, "likely to result
in reduced accuracy". Our build does not - llmcompressor 0.12 links weight
observers across fused groups for a shared global_scale automatically. The
in-house quant is better than the downloaded one on that axis.
Separately: the lora_B inert-adapter gate PASSED on checkpoint-100 - 205/205
non-zero, median norm 0.829, zero vision_tower tensors. That check never ran in
round 1, and it is the only failure mode that stays invisible until the
acceptance gate reports base-identical numbers.
ERP/RP tune → served NVFP4 seat
Pipeline for turning the Gemma-4 26B-A4B ERP/RP LoRA into a servable NVFP4A16
model on ana-ml2. Written 2026-08-24 alongside round 2 of the tune.
Live copies run from /tank/erp-tune/serve/ on ana-ml2. Model-agnostic quant
lessons belong in
docs/pfi/model-quantization-playbook.md;
the Gemma-4-specific ones are in
docs/pfi/gemma4-erp-tune-sizing.md.
Order
Q=/tank/aimodels/quant-work/.venv/bin/python # llmcompressor 0.12, ct 0.17.1
# 1. merge the adapter into bf16 (CPU, ~48 GB RAM, no GPU)
$Q merge_lora.py \
--base /tank/aimodels/gemma4-26b-a4b-it-heretic-bf16 \
--adapter /tank/erp-tune/run-01/adapter \
--out /tank/erp-tune/serve/merged-bf16
# 2. PROVE the target set before spending GPU time
$Q quant_nvfp4a16.py --model /tank/erp-tune/serve/merged-bf16 \
--out /tmp/x --calib-cache <encode-cache>.jsonl --dry-run
# 3. quantize
$Q quant_nvfp4a16.py --model /tank/erp-tune/serve/merged-bf16 \
--out /tank/erp-tune/serve/nvfp4a16 \
--calib-cache /tank/erp-tune/run-01/encode-cache/encoded-*.jsonl
The three things that would silently ruin this
1. targets=["Linear"] misses every MoE expert. Gemma-4 stores 128 experts
per layer as two fused 3-D nn.Parameters, so a Linear-targeting recipe hits
205 of 427 modules and zero experts — 22.84 B params stay BF16 and nothing
warns you. linearize_moe(model) unfuses them (427 → 11,947 Linears, 11,520
expert targets). Same blind spot that killed QLoRA here via bitsandbytes.
Playbook §3.15. The dry run exists to catch this; use it.
2. Shipping the base's own chat template is train/serve skew. The trainee
base carries a stale 365-line chat_template.jinja; upstream's is 390. The
harness trained through upstream (config key chat_template_path), so the
merged model must ship upstream's. Wrong template presents as a tuning failure
with no error. merge_lora.py copies it explicitly and refuses if absent.
3. Calibration bakes a truncation cap into the tokenizer. Playbook §3.14 —
a fast tokenizer called with truncation=True mutates its Rust backend state
in place, and save_pretrained persists it, clamping every prompt forever.
Sidestepped here by calibrating on the run's encode cache (already-tokenized
records) so the tokenizer is never called with truncation at all. Both scripts
still assert tokenizer.json has no truncation block before declaring success.
Why NVFP4A16 and not the playbook's default mixed W4A4
Playbook §1 prefers mixed NVFP4-W4A4 + FP8. This seat deviates deliberately:
- brokkr-smithy-dev benched the W4A4 quant of this checkpoint at 12% on contradiction detection with CoT off against gen's 81%, while T1/T3/T4/T5 sat at 100%. Not general degradation — the signature of 4-bit input activations on a reasoning-dense task.
- W4A4 KLD is 2–4× worse past ~10k ctx on sm_120; activation-quant noise compounds with KV lookups.
- This is a 16,384-ctx RP seat. Long sessions are the workload.
Cost accepted: A16 forces the Marlin kernel, ~half the prefill of native FP4. Decode is memory-bound and barely moves.
⚠ Several HF repos named …-NVFP4A16 declare input_activations num_bits 4 —
they are W4A4 wearing an A16 label. quant_nvfp4a16.py refuses if the emitted
config says num_bits: 4. Verify before substituting any upstream artifact.
Merge, don't hot-swap
LoRA-on-NVFP4 hot-swap was a silent no-op on vLLM 0.24.0 (#47639, proven
quant-agnostic). Merging first means the quantizer sees ordinary bf16 weights
and the served artifact needs no adapter machinery. merge_lora.py asserts the
merge actually changed sampled target weights — a bit-identical merge would
otherwise ship the base model wearing the tune's name.