# ERP/RP tune → served NVFP4 seat

Pipeline for turning the Gemma-4 26B-A4B ERP/RP LoRA into a servable NVFP4A16
model on `ana-ml2`. Written 2026-08-24 alongside round 2 of the tune.

Live copies run from `/tank/erp-tune/serve/` on ana-ml2. Model-agnostic quant
lessons belong in
[`docs/pfi/model-quantization-playbook.md`](../../docs/pfi/model-quantization-playbook.md);
the Gemma-4-specific ones are in
[`docs/pfi/gemma4-erp-tune-sizing.md`](../../docs/pfi/gemma4-erp-tune-sizing.md).

## Order

```bash
Q=/tank/aimodels/quant-work/.venv/bin/python     # llmcompressor 0.12, ct 0.17.1

# 1. merge the adapter into bf16 (CPU, ~48 GB RAM, no GPU)
$Q merge_lora.py \
    --base    /tank/aimodels/gemma4-26b-a4b-it-heretic-bf16 \
    --adapter /tank/erp-tune/run-01/adapter \
    --out     /tank/erp-tune/serve/merged-bf16

# 2. PROVE the target set before spending GPU time
$Q quant_nvfp4a16.py --model /tank/erp-tune/serve/merged-bf16 \
    --out /tmp/x --calib-cache <encode-cache>.jsonl --dry-run

# 3. quantize
$Q quant_nvfp4a16.py --model /tank/erp-tune/serve/merged-bf16 \
    --out /tank/erp-tune/serve/nvfp4a16 \
    --calib-cache /tank/erp-tune/run-01/encode-cache/encoded-*.jsonl
```

## The three things that would silently ruin this

**1. `targets=["Linear"]` misses every MoE expert.** Gemma-4 stores 128 experts
per layer as two fused 3-D `nn.Parameter`s, so a Linear-targeting recipe hits
205 of 427 modules and **zero** experts — 22.84 B params stay BF16 and nothing
warns you. `linearize_moe(model)` unfuses them (427 → 11,947 Linears, 11,520
expert targets). Same blind spot that killed QLoRA here via `bitsandbytes`.
Playbook §3.15. **The dry run exists to catch this; use it.**

**2. Shipping the base's own chat template is train/serve skew.** The trainee
base carries a *stale* 365-line `chat_template.jinja`; upstream's is 390. The
harness trained through upstream (config key `chat_template_path`), so the
merged model must ship upstream's. Wrong template presents as a tuning failure
with no error. `merge_lora.py` copies it explicitly and refuses if absent.

**3. Calibration bakes a truncation cap into the tokenizer.** Playbook §3.14 —
a fast tokenizer called with `truncation=True` mutates its Rust backend state
in place, and `save_pretrained` persists it, clamping every prompt forever.
Sidestepped here by calibrating on the run's **encode cache** (already-tokenized
records) so the tokenizer is never called with truncation at all. Both scripts
still assert `tokenizer.json` has no `truncation` block before declaring success.

## Why NVFP4**A16** and not the playbook's default mixed W4A4

Playbook §1 prefers mixed NVFP4-W4A4 + FP8. This seat deviates deliberately:

- brokkr-smithy-dev benched the W4A4 quant of this checkpoint at **12% on
  contradiction detection with CoT off against gen's 81%**, while T1/T3/T4/T5
  sat at 100%. Not general degradation — the signature of 4-bit *input
  activations* on a reasoning-dense task.
- W4A4 KLD is 2–4× worse past ~10k ctx on sm_120; activation-quant noise
  compounds with KV lookups.
- This is a 16,384-ctx RP seat. Long sessions **are** the workload.

Cost accepted: A16 forces the Marlin kernel, ~half the prefill of native FP4.
Decode is memory-bound and barely moves.

⚠ Several HF repos named `…-NVFP4A16` declare `input_activations num_bits 4` —
they are W4A4 wearing an A16 label. `quant_nvfp4a16.py` refuses if the emitted
config says `num_bits: 4`. Verify before substituting any upstream artifact.

## Merge, don't hot-swap

LoRA-on-NVFP4 hot-swap was a silent no-op on vLLM 0.24.0 (#47639, proven
quant-agnostic). Merging first means the quantizer sees ordinary bf16 weights
and the served artifact needs no adapter machinery. `merge_lora.py` asserts the
merge actually changed sampled target weights — a bit-identical merge would
otherwise ship the base model wearing the tune's name.