From ab980e93453f649595296aef23c92c678d61e684 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 24 Aug 2026 23:48:25 -0700 Subject: [PATCH] fix(erp-tune-serve): four defects the end-to-end dry run found, all silent Validated the full adapter -> merge -> NVFP4A16 -> serve pipeline against checkpoint-100 of the live run. It works, and it produced a served model generating coherent prose. Getting there surfaced four failures, none of which announced itself as the thing it actually was. 1. transformers 5.15 MIGRATES the config schema on save. It drops Gemma-4's `global_head_dim` / `num_global_key_value_heads` and writes `per_layer_config` instead. transformers 5.10 (what the llmcompressor venv pins) does not know the new key and resolves num_key_value_heads to None: TypeError: unsupported operand type(s) for //: 'int' and 'NoneType' Every working artifact on the box - bf16 base, served nvfp4 prod seat, nvfp4a16 build - uses the OLD schema. Merging changes weights, not architecture, so the merge now downgrades the schema and asserts the result. 2. llmcompressor cannot auto-init a processor for a multimodal checkpoint and dies with a message that names neither the model nor the cause. Calibration here is text-only, so the tokenizer is passed explicitly as `processor`. 3. save_pretrained writes tokenizer files only, so `processor_config.json` was never carried. vLLM then fails at startup with "Can't load feature extractor", which reads as a vision bug and is actually a missing-file bug. Both scripts now carry the base's auxiliary configs. 4. The quant needs more than the 32 GiB free on GPU1 alongside the resident seats. Rather than leave that to a caller, quant_with_gen_down.sh stops vllm-gen and restores it from a trap on EVERY exit path - crash, OOM, kill, or success - because the restore must not depend on the calling session surviving. Uses `docker start`, not `compose up`, so the container comes back with its exact original config. Measured window: ~15 min, gen healthy after. Verified on the resulting artifact: merge 410 adapter tensors, sampled target weights confirmed CHANGED, upstream 390-line chat template shipped (not the base's stale 365) quant 49 GB -> 17 GB, format nvfp4-pack-quantized, a=null (genuine A16), weight_packed 11,725 of which 11,520 expert = 30 x 128 x 3, tokenizer truncation clean serve Marlin NVFP4 kernel + Marlin MoE backend, 40,492-token KV cache, coherent generation with content correctly populated One quality note: the reference nvfp4a16 artifact triggers a vLLM warning that parallel layers (q/k/v) carry different weight global scales, "likely to result in reduced accuracy". Our build does not - llmcompressor 0.12 links weight observers across fused groups for a shared global_scale automatically. The in-house quant is better than the downloaded one on that axis. Separately: the lora_B inert-adapter gate PASSED on checkpoint-100 - 205/205 non-zero, median norm 0.829, zero vision_tower tensors. That check never ran in round 1, and it is the only failure mode that stays invisible until the acceptance gate reports base-identical numbers. --- scripts/erp-tune-serve/merge_lora.py | 44 +++++++++++++++ scripts/erp-tune-serve/quant_nvfp4a16.py | 20 +++++-- scripts/erp-tune-serve/quant_with_gen_down.sh | 53 +++++++++++++++++++ 3 files changed, 112 insertions(+), 5 deletions(-) create mode 100644 scripts/erp-tune-serve/quant_with_gen_down.sh diff --git a/scripts/erp-tune-serve/merge_lora.py b/scripts/erp-tune-serve/merge_lora.py index 7c77a99..ea68c6b 100755 --- a/scripts/erp-tune-serve/merge_lora.py +++ b/scripts/erp-tune-serve/merge_lora.py @@ -97,6 +97,50 @@ def main() -> int: n_lines = len(src.read_text().splitlines()) print(f"[merge] chat_template.jinja <- {src} ({n_lines} lines)", flush=True) + # ⚠ CARRY THE PROCESSOR FILES. This is a multimodal checkpoint, so vLLM + # builds a feature extractor at startup and dies without them: + # OSError: Can't load feature extractor for '' + # `save_pretrained` on the merged model writes tokenizer files only, so + # anything else the base ships as auxiliary config must be copied across. + # Verified against the served nvfp4a16 artifact, which carries exactly this. + for aux in ("processor_config.json", "preprocessor_config.json", + "video_preprocessor_config.json", "special_tokens_map.json"): + src_aux = Path(a.base) / aux + if src_aux.exists() and not (out / aux).exists(): + shutil.copy2(src_aux, out / aux) + print(f"[merge] carried {aux}", flush=True) + + # ⚠⚠ CONFIG SCHEMA DOWNGRADE. transformers 5.15 MIGRATES Gemma-4's + # heterogeneous-attention config on save: it drops `global_head_dim` / + # `num_global_key_value_heads` and writes a `per_layer_config` dict instead. + # Older transformers (5.10, which is what the llmcompressor venv pins) does + # not understand the new key and resolves `config.num_key_value_heads` to + # None, dying with: + # TypeError: unsupported operand type(s) for //: 'int' and 'NoneType' + # Every WORKING artifact on this box - the bf16 base, the served nvfp4 prod + # seat, and the nvfp4a16 build - uses the OLD schema. Merging a LoRA changes + # weights, not architecture, so the base's expression of the architecture is + # the correct one to ship. + cfg_path = out / "config.json" + cfg = json.loads(cfg_path.read_text()) + base_cfg = json.loads((Path(a.base) / "config.json").read_text()) + ct, bt = cfg.get("text_config", cfg), base_cfg.get("text_config", base_cfg) + if "per_layer_config" in ct: + ct.pop("per_layer_config") + for k in ("global_head_dim", "num_global_key_value_heads"): + if k in bt: + ct[k] = bt[k] + cfg_path.write_text(json.dumps(cfg, indent=2) + "\n") + print("[merge] config schema downgraded to match the base " + "(per_layer_config -> global_head_dim/num_global_key_value_heads)", + flush=True) + ct2 = json.loads(cfg_path.read_text()).get("text_config", {}) + for k in ("global_head_dim", "num_global_key_value_heads"): + if bt.get(k) is not None and ct2.get(k) != bt.get(k): + print(f"REFUSING: {k} is {ct2.get(k)}, base says {bt.get(k)}", + file=sys.stderr) + return 6 + tj = out / "tokenizer.json" if tj.exists() and json.loads(tj.read_text()).get("truncation"): print("REFUSING: shipped tokenizer carries a truncation cap", file=sys.stderr) diff --git a/scripts/erp-tune-serve/quant_nvfp4a16.py b/scripts/erp-tune-serve/quant_nvfp4a16.py index ff56014..64bb361 100755 --- a/scripts/erp-tune-serve/quant_nvfp4a16.py +++ b/scripts/erp-tune-serve/quant_nvfp4a16.py @@ -146,8 +146,14 @@ def main() -> int: ) print("[oneshot] starting", flush=True) + # ⚠ `processor` must be passed EXPLICITLY. This is a multimodal + # (Gemma4ForConditionalGeneration) checkpoint, and llmcompressor's + # auto-init fails on it with "An error occurred when attempting to + # initialize model processor, which is required when a dataset is + # provided." Calibration here is text-only - the records come from the + # training encode cache - so the tokenizer is the correct processor. oneshot( - model=model, dataset=ds, recipe=recipe, + model=model, dataset=ds, recipe=recipe, processor=tok, max_seq_length=a.seqlen, num_calibration_samples=len(ds), output_dir=str(out), ) @@ -155,10 +161,14 @@ def main() -> int: # ⚠ playbook 3.14 - NEVER ship the calibration tokenizer. Re-read pristine. AutoTokenizer.from_pretrained(a.model, trust_remote_code=True).save_pretrained(out) - src_tpl = Path(a.model) / "chat_template.jinja" - if src_tpl.exists(): - (out / "chat_template.jinja").write_text(src_tpl.read_text()) - print("[post] chat_template.jinja carried over", flush=True) + import shutil as _sh + for aux in ("chat_template.jinja", "processor_config.json", + "preprocessor_config.json", "video_preprocessor_config.json", + "special_tokens_map.json", "generation_config.json"): + src_aux = Path(a.model) / aux + if src_aux.exists(): + _sh.copy2(src_aux, out / aux) + print(f"[post] carried {aux}", flush=True) tj = out / "tokenizer.json" if tj.exists() and json.loads(tj.read_text()).get("truncation"): diff --git a/scripts/erp-tune-serve/quant_with_gen_down.sh b/scripts/erp-tune-serve/quant_with_gen_down.sh new file mode 100644 index 0000000..4707cfa --- /dev/null +++ b/scripts/erp-tune-serve/quant_with_gen_down.sh @@ -0,0 +1,53 @@ +#!/usr/bin/env bash +# Run the NVFP4A16 quant with `vllm-gen` temporarily stopped to free GPU1, +# and ALWAYS bring gen back - crash, OOM, kill, or success. +# +# Operator authorised downing GPU1 residents overnight (2026-08-25) on the +# condition they are restored. The restore therefore must NOT depend on the +# calling session surviving, so it lives in a trap rather than in the caller. +# +# `docker start` (not `compose up`) is deliberate: it restarts the EXISTING +# container with its exact original config, so there is no chance of compose +# recreating the seat with drifted settings or a different image tag. +# +# gen serves `qwen3.8-27b-uncensored` + `-thinking`, and is the backing seat +# for the fleet-wide `summarizer` / `classifier` aliases. Keep the window short. +set -uo pipefail + +GEN=vllm-gen +LOG=/tank/erp-tune/serve/quant.log +OUT=/tank/erp-tune/serve/nvfp4a16-test + +restore() { + echo "[restore] starting $GEN ..." | tee -a "$LOG" + docker start "$GEN" >/dev/null 2>&1 + for i in $(seq 1 60); do + st=$(docker inspect -f '{{.State.Health.Status}}' "$GEN" 2>/dev/null || echo unknown) + run=$(docker inspect -f '{{.State.Running}}' "$GEN" 2>/dev/null || echo false) + if [ "$st" = "healthy" ]; then echo "[restore] $GEN healthy" | tee -a "$LOG"; return 0; fi + if [ "$run" != "true" ] && [ "$i" -gt 3 ]; then + echo "[restore] ⚠ $GEN NOT RUNNING - MANUAL ACTION NEEDED" | tee -a "$LOG"; return 1 + fi + sleep 10 + done + echo "[restore] ⚠ $GEN started but not healthy after 600s - CHECK IT" | tee -a "$LOG" + return 1 +} +trap restore EXIT INT TERM + +echo "[gen] stopping $GEN to free GPU1" | tee -a "$LOG" +docker stop "$GEN" >/dev/null 2>&1 +sleep 8 +nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv,noheader | tee -a "$LOG" + +rm -rf "$OUT" +cd /tank/erp-tune/serve +CUDA_VISIBLE_DEVICES=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ + /tank/aimodels/quant-work/.venv/bin/python -u quant_nvfp4a16.py \ + --model /tank/erp-tune/serve/merged-test \ + --out "$OUT" \ + --calib-cache /tank/erp-tune/run-01/encode-cache/encoded-a4b0796de1260930.jsonl \ + --num-calib "${NUM_CALIB:-16}" --seqlen "${SEQLEN:-4096}" >> "$LOG" 2>&1 +rc=$? +echo "[quant] exit rc=$rc" | tee -a "$LOG" +exit $rc