From b94d1738877c70ad324dd0879e011e01cffc1257 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 14 Sep 2026 04:28:59 -0700 Subject: [PATCH] fix(cyberprev-seat): serve coherent v2 quant from name-repaired source; wire gateway The first cyberprev quant served pure gibberish despite passing every structural check. Root cause: the hotdogs/Qwen3.8-27B-abliterated-cyber-preview checkpoint ships broken -- 850 of 1199 tensors (the whole transformer body) carry a TRIPLED `language_model.` prefix, confirmed present in the upstream HF index (an unsloth export bug, not our download). transformers silently loaded the unmapped tensors as random-init, the pipeline quantized noise, and structural verification passed because it inspects the output config, never that input names mapped. A plain greedy coherence smoke caught it; nothing structural could. Fix: rewrite tensor names against a known-good reference of the same architecture (qwen38-27b-uncensored-bf16), verified the repaired name-set == reference name-set exactly (1199 tensors, 0 collisions), and re-quantized from the corrected source (cyberprev-bf16-fixed). v2 is coherence-verified (spec off AND live with dflash): Paris/Berlin/Rome, correct arithmetic precedence, primes, clean cyber-domain chat with a proper thinking split. dflash k=7 measured at 2.77 mean acceptance length. Seat serves cyberprev-27b / cyberprev-27b-thinking on :8025/GPU0 with dflash k=7, displacing the retired sentinel-r3. Gateway: removed the retired sentinel-r3 / sentinel-r3-reasoning aliases and added cyberprev-27b / cyberprev-reasoning -- NOT repointed, since serving cyberprev weights under sentinel's name is silent substitution. Verified end-to-end through the gateway; sentinel-r3 now 400s. Playbook gains a superseded-claims row: structural verification does not imply a good quant; a coherence smoke is the load-bearing gate, and third-party sources must have their tensor names diffed against a reference before quantizing. Broken v1 quant dir removed. --- docs/pfi/model-quantization-playbook.md | 1 + stacks/cyberprev-seat/.env.example | 10 +++++-- stacks/cyberprev-seat/compose.yaml | 2 +- stacks/litellm/conf/config.yaml | 38 +++++++++++++++++++++++++ 4 files changed, 47 insertions(+), 4 deletions(-) diff --git a/docs/pfi/model-quantization-playbook.md b/docs/pfi/model-quantization-playbook.md index 52cd2d3..f01c4b7 100644 --- a/docs/pfi/model-quantization-playbook.md +++ b/docs/pfi/model-quantization-playbook.md @@ -777,6 +777,7 @@ day if followed: | "NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer" | `reference_nvfp4_w4a4_granite_infeasible` | **NARROWED.** True for *uniform* W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs **with FP8 on attention and late layers** is fine and is the current default (§2). | | "The Qwen3.8 MTP corruption is fixed by disabling prefix caching while keeping MTP; the gen seat runs APC-off" | this playbook §3.7 (now marked), earlier auto-memory | **SUPERSEDED 2026-08-17, and the staleness was only caught 2026-08-26.** APC-off passed a synthetic 7-turn probe and the operator still saw severe degeneration in real use; reverted the same day. The real cause was the **AEON W4A4 quant being defective** (~15-20% degenerate generations), with MTP / prefix-caching / gateway merely AMPLIFYING it (§3.8). The gen seat runs **MTP ON and prefix-caching ON** on the in-house mixed NVFP4+FP8 build — verified against the live container 2026-08-26. ⚠ The lesson inside the lesson: a *passing multi-turn probe* was not sufficient evidence either. | | "transformers' Qwen3.5 DeltaNet linear-attention NaNs in bf16 without causal-conv1d; it is precision-driven cancellation and fp32 resolves it" | `services/coldfusion-abliteration/README.md`, `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md` | **SUPERSEDED 2026-08-20.** Precision was never the variable. The NaN came from **multi-GPU sharding** and **`expandable_segments`** (§3.9, §3.10); fp32 only made it rarer, which is worse than failing. On one GPU with a plain allocator, **bf16 is exactly deterministic through all 64 layers and generates coherent prose** — at 50 GB and 4.3× the throughput of the 111 GB fp32 it replaced. | +| "Structural verification of a quant (layer split, MTP graft, ignore list, tensor dtypes all correct) means the quant is good" | general belief | **FALSE — cost a full cutover of a gibberish seat 2026-09-14.** The `cyberprev` quant passed every structural check AND matched the reference config exactly, yet served pure gibberish, because the SOURCE checkpoint (`hotdogs/Qwen3.8-27B-abliterated-cyber-preview`, an unsloth export) had 850/1199 body tensors carrying a **tripled `language_model.` prefix** — verified present in the UPSTREAM HF index, not our download. transformers silently loaded the unmapped tensors as random-init, quantized noise, and every structural check passed because it inspected the OUTPUT config, never that INPUT names mapped. **A plain greedy coherence smoke (spec off) is the load-bearing gate and caught it; structure never can.** Fix: rewrite tensor names to a known-good reference of the same arch (exact 1199-tensor match), re-quant. Add a name-set diff of the bf16 SOURCE against a reference BEFORE quantizing. | | "A checkpoint with an FP8 PLE table but no `ple_embedding_dtype` declaration cannot be loaded; compressed-tensors qwen4_exp needs a vLLM source patch" | `stacks/flash-next-seat/README.md`, `persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md` | **SUPERSEDED 2026-09-14.** `from_quant_config` checks `ple_embedding_dtype` as **branch 1, before any quant-config type check**, and its `NotImplementedError` is scoped to the **PLE path only**. Declaring the FP8 PLE bypasses it on stock mainline — a **one-key config fix**, not a source build. Proven live: orcarouter (compressed-tensors) serves on gen-large after converting its bf16 PLE to FP8 + declaring it. ⚠ Declare only what is TRUE — declaring FP8 over a bf16 table is the gorbatjovy failure in reverse. | --- diff --git a/stacks/cyberprev-seat/.env.example b/stacks/cyberprev-seat/.env.example index e16cc4a..03cfb46 100644 --- a/stacks/cyberprev-seat/.env.example +++ b/stacks/cyberprev-seat/.env.example @@ -14,9 +14,13 @@ CYBER_PORT=8025 CYBER_CONTAINER_NAME=vllm-cyberprev # ── Model ─────────────────────────────────────────────────────────────────── -# In-house mixed quant of /tank/aimodels/cyberprev-bf16 (51.0 GiB bf16 source), -# built with services/gen-seat-mixed-quant/. compressed-tensors, NOT modelopt_fp4. -CYBER_MODEL=/tank/aimodels/cyberprev-nvfp4-mixed +# In-house mixed quant, compressed-tensors (NOT modelopt_fp4). +# ⚠ Quantized from /tank/aimodels/cyberprev-bf16-FIXED, not the raw download. The raw +# hotdogs export ships 850/1199 body tensors with a TRIPLED `language_model.` prefix +# (upstream bug) -> quant of the raw source is gibberish that passes every structural +# check. cyberprev-bf16-fixed is the name-repaired source (verified tensor-for-tensor +# against qwen38-27b-uncensored-bf16). See docs/pfi/model-quantization-playbook.md. +CYBER_MODEL=/tank/aimodels/cyberprev-nvfp4-mixed-v2 CYBER_QUANT=compressed-tensors # ⚠ Its OWN name. Do not reuse `sentinel-r3` — that seat is retired and its gateway # aliases are deliberately left to 404 rather than repointed at different weights. diff --git a/stacks/cyberprev-seat/compose.yaml b/stacks/cyberprev-seat/compose.yaml index 085c69d..ed14b93 100644 --- a/stacks/cyberprev-seat/compose.yaml +++ b/stacks/cyberprev-seat/compose.yaml @@ -42,7 +42,7 @@ services: - "${CYBER_PORT:-8025}:8000" volumes: - /tank/aimodels/huggingface:/hfcache - - ${CYBER_MODEL:-/tank/aimodels/cyberprev-nvfp4-mixed}:/model:ro + - ${CYBER_MODEL:-/tank/aimodels/cyberprev-nvfp4-mixed-v2}:/model:ro # DFlash2 drafter, mounted unconditionally — inert if CYBER_SPEC_CONFIG selects an MTP # method, which does not reference /drafter. Shared with mog-sec and (formerly) # sentinel-r3: all three share the same 64-layer Qwen3.8-27B base and vocab, so one diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index e944b41..51af742 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -969,6 +969,44 @@ model_list: model_info: mode: chat + # cyberprev-27b -> hotdogs/Qwen3.8-27B-abliterated-cyber-preview (:8025, fv-ml1 GPU0). + # In-house mixed NVFP4 quant, dflash k=7. Replaced the retired sentinel-r3 seat 2026-09-14. + # Sampler mirrors sec/sec-reasoning so an A/B isolates weights, not sampling. The retired + # sentinel-r3 aliases were DELETED, not repointed — cyberprev weights under sentinel's name + # would be silent substitution. + - model_name: cyberprev-27b + litellm_params: + model: hosted_vllm/cyberprev-27b + api_base: http://10.251.50.54:8025/v1 + api_key: os.environ/VLLM_API_KEY + temperature: 0.7 + top_p: 0.8 + presence_penalty: 0.0 + extra_body: + top_k: 20 + min_p: 0.0 + repetition_penalty: 1.0 + chat_template_kwargs: + enable_thinking: false + model_info: + mode: chat + - model_name: cyberprev-reasoning + litellm_params: + model: hosted_vllm/cyberprev-27b-thinking + api_base: http://10.251.50.54:8025/v1 + api_key: os.environ/VLLM_API_KEY + temperature: 1.0 + top_p: 0.95 + presence_penalty: 0.0 + extra_body: + top_k: 20 + min_p: 0.0 + repetition_penalty: 1.0 + chat_template_kwargs: + enable_thinking: true + model_info: + mode: chat + general_settings: master_key: os.environ/LITELLM_MASTER_KEY database_url: os.environ/DATABASE_URL