diff --git a/docs/pfi/model-quantization-playbook.md b/docs/pfi/model-quantization-playbook.md index 24b927d..52cd2d3 100644 --- a/docs/pfi/model-quantization-playbook.md +++ b/docs/pfi/model-quantization-playbook.md @@ -777,6 +777,7 @@ day if followed: | "NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer" | `reference_nvfp4_w4a4_granite_infeasible` | **NARROWED.** True for *uniform* W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs **with FP8 on attention and late layers** is fine and is the current default (§2). | | "The Qwen3.8 MTP corruption is fixed by disabling prefix caching while keeping MTP; the gen seat runs APC-off" | this playbook §3.7 (now marked), earlier auto-memory | **SUPERSEDED 2026-08-17, and the staleness was only caught 2026-08-26.** APC-off passed a synthetic 7-turn probe and the operator still saw severe degeneration in real use; reverted the same day. The real cause was the **AEON W4A4 quant being defective** (~15-20% degenerate generations), with MTP / prefix-caching / gateway merely AMPLIFYING it (§3.8). The gen seat runs **MTP ON and prefix-caching ON** on the in-house mixed NVFP4+FP8 build — verified against the live container 2026-08-26. ⚠ The lesson inside the lesson: a *passing multi-turn probe* was not sufficient evidence either. | | "transformers' Qwen3.5 DeltaNet linear-attention NaNs in bf16 without causal-conv1d; it is precision-driven cancellation and fp32 resolves it" | `services/coldfusion-abliteration/README.md`, `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md` | **SUPERSEDED 2026-08-20.** Precision was never the variable. The NaN came from **multi-GPU sharding** and **`expandable_segments`** (§3.9, §3.10); fp32 only made it rarer, which is worse than failing. On one GPU with a plain allocator, **bf16 is exactly deterministic through all 64 layers and generates coherent prose** — at 50 GB and 4.3× the throughput of the 111 GB fp32 it replaced. | +| "A checkpoint with an FP8 PLE table but no `ple_embedding_dtype` declaration cannot be loaded; compressed-tensors qwen4_exp needs a vLLM source patch" | `stacks/flash-next-seat/README.md`, `persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md` | **SUPERSEDED 2026-09-14.** `from_quant_config` checks `ple_embedding_dtype` as **branch 1, before any quant-config type check**, and its `NotImplementedError` is scoped to the **PLE path only**. Declaring the FP8 PLE bypasses it on stock mainline — a **one-key config fix**, not a source build. Proven live: orcarouter (compressed-tensors) serves on gen-large after converting its bf16 PLE to FP8 + declaring it. ⚠ Declare only what is TRUE — declaring FP8 over a bf16 table is the gorbatjovy failure in reverse. | --- diff --git a/persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md b/persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md index 643390f..7a82b59 100644 --- a/persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md +++ b/persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md @@ -11,7 +11,7 @@ here is on the running box; regenerate the authoritative view with |---|---| | 0 | `mog-sec` (`sec` :8019, dflash k=7) · **`sentinel-r3`** (`sentinel-r3` :8025, dflash k=7 — NEW) | | 1 | **`meromero-charrp`** (`char-rp` :8016, MeroMero-v2-31B dense — restored) · `erp-seat` (`char-rp-fast` :8021) · reward · coder · embed · rerank | -| 2 | `flash-next` (`gen-large` :8022) — **DOWN** (orca swap failed; dealignai stopped) | +| 2 | `flash-next` (`gen-large` :8022) — **UP on orcarouter** (PLE converted bf16→FP8; MTP k=3, 60.4% accept) | | 3 | **RESERVED scratch** — empty, operator directive; benches/quants/probes only | ## What changed tonight @@ -38,48 +38,72 @@ here is on the running box; regenerate the authoritative view with concurrent-contention artifact; warm+isolated it was 121. Operator caught it by testing the running `sec` (102 tok/s) as reference. -## THE BLOCKER — gen-large "orca" swap (resume here) +## ✅ THE BLOCKER — RESOLVED 2026-09-14 (no source build needed) -Operator wants gen-large on **`orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4`** -(weight-only compressed-tensors: NVFP4 experts + FP8 attn + bf16 PLE) instead of the -incumbent **dealignai** build (modelopt_fp4 **W4A4** — 4-bit activations, the -long-context degradation mode). orcarouter is the trusted author (= `gen`), better on -the activation axis. Downloaded to `/tank/aimodels/qwen38-flash-next-orcarouter-nvfp4` -(170 GB, complete, 18 shards + MTP head, integrity-verified). +gen-large now serves **`orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4`** from +`/tank/aimodels/qwen38-flash-next-orcarouter-nvfp4-plefp8`. Healthy, coherent, MTP k=3. -**IT DOES NOT LOAD.** Root cause (read the loader source, definitive): -`Qwen4ExpPLEEmbeddingMethod.from_quant_config` in vLLM — even the LATEST mainline -nightly — has branches only for **ModelOpt** and **FP8** configs; for a -CompressedTensorsConfig it raises `NotImplementedError` BEFORE ever consulting the -ignore list. So: -- dealignai (modelopt_fp4) loads fine; orcarouter (compressed-tensors) cannot, on any - mainline vLLM. -- orcarouter's config also does NOT list its `ngram_embedding` in `ignore` (would need - adding) — but that's moot until the loader has a compressed-tensors branch. -- orcarouter's OWN card prescribes **TP=4 + `--enable-expert-parallel`** and - `vllm>=` — a LITERAL UNFILLED version placeholder. They - shipped a build whose runtime they couldn't pin. +**The earlier diagnosis was right about the symptom and wrong about the cost.** It said the +only auditable path was cherry-picking a PLE-loader branch onto a from-source Blackwell +build. Reading the loader in the running nightly showed otherwise: -**Backport vetted → UNFIT.** `wtdcode/vllm-backport` (docker `lazymio/vllm-backport`, -212★, Apache-2.0, DCO, active) resolved this exact error (issue #72) BUT: its purpose is -OLD GPUs (images sm86/sm80/sm89 only — **no Blackwell/sm_120**), it's built around its -OWN `wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16` quant not orcarouter's NVFP4, and Blackwell -has native FP4/FP8 so the whole backport premise is moot for us. Using it = source-build -for an untested arch to load a quant it never tested. +``` +from_quant_config (vllm/models/qwen4_exp/nvidia/ngram_embedding.py:168) + 1. ple_embedding_dtype == "float8_e4m3fn" -> FP8 method <-- BEFORE any type check + 2. quant_config is None -> unquantized + 3. ModelOptMixedPrecisionConfig -> FP8 / unquantized + 4. ModelOptQuantConfigBase + excluded -> unquantized + 5. not isinstance(quant_config, Fp8Config)-> NotImplementedError <-- the blocker +``` -**Options (operator decision pending):** -- (a) **Restore dealignai** — works today, identical architecture, ~90% of the quality - (loses only the weight-only-vs-W4A4 activation edge). The pragmatic interim. -- (b) Cherry-pick the ~10-line compressed-tensors PLE-loader branch onto a mainline - source build for Blackwell (mainline already has Blackwell FP4 compute). More - auditable than the backport, still a from-source build + maintenance. -- (c) Request orcarouter's own **W4A4** build (they offer it) — would load on the current - mainline image like dealignai. -- (d) Wait for mainline vLLM to merge compressed-tensors qwen4_exp. +Branch 1 is unconditional, and the `NotImplementedError` is **scoped to the PLE embedding +path only** — experts and dense layers of a compressed-tensors qwen4_exp build load through +vLLM's ordinary compressed-tensors paths. Falsified directly before doing any work: built +the real `CompressedTensorsConfig` from orca's own config and called `from_quant_config` +both ways — as-shipped raises, with the declaration returns `Qwen4ExpPLEFp8EmbeddingMethod`. -⚠ Operator said "don't reload dealignai unprompted, priority orca" — but orca needs -runtime work, so gen-large is intentionally DOWN pending the decision. Both configs -backed up on host: `compose.yaml.bak-20260914-orcarouter`, `.env.bak-20260914-orcarouter`. +### What was actually done + +1. **Converted the PLE table bf16 → FP8.** orca's 128 PLE tensors live in exactly ONE shard + (`model-00002-of-00017.safetensors`, 95.4 GiB) with **no other tensors in it** — a clean + split. Converted on GPU3 (reserved scratch) in 139 s into 8 `model-plefp8-*` files. + - global amax **0.0894**; per-shard outlier ratio only **1.66x**, so one global scale fits + - scale chosen **exactly representable in bf16** (2.002716e-04) so no scale-rounding error + stacks on the quantization error; amax maps to **446.17 / 448** → no clipping + - round-trip **2.655 % RMS relative**, 0.002 % underflow, **0 saturation** + - `weight_scale` written **BF16 [1]**, matching gorbatjovy's published format (read from + its actual safetensors header, not guessed) + - MTP head (31 tensors, BF16) and the 333 vision tensors carried through untouched +2. **Declared it**: `text_config.ple_embedding_dtype = "float8_e4m3fn"`. +3. **Renamed `layer_types`**: orca labels its 12 QSA layers `qwen_sparse_attention`; vLLM + accepts only `linear_attention` / `full_attention` and picks QSA via `indexer_n_heads`. + ⚠ **Verified `indexer_n_heads == 4` in BOTH orca and dealignai before renaming** — without + it the rename silently selects PLAIN attention and serves a subtly wrong model that still + looks healthy. Every QSA/indexer key matches dealignai exactly. +4. `.env`: `FN_MODEL` → the converted dir, `FN_QUANT` → `compressed-tensors`. + +### Measured on the live seat + +| | | +|---|---| +| warm decode, conc=1, greedy 300 tok, **n=5** | median **167.5 tok/s** (min 150.0, max 170.4, spread 12.2 %) | +| MTP k=3 | acceptance **60.4 %**, mean acceptance length **2.81** (per-pos 80.6/60.8/40.8 %) | +| KV | 344,155 tokens @ 262,144 ctx, 1.31x concurrency, pinned 10 GiB | +| on card | ~75 GiB; PLE 47.7 GiB pinned host RAM | + +⚠ **Do NOT read 167.5 as a win over dealignai.** dealignai's ~121 tok/s in this file came +from a different harness/prompt; cross-harness comparison is invalid. What IS established is +that weight-only experts (orca is W8 weight-only attn + W4 weight-only experts) did **not** +cost decode speed, which was the standing risk of giving up FP4 tensor-core compute. + +### Still open on this seat + +- Quality A/B orca vs dealignai (the actual reason for the swap — the W4A4 long-context + degradation axis). Needs a controlled harness + noise floor. +- Deep-prefill probe at 262K against THIS checkpoint. Startup is not a depth test. +- Rollback is two `.env` keys; `.env.bak-preorca-20260914-023408` on the host. +- Disk: the convert **copied** ~75 GiB of unchanged shards because hardlinks hit `EXDEV` + (separate bind mounts of the same fs). Harmless; reclaimable by relinking if /tank tightens. ## Other open items diff --git a/persistent-memory.md b/persistent-memory.md index 80d488f..b56d8ca 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-14 ~01:40 PT (FV RECOVERED 2026-09-13 midday; all-night fv-ml1 seat reorg — MTP on gen-large, gen consolidated, char-rp→MeroMero-v2, Sentinel-R3+dflash live. ⚠ gen-large DOWN: orca swap blocked on vLLM compressed-tensors qwen4_exp support.)_ +_Last updated: 2026-09-14 ~01:40 PT (FV RECOVERED 2026-09-13 midday; all-night fv-ml1 seat reorg — MTP on gen-large, gen consolidated, char-rp→MeroMero-v2, Sentinel-R3+dflash live. ✅ gen-large RESOLVED 2026-09-14: orca serving after a PLE bf16→FP8 convert + 2 config keys.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -128,22 +128,48 @@ the mesh serves the route). Full recovery detail in `docs/runbooks/fv-site-dark-20260913.md`; the site-visit measurements (breaker rating, 4-card ammeter) remain open — every power figure is arithmetic on an estimated platform draw. -### ⚠ THE RESUME POINT — gen-large is DOWN +### ✅ RESOLVED — gen-large now serves orcarouter (was the "orca blocker") -Operator wants gen-large swapped from the incumbent **dealignai** W4A4 build to -**orcarouter** weight-only NVFP4. Downloaded (170 GB, verified) but **it does not load**: -no mainline vLLM (even latest nightly) supports the compressed-tensors qwen4_exp PLE -loader — it raises `NotImplementedError` before checking the ignore list. Backport vetted -→ unfit (old-hardware fork, no Blackwell image, its own AWQ quant). Options: restore -dealignai (works today), cherry-pick the ~10-line PLE-loader branch onto a mainline -source build, request orcarouter's own W4A4, or wait for mainline. **Decision pending.** -Operator: don't reload dealignai unprompted — so gen-large stays down. Both configs -backed up on host. → `persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md` +**gen-large (:8022) serves `orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4` as of +2026-09-14**, from `/tank/aimodels/qwen38-flash-next-orcarouter-nvfp4-plefp8`. +Healthy, coherent, MTP k=3 accepting 60.4%. + +The blocker was real but **much cheaper to fix than the earlier note concluded** (it +proposed a from-source vLLM build). Two independent blockers, both config-level: + +1. **PLE loader.** `Qwen4ExpPLEEmbeddingMethod.from_quant_config` checks + `ple_embedding_dtype == "float8_e4m3fn"` as **branch 1, BEFORE any quant-config type + check**, and the `NotImplementedError` for CompressedTensorsConfig is **scoped to the + PLE path only** — experts/dense load through the ordinary compressed-tensors paths. + So declaring an FP8 PLE bypasses the blocker entirely. orca ships a **bf16** PLE, so + the fix was: convert the table bf16→FP8 (one shard file, 128 tensors, clean split) + and add the one config key. **No source build, no backport.** +2. **`ValueError: Invalid layer_type qwen_sparse_attention`.** orca labels its 12 QSA + layers `qwen_sparse_attention`; vLLM accepts only `linear_attention` / + `full_attention` and selects QSA via `indexer_n_heads`. Renamed the 12 entries. + ⚠ Verified `indexer_n_heads=4` in BOTH orca and dealignai first — without it the + rename would silently select PLAIN attention and serve a subtly wrong model. + +**Conversion, measured:** global amax 0.0894, per-shard outlier ratio only 1.66x, so one +global scale is well-conditioned. Scale chosen **exactly representable in bf16** +(2.002716e-04) so no scale-rounding error stacks on quantization; max maps to 446.17/448, +no clipping. Round-trip **2.655% RMS relative**, 0.002% underflow, 0 saturation. +MTP head (31 tensors, BF16) preserved. `weight_scale` BF16[1] matches gorbatjovy's format. + +**Measured on the live seat** (warm, conc=1, greedy 300-tok, n=5): median **167.5 tok/s** +(min 150.0 / max 170.4, spread 12.2%), MTP acceptance 60.4%, mean acceptance length 2.81, +KV 344,155 tokens @ 262,144 ctx, 1.31x concurrency. ⚠ The reorg note's dealignai figure +(~121 tok/s) came from a DIFFERENT harness — do not treat this as a measured win over +dealignai; a controlled A/B needs dealignai back on this same harness plus a noise floor. + +⚠ Not yet done: quality A/B vs dealignai, and a deep-prefill probe at 262K on this +checkpoint. `.env` rollback is two keys (`.env.bak-preorca-20260914-023408`). +→ `persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md` ### fv-ml1 seat topology now - **GPU0:** `sec`/mog-sec (:8019, dflash k=7) · `sentinel-r3` (:8025, dflash k=7 — NEW) - **GPU1:** `char-rp`/MeroMero-v2-31B (:8016, restored from a leftover 26B) · `char-rp-fast`/erp-seat (:8021) · reward · coder · embed · rerank -- **GPU2:** `gen-large`/flash-next (:8022) — **DOWN** +- **GPU2:** `gen-large`/flash-next (:8022) — **UP, orcarouter** (MTP k=3, 60.4% accept) - **GPU3:** RESERVED scratch (empty; benches/quants/probes only) Detail + the full change list (MTP campaign, gen consolidation freeing 38 GB, char-rp @@ -163,7 +189,7 @@ restore, Sentinel dflash cutover, the orca blocker, cyberprev quant crash, open ## Recent decisions -- `[2026-09-14]` fv-ml1 all-night seat reorg — MTP k=3 on gen-large (+52%@conc1), gen consolidated onto flash-next (27B dense retired, 38 GB freed), char-rp restored to MeroMero-v2-31B, Sentinel-R3 served + dflash cutover (beat MTP 2.40 vs 2.18). ⚠ gen-large DOWN: orcarouter swap blocked — no mainline vLLM loads compressed-tensors qwen4_exp; backport vetted unfit. → `persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md` +- `[2026-09-14]` fv-ml1 all-night seat reorg — MTP k=3 on gen-large (+52%@conc1), gen consolidated onto flash-next (27B dense retired, 38 GB freed), char-rp restored to MeroMero-v2-31B, Sentinel-R3 served + dflash cutover (beat MTP 2.40 vs 2.18). ✅ gen-large RESOLVED 2026-09-14 — orcarouter serving: PLE bf16→FP8 convert + `ple_embedding_dtype` + `layer_types` rename; NO source build needed. → `persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md` - `[2026-09-13]` ⭐ **STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint.** ⭐ **Two DIFFERENT boxes, clarified by operator 2026-09-13:** **fv-ml1** = 4x **Blackwell** RTX PRO 6000 **Max-Q @ 300 W** (Max-Q is the reduced-TGP SKU; Workstation Edition is 600 W), 391 GB VRAM, deployed. **ana-ml3** = 2x **Ada Generation** RTX 6000 @ 300 W, 96 GB VRAM, **NOT YET DEPLOYED**. **CAPS: fv-ml1 → 250 W/card (83% of TGP, ~5% throughput cost); ana-ml3 → 200 W/card (67%, ~10-15%).** ⚠⚠⚠ **`nvidia-smi -pl` caps BOARD power, not WALL power** — 4x250 = 1000 W board + ~180-300 W host components = 1180-1300 W, ÷ ~0.90 PSU efficiency = **~1310-1445 W AT THE PLUG vs a 15 A circuit's 1440 W NEC continuous limit. 250 W lands ON the line, not under it** (200 W would give ~1090-1220 W, comfortable). **Procedure: set 250 W, then VERIFY at the plug under four-card load; fall back to 200 W if it reads near 1440 W.** ⚠ Caps bound SUSTAINED draw, not transients — breakers tolerate brief overload, UPS overload protection does not, so 250 W implicitly commits the fv-ml1 chassis to the PDU rather than behind the 1500 VA unit (even capped it exceeds that UPS). ⭐ **BANK-level cap asked about and answered:** `DCGM_CONFIG_POWER_BUDGET_GROUP` ("power budget for the entire group") exists alongside `DCGM_CONFIG_POWER_CAP_INDIVIDUAL`, but the docs do not say how it distributes — and the only primitive underneath is NVML's PER-GPU `nvmlDeviceSetPowerManagementLimit` (**no bank-level register**), so any group budget resolves to N per-GPU writes. "Each card free until all are loaded" is therefore a CONTROL LOOP, not a hardware feature. ⚠⚠ If one is written it must be **safe-by-default, opportunistic upward** — boot at budget/N and only RAISE after observing idle neighbours; a reactive loop overshoots during a load RAMP, which is the all-cards-at-once case it exists to prevent. **Not yet worth it: 4x250 = 1000 W IS the bank budget**, so the daemon's whole prize is the one-card-busy case (~17% more board power ≈ ~5% throughput) — rare for a SERVING fleet (one seat per card), valuable for a TRAINING window. **DCGM** = NVIDIA's first-party **Data Center GPU Manager** (Apache-2.0), layered ABOVE NVML: nvidia-smi → NVML per-GPU primitives → DCGM daemon/dcgmi for health, diagnostics, config enforcement, groups. ✅ **VERIFIED 2026-09-13 (I had claimed the opposite and was WRONG): DCGM SUPPORTS our cards and power config works on them.** Supported platforms cover *"All NVIDIA Maxwell and newer NON-DATACENTER (e.g. GeForce or Quadro) GPUs"*, and the feature-overview table marks **Configuration Management ✓ for Tesla/Titan/Quadro/GeForce alike** — including *"Power Limit: set the maximum allowed power consumption"*. What IS gated on non-datacenter cards is **diagnostics** (Level 1 only vs All Levels on Tesla), not config. ⚠ Soft edge: the table says "Quadro", the former professional-line name; RTX 6000 Ada / RTX PRO 6000 are its successors and should fall in that column, but the table predates the rename — one command on the box settles it. ⭐ So the group-budget test IS worth running; what remains unsettled is *distribution*, not availability. ✅ **The static cap needs none of it**: `nvidia-smi -pl 250` is plain NVML and works on these cards; DCGM is probably not even installed (beszel-agent-nvidia shells out to nvidia-smi). ⚠ ana-ml3's 200 W on 2 cards (2x200 = 400 W) — and ⭐ ana-ml3 lands in the **Anaheim** rack whose breaker tripped 2026-08-26 and 2026-09-11, one of those caused by this very chassis before it relocated, so the cap there is remediation of a known-bad circuit, not precaution. 4x200 W = 800 W of card, which fits a real circuit with a real UPS. This is the generalised lesson of the FV outage: decide the power envelope first and size the cards into it. At 300 W TGP, 200 W is a **67% cap — the favourable part of the concave perf/watt curve, ~10-15% throughput cost**, not the severe 33% cap a 600 W part would have meant; and 200 W is very unlikely to sit below a 300 W card's enforceable floor (still confirm with `nvidia-smi -q -d POWER | grep -iE 'power limit|default'`). ⭐⭐ **The outage arithmetic now has numbers:** 2x Blackwell Max-Q @300 W ≈ 600 W of card + host (566 GB RAM, drives, fans, PSU losses) ≈ 200-350 W = **~800-950 W against a 1500 VA Eaton's real ~900-1200 W rating** — at or just over the line, which is what explains a full day on ONE card (~500-650 W, inside) and death minutes into the SECOND. The host term is the only guess; idle-at-the-plug measures it. ⚠⚠ **And FOUR cards is a BREAKER problem, not a UPS problem:** 4x300 + ~300 host ≈ **1500 W vs a 15 A circuit's 1440 W continuous (80%) derating** — so **capping belongs at fv-ml1 too**, or it needs a 20 A feed, before anyone loads all four cards again. Today's incident only ever had TWO cards working. ⭐ **Decode tolerates caps far better than training** (memory-bandwidth-bound, concave curve). ⚠⚠ **ana-ml3's Ada is sm_89: native FP8 but NO NVFP4** (Blackwell-only) — most of our in-house quants are NVFP4 and will NOT run accelerated there; ana-ml3's seats want FP8 W8A8, or NVFP4 checkpoints stay on fv-ml1. ⭐ **This unparks [[parked_triton_backend_ampere_fp8]]** — a hard no on Ampere (fp8e4nv unsupported sm_86), explicitly deferred TO Ada, and sm_89 has the FP8 support it needs. **ana-ml3 VRAM is 2x48 = 96 GB** vs fv-ml1's 391 GB, so big-model placement stays at FV (Flash-Next needs 74 GiB resident on ONE card — the offload moves the table, not the experts — so it cannot run on a 48 GB Ada card at all). ⚠ **PERSIST the cap** (systemd unit + persistence mode, ordered before Docker): a hand-set limit stops holding at the next reboot, which is likely to be the very power event it existed to prevent. → `docs/runbooks/fv-site-dark-20260913.md` diff --git a/stacks/flash-next-seat/.env.example b/stacks/flash-next-seat/.env.example index 1a893f0..2196f84 100644 --- a/stacks/flash-next-seat/.env.example +++ b/stacks/flash-next-seat/.env.example @@ -23,9 +23,14 @@ FN_PORT=8022 FN_CONTAINER_NAME=vllm-flash-next # ── Model ─────────────────────────────────────────────────────────────────── -# dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b990578ef3031eccf9f28e675a289a09ee9 -FN_MODEL=/tank/aimodels/qwen38-flash-next-abliterated-nvfp4 -FN_QUANT=modelopt_fp4 +# orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, with its bf16 PLE converted to FP8 +# in-house (2026-09-14) so vLLM's from_quant_config branch 1 selects the FP8 PLE method. +# ⚠ compressed-tensors, NOT modelopt_fp4 — this checkpoint is weight-only on BOTH axes +# (W8 float attn / W4 float experts, input_activations: null), which is why it was chosen +# over the dealignai W4A4 build. Rollback: the dealignai pair below, both still on disk. +# FN_MODEL=/tank/aimodels/qwen38-flash-next-abliterated-nvfp4 FN_QUANT=modelopt_fp4 +FN_MODEL=/tank/aimodels/qwen38-flash-next-orcarouter-nvfp4-plefp8 +FN_QUANT=compressed-tensors FN_SERVED_NAME=qwen3.8-flash-next-uncensored FN_SERVED_NAME_THINK=qwen3.8-flash-next-uncensored-thinking diff --git a/stacks/flash-next-seat/README.md b/stacks/flash-next-seat/README.md index 4dfadd9..8d6d0ff 100644 --- a/stacks/flash-next-seat/README.md +++ b/stacks/flash-next-seat/README.md @@ -10,12 +10,12 @@ stream with async prefetch. | | | |---|---| -| Checkpoint | `dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4` @ `be794b99…` (126.0 GiB) | -| On the card | ~78 GiB of 95.6 GiB — routed experts NVFP4 W4A4, rest at source precision | -| In host RAM | 47.7 GiB pinned, FP8 E4M3, 10 `model-plefp8-*` shards + per-table scalar scale | -| Context | 131,072 to start (native ceiling 262,144) — see *Raising context* | -| Speculative decoding | **none** — see *Why MTP is off* | -| Gateway wiring | **none yet** — this seat is not in LiteLLM; `gen` is untouched | +| Checkpoint | `orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4`, PLE converted bf16→FP8 in-house 2026-09-14 (123.2 GiB) | +| On the card | ~75 GiB of 95.6 GiB — **weight-only on both axes**: W4 float experts, W8 float attn, `input_activations: null` | +| In host RAM | 47.7 GiB pinned, FP8 E4M3, 8 `model-plefp8-*` shards + one global BF16 scale | +| Context | **262,144** (full native) — KV 344,155 tokens, 1.31x concurrency | +| Speculative decoding | **MTP k=3** — 60.4% acceptance, mean acceptance length 2.81 (measured here, n=5) | +| Gateway wiring | **8 aliases** — gen, gen-reasoning, summarizer(-large), classifier, chat-judge, image-judge, qwen-image-bench | ## Deploy @@ -39,59 +39,99 @@ Four ideas, and three of them shape the serving config: framing: capacity with almost no per-token compute. - **Gated residual / hyper-connections.** Four residual branches; excluded from quantization in this checkpoint. -- **MTP head.** Present and preserved byte-identically. Deliberately unused. +- **MTP head.** Present, preserved byte-identically, and **in use at k=3**. -## Why this checkpoint, and the trap that disqualifies most others +## Why this checkpoint, and the two traps in front of it -vLLM selects the PLE table's weight format from **`text_config.ple_embedding_dtype`**, -as the *first* branch of `Qwen4ExpPLEEmbeddingMethod.from_quant_config`. This -checkpoint declares `"float8_e4m3fn"`. +Chosen for the **activation axis**: orcarouter's build is weight-only on *both* halves — +`config_groups` gives W8 float for attention/dense and W4 float for the experts, with +`input_activations: null` on each. The displaced dealignai build is ModelOpt **W4A4** +(4-bit activations), the long-context degradation mode. Same author as the `gen` seat. -A build that ships an FP8 PLE table **without** that declaration resolves instead -through ModelOpt's `*.ple.*` exclude to the *unquantized* method, never registers -the `weight_scale` parameter, and dies on load with `no module or parameter named -'ngram_embedding.weight_scale'`. `gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8` -is exactly this case. **Check that field before trying another build.** +It did not load out of the box, and there were **two independent config-level blockers**. +Both are recorded here because each looks like a capability gap and neither is one. -Chosen over better-liked alternatives because its provenance states protocols and -repeat counts rather than adjectives. From its own `qualification-notes.md` and -metrics files, kept in the model directory: +### Trap 1 — the PLE loader (and the claim we had wrong) -- HarmBench, 240 genuinely-harmful behaviours, greedy: 100% compliance at reasoning - off / low / xhigh. -- MMLU 82.11% → 81.93% (−0.18 pp) on an identical harness, 2,280 questions. -- GSM8K 97.27% (1283/1319), full set, single-shot, temp 0.6 — inside the stated - BF16 reference band 97.12–97.50. -- AIME26 pass@1 98.75% (237/240, **SEM 0.61 pp, 30 problems × 8 repeats**), - majority@8 100%, `max_tokens` 130,000, 4.9M completion tokens, stop_rate 99.17%. -- Byte-equality audit of unchanged tensors: 1,562 tensors / 118.4 GB compared, all - passed, **including all 31 MTP tensors**. +vLLM picks the PLE table's format in `Qwen4ExpPLEEmbeddingMethod.from_quant_config`: -⚠ Two honest gaps in that evidence. The routed experts are **NVFP4 W4A4**, and -nobody — including the publisher — has measured this checkpoint at the full 262K -context; AIME26's 130K-token generations are the deepest evidence that exists. -Separately, `validate_checkpoint_report.json` in the repo describes the *earlier -BF16-PLE revision* (204 shards / 173.6 GiB), not the published FP8-PLE one. +``` +1. ple_embedding_dtype == "float8_e4m3fn" -> FP8 method <-- BEFORE any type check +2. quant_config is None -> unquantized +3. ModelOptMixedPrecisionConfig -> FP8 / unquantized +4. ModelOptQuantConfigBase + excluded -> unquantized +5. not isinstance(quant_config, Fp8Config)-> NotImplementedError +``` -Rejected alternatives, for the record: -`orcarouter/…-Uncensored-NVFP4` is gated (access request pending nothing — not -requested); `nvidia/…-NVFP4` is the cleanest ModelOpt MIXED_PRECISION build but is -not abliterated; `lovedheart/…-Pruned-RTXPRO-6000` prunes to 448 of 512 experts. +⚠ **This README previously said an FP8 PLE without the declaration is disqualifying, and +that compressed-tensors needs a vLLM source patch. Both were wrong** (corrected 2026-09-14; +see the quantization playbook's superseded-claims table). Branch 1 is **unconditional**, and +the `NotImplementedError` is **scoped to the PLE path only** — experts and dense layers of a +compressed-tensors build load through vLLM's ordinary compressed-tensors paths. So declaring +an FP8 PLE bypasses the blocker on stock mainline. -## Why MTP is off +orcarouter ships a **bf16** PLE, so the honest fix was to *make the declaration true*: +convert the table to FP8, then declare it. Its 128 PLE tensors sit in exactly one shard file +with nothing else in it, which makes that a clean, cheap rewrite. -Against our house graft-MTP habit, and on purpose. +⚠ **Declare only what is true.** `gorbatjovy/...-NVFP4-plefp8` ships an FP8 table with no +declaration and dies on `ngram_embedding.weight_scale`; declaring FP8 over a *bf16* table is +that same failure in reverse. The declaration is a claim about the bytes, not a switch. -vLLM's own recipe for this model measured MTP on 4×H100 as **worse at every -concurrency tested** — 8–36% lower request throughput, 32–173% higher per-token -latency, driven by ~36% acceptance — and says do not enable it by default. Open -issue **#55357** reports episodic 0% draft acceptance with repetition collapse -inside thinking blocks. Open **#55496** reports ModelOpt `MIXED_PRECISION` failing -to load FP8_BLOCK_SCALES MTP experts. +### Trap 2 — `Invalid layer_type qwen_sparse_attention` -Turning it on is two lines in `compose.yaml` (documented in place). If you do, -measure it **here**, with repeats, against this seat's own baseline — the numbers -above are someone else's hardware. +orcarouter labels its 12 QSA layers `qwen_sparse_attention`. vLLM accepts only +`linear_attention` and `full_attention`, and selects QSA *within* `full_attention` when +`indexer_n_heads` is present. The fix is renaming the 12 entries. + +⚠⚠ **Check `indexer_n_heads` before renaming.** Without it the rename silently selects plain +`Qwen3NextAttention` instead of `Qwen4ExpQSAAttention` — a subtly wrong model that loads, +serves, and passes a healthcheck. Verified `indexer_n_heads == 4` in both this checkpoint and +the dealignai one, along with every other indexer/QSA key, before touching it. + +### The conversion, and what it cost + +Global amax 0.0894 with a per-shard outlier ratio of only **1.66x**, so the single global +scale this method uses is well-conditioned here. The scale is chosen **exactly representable +in bf16** (2.002716e-04) so no scale-rounding error stacks on the quantization error; amax +maps to 446.17 of 448, so nothing clips. Round-trip **2.655% RMS relative**, 0.002% underflow, +zero saturation — and the same FP8-PLE treatment dealignai already shipped, so it is not a +regression against the seat it replaced. `weight_scale` is written BF16 [1] to match the +published format. MTP head (31 tensors) and the vision tower carry through untouched. + +⚠ Still unmeasured: a controlled quality A/B against dealignai — which is the entire reason +for the swap — and a deep-prefill probe at 262K on this checkpoint. Rollback is two `.env` +keys; the dealignai checkpoint is still on disk. + +Rejected alternatives, for the record: `nvidia/…-NVFP4` is the cleanest ModelOpt build but is +not abliterated; `lovedheart/…-Pruned-RTXPRO-6000` prunes to 448 of 512 experts; +`windowsxp811203/…-Abliterated-NVFP4` stores its 95 GiB PLE as a single malformed +`ple_embedding.shard_.weight` instead of 128 `ngram_embedding.shard_N.weight` and has never +been served by its own author. + +## Why MTP is ON at k=3 (reversing this seat's original default) + +This seat shipped with speculative decoding off, citing vLLM's recipe: on 4xH100 at TP=4 +that recipe measured MTP **worse at every concurrency** (8-36% lower throughput, 32-173% +higher per-token latency, ~36% acceptance) and says do not default it on. Open #55357 +reports episodic 0% acceptance with repetition collapse inside thinking blocks. + +**Measured here, that inverted.** The campaign in `services/flash-next-mtp-bench/` found MTP +a win at every k and every concurrency tested on one Blackwell card (+29/41/27% at k=1, ++42/52/38% at k=2, +52/51/34% at k=3 across conc 1/4/8). k=3 is deployed because this is a +single-user fleet and conc=1 dominates. + +On the current orcarouter checkpoint, measured 2026-09-14: **60.4% acceptance, mean +acceptance length 2.81** (per-position 80.6 / 60.8 / 40.8%), warm decode median **167.5 +tok/s** at conc=1 (n=5, spread 12.2%). + +⚠ **MTP costs KV.** The draft head adds ~5.08 GiB of weights and raises per-token KV cost +~16%; `FN_KV_CACHE_MEMORY` was cut 14 -> 10 GiB for it. At 14 GiB the engine OOMs at init +with MTP on. If it OOMs, drop to 8589934592. + +⚠ The recipe's numbers are someone else's hardware, and so are ours to anyone else. Re-measure +on the seat, warm, with repeats — the first decode bench during the reorg read 39 tok/s and +that was a cold-boot + contention artifact, not a result. ## The upstream situation, as of 2026-09-13