diff --git a/docs/pfi/model-quantization-playbook.md b/docs/pfi/model-quantization-playbook.md index 7b1a3a5..4f0c905 100644 --- a/docs/pfi/model-quantization-playbook.md +++ b/docs/pfi/model-quantization-playbook.md @@ -209,14 +209,31 @@ Open upstream question (queried dvalin/bil-smithy 2026-08-17). Run a 7-turn varied-topic conversation and watch turns past ~2k cumulative tokens for length-collapse and cross-turn bleed. -**THE MITIGATION (resolved 2026-08-17): disable prefix caching, keep MTP.** The -corruption is gated on MTP × prefix-caching *together* (vllm#43559 / #47194) — with -`--no-enable-prefix-caching` the GDN cache runs in a mode where the buggy -partial-accept align-path is inert. Confirmed on our stack: AEON W4A4, MTP on + -prefix-caching off → the 7-turn varied series stays coherent through 3.9k tokens, -zero bleed, at **104.6 tok/s / 53.6% acceptance** — i.e. the FULL MTP speedup back -(vs ~half with MTP off), losing only prefix-cache reuse. The gen seat runs this -config as of 2026-08-17. +⛔ **THE APC-OFF MITIGATION DID NOT HOLD — SUPERSEDED 2026-08-17, see §7.** What +follows is kept for its history and **must not be applied**: *"disable prefix caching, +keep MTP. The corruption is gated on MTP × prefix-caching together (vllm#43559 / +#47194); with `--no-enable-prefix-caching` the buggy partial-accept align-path is +inert. Confirmed on our stack: AEON W4A4, MTP on + prefix-caching off → the 7-turn +varied series stays coherent through 3.9k tokens, zero bleed, at 104.6 tok/s / 53.6% +acceptance."* + +**It passed that synthetic 7-turn probe and the operator still saw severe degeneration +in real use.** Reverted the same day. The probe was structurally under-covering the +real workload on both content distribution and depth — which is §3.7's own standing +rule (*gate on a multi-turn coherence probe, not single-shot acceptance*) failing at +one level up: the multi-turn probe was itself too small to gate on. + +**WHAT ACTUALLY RESOLVED IT.** The multi-day hunt root-caused to the **AEON W4A4 quant +being defective** — ~15-20% of generations went degenerate — with MTP, prefix-caching +and the gateway all merely *amplifying* it. That is why every partial mitigation +"helped" without fixing anything (§3.8). The gen seat today runs the in-house +Heretic **mixed NVFP4+FP8** build (FP8 attention, not W4A4) on vLLM nightly carrying +#51113, with **MTP ON and prefix-caching ON**, and is coherent in real use. + +Verified against the live seat 2026-08-26: `vllm-gen` runs `--enable-prefix-caching` +with `qwen3_5_mtp` / `num_speculative_tokens 3`. The compose file +(`stacks/gen-seat/compose.yaml`) carries the full history inline and is the current +authority; this section was stale against it for nine days. Things that do **not** work, ruled out: `num_speculative_tokens=1` (corruption is depth-independent — reproduces at n=1 and n=2, deterministically probed upstream); @@ -643,6 +660,7 @@ day if followed: | "Use modelopt, NOT compressed-tensors — compressed-tensors can't load the BF16 MTP head, 0% acceptance" | `docs/runbooks/heretic2-nvfp4-mtp-seat.md` §landmine 2 | **SUPERSEDED 2026-08-14.** The 0% was the missing `re:^mtp.*` ignore (§3.3), not the format. compressed-tensors + the ignore gives 47.7–83.2% acceptance, live. Use compressed-tensors. | | "Abliteration desyncs the MTP head → uncensored models can't do MTP" | earlier auto-memory | **SUPERSEDED 2026-08-14.** A modest abliteration preserves MTP (83.7% at bf16). Test MTP on **bf16 first** to isolate abliteration from quant/graft confounds — and isolate before deleting a 50 GB source. | | "NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer" | `reference_nvfp4_w4a4_granite_infeasible` | **NARROWED.** True for *uniform* W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs **with FP8 on attention and late layers** is fine and is the current default (§2). | +| "The Qwen3.8 MTP corruption is fixed by disabling prefix caching while keeping MTP; the gen seat runs APC-off" | this playbook §3.7 (now marked), earlier auto-memory | **SUPERSEDED 2026-08-17, and the staleness was only caught 2026-08-26.** APC-off passed a synthetic 7-turn probe and the operator still saw severe degeneration in real use; reverted the same day. The real cause was the **AEON W4A4 quant being defective** (~15-20% degenerate generations), with MTP / prefix-caching / gateway merely AMPLIFYING it (§3.8). The gen seat runs **MTP ON and prefix-caching ON** on the in-house mixed NVFP4+FP8 build — verified against the live container 2026-08-26. ⚠ The lesson inside the lesson: a *passing multi-turn probe* was not sufficient evidence either. | | "transformers' Qwen3.5 DeltaNet linear-attention NaNs in bf16 without causal-conv1d; it is precision-driven cancellation and fp32 resolves it" | `services/coldfusion-abliteration/README.md`, `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md` | **SUPERSEDED 2026-08-20.** Precision was never the variable. The NaN came from **multi-GPU sharding** and **`expandable_segments`** (§3.9, §3.10); fp32 only made it rarer, which is worse than failing. On one GPU with a plain allocator, **bf16 is exactly deterministic through all 64 layers and generates coherent prose** — at 50 GB and 4.3× the throughput of the 111 GB fp32 it replaced. | ---