From 1a36e60d3ad10cd5f0536cd8b8f3fe52b6e61692 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 26 Aug 2026 16:35:43 -0700 Subject: [PATCH] =?UTF-8?q?docs(quant-playbook):=20=C2=A73.7's=20APC-off?= =?UTF-8?q?=20mitigation=20was=20reverted=20nine=20days=20ago=20and=20the?= =?UTF-8?q?=20section=20never=20said=20so?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found while answering a question from the operator, relayed via brokkr-smithy-dev, about whether a recorded Qwen3.8 degeneracy at ~1,700 tokens relates to a length sensitivity just measured on the tuned Gemma-4. The record is §3.7 and the number is ~2,000 -- but reading it to answer that question surfaced that the section is stale. §3.7 presented "disable prefix caching, keep MTP" as THE MITIGATION, resolved 2026-08-17, and stated the gen seat runs that config. It does not and has not since that same day: APC-off passed a synthetic 7-turn probe and the operator still saw severe degeneration in real use, so it was reverted. The multi-day hunt resolved to the AEON W4A4 quant being defective, with MTP / prefix-caching / gateway merely amplifying it (§3.8 records the corrected causal story; §3.7 was never updated to match). Verified against the live container rather than against the compose file alone: vllm-gen runs --enable-prefix-caching with qwen3_5_mtp / num_speculative_tokens 3. stacks/gen-seat/compose.yaml carries the full corrected history inline and is the current authority. §3.7's superseded text is kept and fenced rather than deleted -- it is the history of a mitigation that looked right and was not. Added a dated row to §7 per the standing rule that a wrong playbook claim gets a superseded-claims entry, not just a fix. The lesson inside the lesson is worth more than the correction: §3.7's own standing rule is "gate MTP on a multi-turn coherence probe, not just single-shot acceptance." The APC-off mitigation was gated on exactly that probe, passed it, and still failed in real use -- the multi-turn probe was itself too small to gate on. A passing probe is not sufficient evidence at any size that has not been calibrated against real use. --- docs/pfi/model-quantization-playbook.md | 34 +++++++++++++++++++------ 1 file changed, 26 insertions(+), 8 deletions(-) diff --git a/docs/pfi/model-quantization-playbook.md b/docs/pfi/model-quantization-playbook.md index 7b1a3a5..4f0c905 100644 --- a/docs/pfi/model-quantization-playbook.md +++ b/docs/pfi/model-quantization-playbook.md @@ -209,14 +209,31 @@ Open upstream question (queried dvalin/bil-smithy 2026-08-17). Run a 7-turn varied-topic conversation and watch turns past ~2k cumulative tokens for length-collapse and cross-turn bleed. -**THE MITIGATION (resolved 2026-08-17): disable prefix caching, keep MTP.** The -corruption is gated on MTP × prefix-caching *together* (vllm#43559 / #47194) — with -`--no-enable-prefix-caching` the GDN cache runs in a mode where the buggy -partial-accept align-path is inert. Confirmed on our stack: AEON W4A4, MTP on + -prefix-caching off → the 7-turn varied series stays coherent through 3.9k tokens, -zero bleed, at **104.6 tok/s / 53.6% acceptance** — i.e. the FULL MTP speedup back -(vs ~half with MTP off), losing only prefix-cache reuse. The gen seat runs this -config as of 2026-08-17. +⛔ **THE APC-OFF MITIGATION DID NOT HOLD — SUPERSEDED 2026-08-17, see §7.** What +follows is kept for its history and **must not be applied**: *"disable prefix caching, +keep MTP. The corruption is gated on MTP × prefix-caching together (vllm#43559 / +#47194); with `--no-enable-prefix-caching` the buggy partial-accept align-path is +inert. Confirmed on our stack: AEON W4A4, MTP on + prefix-caching off → the 7-turn +varied series stays coherent through 3.9k tokens, zero bleed, at 104.6 tok/s / 53.6% +acceptance."* + +**It passed that synthetic 7-turn probe and the operator still saw severe degeneration +in real use.** Reverted the same day. The probe was structurally under-covering the +real workload on both content distribution and depth — which is §3.7's own standing +rule (*gate on a multi-turn coherence probe, not single-shot acceptance*) failing at +one level up: the multi-turn probe was itself too small to gate on. + +**WHAT ACTUALLY RESOLVED IT.** The multi-day hunt root-caused to the **AEON W4A4 quant +being defective** — ~15-20% of generations went degenerate — with MTP, prefix-caching +and the gateway all merely *amplifying* it. That is why every partial mitigation +"helped" without fixing anything (§3.8). The gen seat today runs the in-house +Heretic **mixed NVFP4+FP8** build (FP8 attention, not W4A4) on vLLM nightly carrying +#51113, with **MTP ON and prefix-caching ON**, and is coherent in real use. + +Verified against the live seat 2026-08-26: `vllm-gen` runs `--enable-prefix-caching` +with `qwen3_5_mtp` / `num_speculative_tokens 3`. The compose file +(`stacks/gen-seat/compose.yaml`) carries the full history inline and is the current +authority; this section was stale against it for nine days. Things that do **not** work, ruled out: `num_speculative_tokens=1` (corruption is depth-independent — reproduces at n=1 and n=2, deterministically probed upstream); @@ -643,6 +660,7 @@ day if followed: | "Use modelopt, NOT compressed-tensors — compressed-tensors can't load the BF16 MTP head, 0% acceptance" | `docs/runbooks/heretic2-nvfp4-mtp-seat.md` §landmine 2 | **SUPERSEDED 2026-08-14.** The 0% was the missing `re:^mtp.*` ignore (§3.3), not the format. compressed-tensors + the ignore gives 47.7–83.2% acceptance, live. Use compressed-tensors. | | "Abliteration desyncs the MTP head → uncensored models can't do MTP" | earlier auto-memory | **SUPERSEDED 2026-08-14.** A modest abliteration preserves MTP (83.7% at bf16). Test MTP on **bf16 first** to isolate abliteration from quant/graft confounds — and isolate before deleting a 50 GB source. | | "NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer" | `reference_nvfp4_w4a4_granite_infeasible` | **NARROWED.** True for *uniform* W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs **with FP8 on attention and late layers** is fine and is the current default (§2). | +| "The Qwen3.8 MTP corruption is fixed by disabling prefix caching while keeping MTP; the gen seat runs APC-off" | this playbook §3.7 (now marked), earlier auto-memory | **SUPERSEDED 2026-08-17, and the staleness was only caught 2026-08-26.** APC-off passed a synthetic 7-turn probe and the operator still saw severe degeneration in real use; reverted the same day. The real cause was the **AEON W4A4 quant being defective** (~15-20% degenerate generations), with MTP / prefix-caching / gateway merely AMPLIFYING it (§3.8). The gen seat runs **MTP ON and prefix-caching ON** on the in-house mixed NVFP4+FP8 build — verified against the live container 2026-08-26. ⚠ The lesson inside the lesson: a *passing multi-turn probe* was not sufficient evidence either. | | "transformers' Qwen3.5 DeltaNet linear-attention NaNs in bf16 without causal-conv1d; it is precision-driven cancellation and fp32 resolves it" | `services/coldfusion-abliteration/README.md`, `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md` | **SUPERSEDED 2026-08-20.** Precision was never the variable. The NaN came from **multi-GPU sharding** and **`expandable_segments`** (§3.9, §3.10); fp32 only made it rarer, which is worse than failing. On one GPU with a plain allocator, **bf16 is exactly deterministic through all 64 layers and generates coherent prose** — at 50 GB and 4.3× the throughput of the 111 GB fp32 it replaced. | ---