docs(quant-playbook): §3.7's APC-off mitigation was reverted nine days ago and the section never said so

Found while answering a question from the operator, relayed via brokkr-smithy-dev,
about whether a recorded Qwen3.8 degeneracy at ~1,700 tokens relates to a length
sensitivity just measured on the tuned Gemma-4. The record is §3.7 and the number is
~2,000 -- but reading it to answer that question surfaced that the section is stale.

§3.7 presented "disable prefix caching, keep MTP" as THE MITIGATION, resolved
2026-08-17, and stated the gen seat runs that config. It does not and has not since
that same day: APC-off passed a synthetic 7-turn probe and the operator still saw
severe degeneration in real use, so it was reverted. The multi-day hunt resolved to
the AEON W4A4 quant being defective, with MTP / prefix-caching / gateway merely
amplifying it (§3.8 records the corrected causal story; §3.7 was never updated to
match).

Verified against the live container rather than against the compose file alone:
vllm-gen runs --enable-prefix-caching with qwen3_5_mtp / num_speculative_tokens 3.
stacks/gen-seat/compose.yaml carries the full corrected history inline and is the
current authority.

§3.7's superseded text is kept and fenced rather than deleted -- it is the history of
a mitigation that looked right and was not. Added a dated row to §7 per the standing
rule that a wrong playbook claim gets a superseded-claims entry, not just a fix.

The lesson inside the lesson is worth more than the correction: §3.7's own standing
rule is "gate MTP on a multi-turn coherence probe, not just single-shot acceptance."
The APC-off mitigation was gated on exactly that probe, passed it, and still failed in
real use -- the multi-turn probe was itself too small to gate on. A passing probe is
not sufficient evidence at any size that has not been calibrated against real use.
This commit is contained in:
2026-08-26 16:35:43 -07:00
parent 5a51e76135
commit 1a36e60d3a
+26 -8
View File
@@ -209,14 +209,31 @@ Open upstream question (queried dvalin/bil-smithy 2026-08-17).
Run a 7-turn varied-topic conversation and watch turns past ~2k cumulative tokens
for length-collapse and cross-turn bleed.
**THE MITIGATION (resolved 2026-08-17): disable prefix caching, keep MTP.** The
corruption is gated on MTP × prefix-caching *together* (vllm#43559 / #47194) — with
`--no-enable-prefix-caching` the GDN cache runs in a mode where the buggy
partial-accept align-path is inert. Confirmed on our stack: AEON W4A4, MTP on +
prefix-caching off → the 7-turn varied series stays coherent through 3.9k tokens,
zero bleed, at **104.6 tok/s / 53.6% acceptance** — i.e. the FULL MTP speedup back
(vs ~half with MTP off), losing only prefix-cache reuse. The gen seat runs this
config as of 2026-08-17.
**THE APC-OFF MITIGATION DID NOT HOLD — SUPERSEDED 2026-08-17, see §7.** What
follows is kept for its history and **must not be applied**: *"disable prefix caching,
keep MTP. The corruption is gated on MTP × prefix-caching together (vllm#43559 /
#47194); with `--no-enable-prefix-caching` the buggy partial-accept align-path is
inert. Confirmed on our stack: AEON W4A4, MTP on + prefix-caching off → the 7-turn
varied series stays coherent through 3.9k tokens, zero bleed, at 104.6 tok/s / 53.6%
acceptance."*
**It passed that synthetic 7-turn probe and the operator still saw severe degeneration
in real use.** Reverted the same day. The probe was structurally under-covering the
real workload on both content distribution and depth — which is §3.7's own standing
rule (*gate on a multi-turn coherence probe, not single-shot acceptance*) failing at
one level up: the multi-turn probe was itself too small to gate on.
**WHAT ACTUALLY RESOLVED IT.** The multi-day hunt root-caused to the **AEON W4A4 quant
being defective** — ~15-20% of generations went degenerate — with MTP, prefix-caching
and the gateway all merely *amplifying* it. That is why every partial mitigation
"helped" without fixing anything (§3.8). The gen seat today runs the in-house
Heretic **mixed NVFP4+FP8** build (FP8 attention, not W4A4) on vLLM nightly carrying
#51113, with **MTP ON and prefix-caching ON**, and is coherent in real use.
Verified against the live seat 2026-08-26: `vllm-gen` runs `--enable-prefix-caching`
with `qwen3_5_mtp` / `num_speculative_tokens 3`. The compose file
(`stacks/gen-seat/compose.yaml`) carries the full history inline and is the current
authority; this section was stale against it for nine days.
Things that do **not** work, ruled out: `num_speculative_tokens=1` (corruption is
depth-independent — reproduces at n=1 and n=2, deterministically probed upstream);
@@ -643,6 +660,7 @@ day if followed:
| "Use modelopt, NOT compressed-tensors — compressed-tensors can't load the BF16 MTP head, 0% acceptance" | `docs/runbooks/heretic2-nvfp4-mtp-seat.md` §landmine 2 | **SUPERSEDED 2026-08-14.** The 0% was the missing `re:^mtp.*` ignore (§3.3), not the format. compressed-tensors + the ignore gives 47.783.2% acceptance, live. Use compressed-tensors. |
| "Abliteration desyncs the MTP head → uncensored models can't do MTP" | earlier auto-memory | **SUPERSEDED 2026-08-14.** A modest abliteration preserves MTP (83.7% at bf16). Test MTP on **bf16 first** to isolate abliteration from quant/graft confounds — and isolate before deleting a 50 GB source. |
| "NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer" | `reference_nvfp4_w4a4_granite_infeasible` | **NARROWED.** True for *uniform* W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs **with FP8 on attention and late layers** is fine and is the current default (§2). |
| "The Qwen3.8 MTP corruption is fixed by disabling prefix caching while keeping MTP; the gen seat runs APC-off" | this playbook §3.7 (now marked), earlier auto-memory | **SUPERSEDED 2026-08-17, and the staleness was only caught 2026-08-26.** APC-off passed a synthetic 7-turn probe and the operator still saw severe degeneration in real use; reverted the same day. The real cause was the **AEON W4A4 quant being defective** (~15-20% degenerate generations), with MTP / prefix-caching / gateway merely AMPLIFYING it (§3.8). The gen seat runs **MTP ON and prefix-caching ON** on the in-house mixed NVFP4+FP8 build — verified against the live container 2026-08-26. ⚠ The lesson inside the lesson: a *passing multi-turn probe* was not sufficient evidence either. |
| "transformers' Qwen3.5 DeltaNet linear-attention NaNs in bf16 without causal-conv1d; it is precision-driven cancellation and fp32 resolves it" | `services/coldfusion-abliteration/README.md`, `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md` | **SUPERSEDED 2026-08-20.** Precision was never the variable. The NaN came from **multi-GPU sharding** and **`expandable_segments`** (§3.9, §3.10); fp32 only made it rarer, which is worse than failing. On one GPU with a plain allocator, **bf16 is exactly deterministic through all 64 layers and generates coherent prose** — at 50 GB and 4.3× the throughput of the 111 GB fp32 it replaced. |
---