From 20ac53052b351edd06d91057231732595e5a12d2 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sat, 22 Aug 2026 00:58:35 -0700 Subject: [PATCH] memory: retract the MTP-head degeneration hypothesis; n=1 was never evidence Operator ruling: the multi-turn degeneration lives in the un-fixed vLLM, not in the weights. The hypothesis that sec's stock-graft MTP head causes it is withdrawn. Two failures produced it. First, a false dichotomy treated as a deduction: having verified gen and sec run an identical engine, I concluded config was eliminated and therefore the weights were responsible. That does not follow. An engine bug present in both seats is not exonerated by the seats being identical; it only means the engine cannot explain a difference between them. It can still explain the failure. Second, and more instructive, the difference being explained may not exist. The premise was a single operator observation made during a session with many concurrent changes. That cannot carry a causal claim, and it became the load-bearing support for a root-cause narrative it could not hold. The same caveat now attaches to the coherent-to-10k observation on the new build: same n, same uncontrolled conditions, opposite direction. The comparison is weak at both ends, so the file no longer presents either sighting as a result. What survives as measured fact is unchanged and still recorded: sec's MTP head is byte-identical to the uncensored base across all 15 tensors, gen's was abliterated in-band, and acceptance differs slightly. None of that is shown to cause degeneration. Adds the generalisable lesson: an observation made while many things are changing cannot support a causal conclusion. It is the inverse of the warning already in the gen-seat compose file, which guards against trusting a negative result from a synthetic probe; this guards against trusting a positive sighting from an uncontrolled session. --- .../2026-08-22-dflash2-spec-decode.md | 65 ++++++++++++++----- persistent-memory.md | 2 +- 2 files changed, 48 insertions(+), 19 deletions(-) diff --git a/persistent-memory.d/2026-08-22-dflash2-spec-decode.md b/persistent-memory.d/2026-08-22-dflash2-spec-decode.md index a15ec36..7d2a931 100644 --- a/persistent-memory.d/2026-08-22-dflash2-spec-decode.md +++ b/persistent-memory.d/2026-08-22-dflash2-spec-decode.md @@ -70,31 +70,60 @@ plausible (the drafter reads quantized hidden states at its five taps). Second c prompt distribution (ours general-purpose, theirs GSM8K/MATH/HumanEval/MBPP/MT-Bench). **Neither is confirmed.** Settling it needs a BF16 target seat (~56 GB) β€” a real GPU window. -## πŸ”Ά HYPOTHESIS β€” why sec degenerated and gen did not +## ❌ RETRACTED β€” the "MTP head mismatch causes the degeneration" hypothesis -Operator confirmed **sec degenerates at ~2k tokens as served**, gen does not, on identical -engines. Config was eliminated as a variable: +**Operator ruling, 2026-08-22: this hypothesis is WRONG. The degeneration lives in the un-fixed +vLLM, not in the weights.** Recorded here rather than deleted, because it was reasoned to +confidently enough that a future session could re-derive it. -- **Same vLLM image ID** `sha256:bd3236cff208…`, same live version `0.27.2rc1.dev150+g311b3513a` - read from inside both running processes (tag equality alone would not prove this). -- Same `qwen3_5_mtp` k=3, same `--enable-prefix-caching`, same `fp8` KV, same `float32` mamba - cache. Only deltas were `gpu-memory-utilization` 0.43 vs 0.44 and the served name. +**Two independent failures produced it, and the second is the instructive one:** -**The standing hypothesis is the MTP head.** Per our own provenance (verified with -`compare_mtp_head.py`): sec's head is **byte-identical to `qwen38-27b-uncensored-bf16`, all 15 -tensors** β€” a stock head against a security-finetuned body, because the -`Qwen3_5ForConditionalGeneration` wrapper never loads the head so the finetuning could not reach -it. gen's orcarouter head was **abliterated in-band by the author**, matched to its body. -Acceptance corroborates (gen 58.4% vs sec 55.9%). +1. **I treated a false dichotomy as a deduction.** Having verified gen and sec run an identical + engine (same image ID `sha256:bd3236cff208…`, same live version + `0.27.2rc1.dev150+g311b3513a` read from inside both processes, same flags bar + `gpu-memory-utilization` 0.43 vs 0.44), I concluded "config is eliminated, therefore it is the + weights." That does not follow. **An engine bug present in BOTH seats is not exonerated by the + two seats being identical** β€” it just means the engine cannot explain a *difference*. It can + still explain the *failure*. +2. **The difference I was explaining may not exist.** The premise was a single operator + observation of sec degenerating at ~2k, made during a session with many concurrent changes. + **n=1 under heavy concurrent modification is not evidence** β€” see the meta-lesson below. -**⚠ This is consistent with everything measured but is NOT proven.** Nobody has shown the head -mismatch *causes* the degeneration. +**What survives as fact** (measured, still true, just not causal): sec's MTP head *is* +byte-identical to `qwen38-27b-uncensored-bf16` across all 15 tensors β€” a stock head on a +security-finetuned body, because the `Qwen3_5ForConditionalGeneration` wrapper never loads the +head, so the finetuning could not reach it. gen's orcarouter head *was* abliterated in-band by +its author. Acceptance differs slightly (gen 58.4%, sec 55.9%). **All true. None of it shown to +cause multi-turn degeneration.** -## ⚠️ CONFOUNDED β€” what fixed sec is not yet known +**Current standing explanation: the degeneration is an engine bug in the un-fixed vLLM.** Both +production seats run `311b3513`, which is **172 commits behind GDN spec-decode fix #53077** +(merged 2026-08-20). `#51113` is present in that build and is therefore **necessary but +insufficient** on its own. + +## ⭐⭐ META-LESSON β€” n=1 during a busy session is not evidence + +The operator's own framing, and it generalises past this incident: **an observation made while +many things are being changed at once cannot carry a causal claim, no matter how confidently it +is reported.** Tonight that single observation became the load-bearing premise for a weights-side +hypothesis, a root-cause narrative, and very nearly a recommendation. + +This is the same failure the gen-seat compose file already warns about in different words β€” *"a +passing probe is NOT sufficient evidence"* β€” inverted. That note guards against trusting a +**negative** result from a synthetic test. This one guards against trusting a **positive** +sighting from an uncontrolled session. Both reduce to: **hold the system still, or do not draw +causal conclusions from it.** + +Applies equally to the "coherent to 10k" observation below β€” same n, same conditions, opposite +direction. Neither observation is worth more than the other. + +## ⚠️ CONFOUNDED β€” and the "before" state is itself unreliable sec now runs DFlash2 on a newer build and the operator reports **coherent to 10k tokens with -adversarial nonsense prompts**, where it degenerated at 2k before. **Two variables changed at -once:** +adversarial nonsense prompts**. ⚠ Treat this the same way as the 2k sighting it is being compared +against: **n=1, uncontrolled session, not evidence.** The comparison is weak on *both* ends. + +**Two variables changed at once:** 1. **Engine**: `311b3513` β†’ `e9d1398d`, **+259 commits, `behind_by=0`** (a strict superset), including GDN spec-decode fix **#53077** (merged 2026-08-20) that production is **172 commits diff --git a/persistent-memory.md b/persistent-memory.md index b0dd73f..7255afa 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -123,7 +123,7 @@ _As of 2026-08-21 (late) β€” **the big AI-seat overhaul session; three seats set - **FLEET CLEANUP:** LFM2.5 retired; reranker consolidated (**nevermore had been silently failing 8 days** β€” pinned to the retired `granite-4.1-8b` alias; repointed to `summarizer`/`reranker`); `:8002`/`:8014`/granite service retired; A3 promoted to `stacks/vllm`. - ⏳ **OPEN:** file the upstream vLLM issue (operator); OWUI image-tag drift (`:main` vs pinned v0.11.0); Cold-Fusion NVFP4 quants delete/keep; Brokkr duplicate `reranker-a3-bge-v2-m3` alias; `/tank` DEGRADED **70+ days** (parked, operatorβ†’colo w/ cold spare); **MANY commits unpushed** (push is operator's call)._ -- **🟑 DFLASH2 SPEC-DECODE β€” MEASURED, and `sec` IS CURRENTLY RUNNING ON IT (2026-08-22, experimental).** `incoai/Qwen3.8-27B-DFlash2` (2B block-diffusion drafter, Apache-2.0) merged into vLLM as **#52816** on 2026-08-21; method string is **`"dflash"`**. **βœ… MEASURED on our stack** (the card only claims stock BF16 on H200): works with an **abliterated + NVFP4 `compressed-tensors` target on Blackwell sm_120**. gen: MTP k=3 **2.753** tok/forward @ 114.9 tok/s β†’ DFlash2 k=7 **3.254** @ 131.9. sec: **2.676** @ 110.5 β†’ **3.252** @ 130.0. **⭐ THE k=7 MTP CONTROL INVERTED THE OBVIOUS READ β€” do not "just raise num_speculative_tokens":** MTP k=7 improves acceptance (3.041) but **collapses throughput to 74.0 tok/s**, because our head is a single module run autoregressively so k tokens cost k sequential passes. DFlash2 wins by making depth cheap, NOT by drafting better (our MTP is *better* at position 0, 79.6% vs 75.4%). **⭐ The drafter is model-agnostic β€” 3.254 vs 3.252 across two different finetunes (0.06%)** β€” so ONE weights file serves both seats, but it is **EAGLE3-style coupled** (`load_model(target_model)`, taps target hidden states at layers 5/19/33/47/61) so **VRAM is 3.85 GB PER SEAT, never shared**. **⚠️ `sec` NOW RUNS FROM A STANDALONE CONTAINER `vllm-sec-dflash2`, NOT its compose stack** (stopped, unmodified) β€” on `nightly-e9d1398d`, DFlash2 k=7, **480,000 ctx / 526,617 KV**, 2048Β² vision. **ROLLBACK = `docker rm -f vllm-sec-dflash2` + `compose up -d`.** Operator reports **coherent to 10k** where it degenerated at 2k β€” but **⚠️ CONFOUNDED: engine (+259 commits, incl. GDN fix #53077 that production is 172 behind) and drafter both changed; isolate by running MTP k=3 on the same build.** **#51113 is in BOTH builds β€” necessary but INSUFFICIENT.** πŸ”Ά Hypotheses NOT proven: that sec's stock-graft MTP head (byte-identical to the uncensored base, vs gen's in-band-abliterated one) *causes* the degeneration; that NVFP4 explains our ~3.25 vs the card's 4.10–5.46. ❌ Wrong turns recorded: version strings β‰  lineage (use compare API `behind_by`), Docker Hub push timestamps β‰  source freshness (grep the image), and **"1M needs YaRN, absent from config" is FALSE** β€” YaRN is fully present on the sec quant (`factor 4.0`, orig 262144). Full arc + epistemic labels β†’ `persistent-memory.d/2026-08-22-dflash2-spec-decode.md` +- **🟑 DFLASH2 SPEC-DECODE β€” MEASURED, and `sec` IS CURRENTLY RUNNING ON IT (2026-08-22, experimental).** `incoai/Qwen3.8-27B-DFlash2` (2B block-diffusion drafter, Apache-2.0) merged into vLLM as **#52816** on 2026-08-21; method string is **`"dflash"`**. **βœ… MEASURED on our stack** (the card only claims stock BF16 on H200): works with an **abliterated + NVFP4 `compressed-tensors` target on Blackwell sm_120**. gen: MTP k=3 **2.753** tok/forward @ 114.9 tok/s β†’ DFlash2 k=7 **3.254** @ 131.9. sec: **2.676** @ 110.5 β†’ **3.252** @ 130.0. **⭐ THE k=7 MTP CONTROL INVERTED THE OBVIOUS READ β€” do not "just raise num_speculative_tokens":** MTP k=7 improves acceptance (3.041) but **collapses throughput to 74.0 tok/s**, because our head is a single module run autoregressively so k tokens cost k sequential passes. DFlash2 wins by making depth cheap, NOT by drafting better (our MTP is *better* at position 0, 79.6% vs 75.4%). **⭐ The drafter is model-agnostic β€” 3.254 vs 3.252 across two different finetunes (0.06%)** β€” so ONE weights file serves both seats, but it is **EAGLE3-style coupled** (`load_model(target_model)`, taps target hidden states at layers 5/19/33/47/61) so **VRAM is 3.85 GB PER SEAT, never shared**. **⚠️ `sec` NOW RUNS FROM A STANDALONE CONTAINER `vllm-sec-dflash2`, NOT its compose stack** (stopped, unmodified) β€” on `nightly-e9d1398d`, DFlash2 k=7, **480,000 ctx / 526,617 KV**, 2048Β² vision. **ROLLBACK = `docker rm -f vllm-sec-dflash2` + `compose up -d`.** **⚠️ DEGENERATION: operator ruling 2026-08-22 β€” it lives in the UN-FIXED vLLM, not the weights. My "sec's stock-graft MTP head causes it" hypothesis is RETRACTED** (false dichotomy: an engine bug in BOTH seats is not exonerated by the seats being identical β€” it just can't explain a *difference*). **⭐⭐ AND THE UNDERLYING OBSERVATION IS NOT EVIDENCE: n=1 taken during a session with many concurrent changes.** Same caveat applies to the "coherent to 10k" sighting β€” same n, same conditions, opposite direction; the comparison is weak at BOTH ends. Production is **172 commits behind GDN spec-decode fix #53077**; **#51113 is in both builds β€” necessary but INSUFFICIENT.** Any real conclusion needs a held-still system: MTP k=3 on the new build, controlled. πŸ”Ά Still unproven: that NVFP4 explains our ~3.25 vs the card's 4.10–5.46. ❌ Wrong turns recorded: version strings β‰  lineage (use compare API `behind_by`), Docker Hub push timestamps β‰  source freshness (grep the image), and **"1M needs YaRN, absent from config" is FALSE** β€” YaRN is fully present on the sec quant (`factor 4.0`, orig 262144). Full arc + epistemic labels β†’ `persistent-memory.d/2026-08-22-dflash2-spec-decode.md` - **🟒 SPEACHES ASR β€” LIVE on irv-ml1 A6000 :8204 (2026-08-21, operator-approved).** OpenAI-compatible faster-whisper for **Eyra** (meeting recorder, `eyra-dev`). `Systran/faster-whisper-large-v3` + `Systran/faster-distil-whisper-large-v3`, fp16, both resident (`STT_MODEL_TTL=-1`), ~5.9 GB VRAM / 20 GB still free. **Deliberately co-exists with `parakeet` (:8765)** β€” parakeet is a TDT/transducer returning bare `{"text":…}` with **no `no_speech_prob` concept**, so it structurally cannot serve this consumer. **β˜… THE MEASURED FINDING IS WORTH MORE THAN THE DEPLOY: `no_speech_prob` ALONE IS A WEAK HALLUCINATION GATE.** Silence and pink room tone both produced the classic Whisper `"Thank you."` hallucination while `no_speech_prob` stayed **under 0.11** β€” a conventional `>0.6` threshold passes both through. `avg_logprob` separates ~6Γ— better (βˆ’0.114 speech vs βˆ’0.650/βˆ’0.724 non-speech); `compression_ratio` 1.141 vs 0.556. **Consumers must gate on a composite.** (Synthetic inputs β€” shape of the separation, not calibration constants.) **VAD pinned OFF** (`_UNSTABLE_VAD_FILTER=False`) at consumer request β€” they VAD-gate upstream; consequence is the service will transcribe silence and does not defend itself. **β˜… IMAGE PINNED BY DIGEST, not `:latest-cuda`** β€” the VAD flag is an upstream-declared *unstable* var name, so a floating bump could rename it, restore VAD, and move gate semantics with **no error and no log line**; bumping = deliberate + re-run the checks. ⚠ Two gotchas: `PRELOAD_MODELS` **does not download** (only loads already-cached β€” use `POST /v1/models/{id}`), and the bind-mounted cache needs a `hub/` subdir or **every** `/v1/models` 500s `CacheNotFound` **while `/health` still returns 200**. ⚠ docker `device_ids: ["1"]` = A6000 (native CUDA inverts this β€” sees A6000 as `cuda:0`). ⏳ **NOT DONE by design: no LiteLLM gateway alias** β€” agreed sequencing was direct-first; adding it means editing the shared gateway + re-running fidelity. ⏳ Eyra's **diarization workload** (GPU pyannote, gated weights, own HF token) is a **future VRAM claim on this same A6000** β€” arrives at their diarize milestone. Canonical `stacks/speaches/`, commit `aa5863c`.