diff --git a/persistent-memory.d/2026-08-22-dflash2-spec-decode.md b/persistent-memory.d/2026-08-22-dflash2-spec-decode.md new file mode 100644 index 0000000..a15ec36 --- /dev/null +++ b/persistent-memory.d/2026-08-22-dflash2-spec-decode.md @@ -0,0 +1,149 @@ +# DFlash2 speculative decoding — measured on our own stack (2026-08-22) + +Operator-driven session. **Read the epistemic labels.** During the chase we generalised from +observations that later proved wrong; this file separates what was *measured* from what remains +*hypothesis*, and records the wrong turns so nobody re-derives them. + +## What DFlash2 is + +A **2B draft model** (3.85 GB bf16) for speculative decoding against Qwen3.8-27B — +`incoai/Qwen3.8-27B-DFlash2`, Apache-2.0, blog `inco.ai/blog/dflash2`, upstream `z-lab/dflash`. +Block diffusion: drafts a whole 8-token block in one pass, with a candidate selector tracing a +path through per-slot top-K. Lossless (greedy matches the target). + +vLLM support merged **2026-08-21 05:27 UTC** as PR **#52816** (`b389ac29`). Method string is +**`"dflash"`**, not `dflash2`. + +## ✅ MEASURED — throughput and acceptance + +Single instrument (`specbench.py`, 8 fixed prompts, temp 0, max_tokens 256), delta against +vLLM's own `spec_decode` counters. The MTP k=3 numbers reproduce our recorded 58.4% / 55.3% +figures exactly, which is what validates the instrument. + +| seat | config | accepted tok/forward | throughput | +|---|---|---|---| +| gen (orcarouter) | MTP k=3 *(production)* | 2.753 | 114.9 tok/s | +| gen | MTP k=7 *(control)* | 3.041 | **74.0 tok/s** | +| gen | **DFlash2 k=7** | **3.254** | **131.9 tok/s** | +| sec (M.O.G.-SEC) | MTP k=3 *(production)* | 2.676 | 110.5 tok/s | +| sec | **DFlash2 k=7** | **3.252** | **130.0 tok/s** | + +**⭐ The k=7 MTP control was essential and inverted the obvious read.** Going deeper on MTP +*improves acceptance* (2.753 → 3.041) while **destroying throughput** (114.9 → 74.0). Our MTP +head is a single module (`mtp_num_hidden_layers=1`, only `mtp.layers.0`, 15 tensors) run +autoregressively, so k draft tokens cost k sequential forward passes. **"Just raise +num_speculative_tokens" is a trap** — without the control I would have recommended it. + +DFlash2's win is therefore **not better per-token acceptance** — our MTP is actually *better* at +position 0 (79.6% vs 75.4%). It is that block drafting makes depth nearly free. + +**⭐ The drafter is model-agnostic across finetunes — 3.254 (gen) vs 3.252 (sec), a 0.06% +difference**, with superimposable per-position curves. One drafter file on `/tank` serves both. + +## ✅ MEASURED — how DFlash2 runs (answers "can one drafter serve both seats?") + +**EAGLE3-style coupled, not standalone.** In vLLM: `load_model(self, target_model)` binds it to a +specific target object; `pass_hidden_states_to_model=True`; `gpu_model_runner` reads +`dflash_config.target_layer_ids` → `[i+1 …]` to register auxiliary hidden-state capture on the +target at layers **5, 19, 33, 47, 61**. It even reads the target's RoPE style at load. + +Consequences: +- **Weights file is shareable** (one download, both seats mount it) — gen and sec are + architecturally identical on every dimension the drafter needs: 64 layers (deepest tap 61), + hidden 5120, intermediate 17408, vocab 248,320 > mask token 248,070. +- **VRAM is NOT shareable — 3.85 GB per seat.** The drafter lives inside the target's engine + process, consuming hidden states mid-forward. Two seats are two processes; there is no + cross-process sharing mechanism and there could not be. + +## ✅ MEASURED — it works on our stack, which the card does not claim + +The card tests stock BF16 on an H200 with FlashAttention 3. Verified here instead: +**abliterated + NVFP4 `compressed-tensors` target ✓, Blackwell sm_120 ✓, DFlash2 CUDA graphs +captured ✓.** None of that was documented anywhere. + +## 🔶 HYPOTHESIS — why our acceptance trails the published numbers + +Both our targets land at ~3.25 accepted length against the card's 4.10–5.46 on stock BF16. +**Finetune drift is ruled out** — two *different* finetunes gave identical results to three +decimals. The shared variable is **NVFP4 quantization of the target**, which is mechanically +plausible (the drafter reads quantized hidden states at its five taps). Second candidate: +prompt distribution (ours general-purpose, theirs GSM8K/MATH/HumanEval/MBPP/MT-Bench). +**Neither is confirmed.** Settling it needs a BF16 target seat (~56 GB) — a real GPU window. + +## 🔶 HYPOTHESIS — why sec degenerated and gen did not + +Operator confirmed **sec degenerates at ~2k tokens as served**, gen does not, on identical +engines. Config was eliminated as a variable: + +- **Same vLLM image ID** `sha256:bd3236cff208…`, same live version `0.27.2rc1.dev150+g311b3513a` + read from inside both running processes (tag equality alone would not prove this). +- Same `qwen3_5_mtp` k=3, same `--enable-prefix-caching`, same `fp8` KV, same `float32` mamba + cache. Only deltas were `gpu-memory-utilization` 0.43 vs 0.44 and the served name. + +**The standing hypothesis is the MTP head.** Per our own provenance (verified with +`compare_mtp_head.py`): sec's head is **byte-identical to `qwen38-27b-uncensored-bf16`, all 15 +tensors** — a stock head against a security-finetuned body, because the +`Qwen3_5ForConditionalGeneration` wrapper never loads the head so the finetuning could not reach +it. gen's orcarouter head was **abliterated in-band by the author**, matched to its body. +Acceptance corroborates (gen 58.4% vs sec 55.9%). + +**⚠ This is consistent with everything measured but is NOT proven.** Nobody has shown the head +mismatch *causes* the degeneration. + +## ⚠️ CONFOUNDED — what fixed sec is not yet known + +sec now runs DFlash2 on a newer build and the operator reports **coherent to 10k tokens with +adversarial nonsense prompts**, where it degenerated at 2k before. **Two variables changed at +once:** + +1. **Engine**: `311b3513` → `e9d1398d`, **+259 commits, `behind_by=0`** (a strict superset), + including GDN spec-decode fix **#53077** (merged 2026-08-20) that production is **172 commits + behind**. +2. **Drafter**: frozen MTP head → DFlash2 reading live hidden states. + +**Isolating it = run MTP k=3 on the same new build.** Not yet done. + +**#51113 is present in BOTH builds** (verified by ancestry, `behind_by=0` each) — so the +"proper upstream fix" our compose comment credits is **necessary but insufficient**; sec ran it +and still degenerated. Related open upstream: **#53180** (quantized Qwen3.8-27B hybrid GDN + MTP +producing *silent* degenerate output, no fix), **#41884** (DFlash + prefix caching on hybrid, +IndexError, workaround is disabling one). + +## ❌ WRONG TURNS — do not repeat + +- **Version strings are not lineage.** The DFlash2 build reports `0.26.1rc1.dev1048` and our + production nightly `0.27.2rc1.dev150`, which *looks* like a regression. It is a setuptools_scm + tag-reachability artifact. **Use the GitHub compare API and check `behind_by`.** +- **Docker Hub push timestamps lie about source freshness.** `nightly-ba07e4a4` was *pushed* + 06:12 UTC, comfortably after the 05:27 merge — but *cut* from a 03:46 commit that predates it. + **Grep the image for the symbols you need.** Believing the timestamp would have cost an RP-seat + outage to serve a model the engine could not instantiate. +- **`--max-num-batched-tokens` was not the image truncation.** Raising it 16,384 → 32,768 on that + theory changed nothing and cost ~3 GiB of peak activation, which came straight out of the KV + pool. The cap was the tokenizer (§3.14 of the playbook). +- **"1M needs YaRN, absent from config" is FALSE for the sec quant.** It is fully present: + `rope_type: yarn`, `factor: 4.0`, `original_max_position_embeddings: 262144`, + `max_position_embeddings: 1000000`. Context is a KV-memory choice, not a model limit. + +## Live state — sec is NOT running from its compose stack + +`vllm-sec-dflash2`, a **standalone container** on sec's port with sec's served names, so the +`sec` / `sec-reasoning` gateway aliases work unchanged. `/opt/docker/compose/mog-sec` is +**stopped but unmodified**. + +| | production sec | current | +|---|---|---| +| image | `nightly-311b3513` | `nightly-e9d1398d` | +| speculation | MTP k=3 | **DFlash2 k=7**, drafter `/tank/aimodels/qwen38-27b-dflash2-drafter` | +| max-model-len | 262,144 | **480,000** | +| KV pool | 418,218 (1.60×) | **526,617 (1.10×)** | +| images | 4096² → 16,384 tok | **2048² → ~5,125 tok** (`--mm-processor-kwargs` size cap) | + +**ROLLBACK is two commands:** `docker rm -f vllm-sec-dflash2` then `docker compose up -d` in +`/opt/docker/compose/mog-sec`. + +⚠ **`--gpu-memory-utilization 0.55` is the stable ceiling** while GPU1's other tenants are up. +0.58 sized KV at 594,172 then **OOM'd during CUDA graph capture** — the process reached 57.49 GiB +against ~57.6 free. Real 1M context needs ~49 GiB of KV and therefore evicting most of GPU1. + +Launcher: `/tmp/run_sec_dflash2.sh` on ana-ml2 (ephemeral — re-derive from this table if lost). diff --git a/persistent-memory.md b/persistent-memory.md index 20dd7d6..b0dd73f 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -123,6 +123,8 @@ _As of 2026-08-21 (late) — **the big AI-seat overhaul session; three seats set - **FLEET CLEANUP:** LFM2.5 retired; reranker consolidated (**nevermore had been silently failing 8 days** — pinned to the retired `granite-4.1-8b` alias; repointed to `summarizer`/`reranker`); `:8002`/`:8014`/granite service retired; A3 promoted to `stacks/vllm`. - ⏳ **OPEN:** file the upstream vLLM issue (operator); OWUI image-tag drift (`:main` vs pinned v0.11.0); Cold-Fusion NVFP4 quants delete/keep; Brokkr duplicate `reranker-a3-bge-v2-m3` alias; `/tank` DEGRADED **70+ days** (parked, operator→colo w/ cold spare); **MANY commits unpushed** (push is operator's call)._ +- **🟡 DFLASH2 SPEC-DECODE — MEASURED, and `sec` IS CURRENTLY RUNNING ON IT (2026-08-22, experimental).** `incoai/Qwen3.8-27B-DFlash2` (2B block-diffusion drafter, Apache-2.0) merged into vLLM as **#52816** on 2026-08-21; method string is **`"dflash"`**. **✅ MEASURED on our stack** (the card only claims stock BF16 on H200): works with an **abliterated + NVFP4 `compressed-tensors` target on Blackwell sm_120**. gen: MTP k=3 **2.753** tok/forward @ 114.9 tok/s → DFlash2 k=7 **3.254** @ 131.9. sec: **2.676** @ 110.5 → **3.252** @ 130.0. **⭐ THE k=7 MTP CONTROL INVERTED THE OBVIOUS READ — do not "just raise num_speculative_tokens":** MTP k=7 improves acceptance (3.041) but **collapses throughput to 74.0 tok/s**, because our head is a single module run autoregressively so k tokens cost k sequential passes. DFlash2 wins by making depth cheap, NOT by drafting better (our MTP is *better* at position 0, 79.6% vs 75.4%). **⭐ The drafter is model-agnostic — 3.254 vs 3.252 across two different finetunes (0.06%)** — so ONE weights file serves both seats, but it is **EAGLE3-style coupled** (`load_model(target_model)`, taps target hidden states at layers 5/19/33/47/61) so **VRAM is 3.85 GB PER SEAT, never shared**. **⚠️ `sec` NOW RUNS FROM A STANDALONE CONTAINER `vllm-sec-dflash2`, NOT its compose stack** (stopped, unmodified) — on `nightly-e9d1398d`, DFlash2 k=7, **480,000 ctx / 526,617 KV**, 2048² vision. **ROLLBACK = `docker rm -f vllm-sec-dflash2` + `compose up -d`.** Operator reports **coherent to 10k** where it degenerated at 2k — but **⚠️ CONFOUNDED: engine (+259 commits, incl. GDN fix #53077 that production is 172 behind) and drafter both changed; isolate by running MTP k=3 on the same build.** **#51113 is in BOTH builds — necessary but INSUFFICIENT.** 🔶 Hypotheses NOT proven: that sec's stock-graft MTP head (byte-identical to the uncensored base, vs gen's in-band-abliterated one) *causes* the degeneration; that NVFP4 explains our ~3.25 vs the card's 4.10–5.46. ❌ Wrong turns recorded: version strings ≠ lineage (use compare API `behind_by`), Docker Hub push timestamps ≠ source freshness (grep the image), and **"1M needs YaRN, absent from config" is FALSE** — YaRN is fully present on the sec quant (`factor 4.0`, orig 262144). Full arc + epistemic labels → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md` + - **🟢 SPEACHES ASR — LIVE on irv-ml1 A6000 :8204 (2026-08-21, operator-approved).** OpenAI-compatible faster-whisper for **Eyra** (meeting recorder, `eyra-dev`). `Systran/faster-whisper-large-v3` + `Systran/faster-distil-whisper-large-v3`, fp16, both resident (`STT_MODEL_TTL=-1`), ~5.9 GB VRAM / 20 GB still free. **Deliberately co-exists with `parakeet` (:8765)** — parakeet is a TDT/transducer returning bare `{"text":…}` with **no `no_speech_prob` concept**, so it structurally cannot serve this consumer. **★ THE MEASURED FINDING IS WORTH MORE THAN THE DEPLOY: `no_speech_prob` ALONE IS A WEAK HALLUCINATION GATE.** Silence and pink room tone both produced the classic Whisper `"Thank you."` hallucination while `no_speech_prob` stayed **under 0.11** — a conventional `>0.6` threshold passes both through. `avg_logprob` separates ~6× better (−0.114 speech vs −0.650/−0.724 non-speech); `compression_ratio` 1.141 vs 0.556. **Consumers must gate on a composite.** (Synthetic inputs — shape of the separation, not calibration constants.) **VAD pinned OFF** (`_UNSTABLE_VAD_FILTER=False`) at consumer request — they VAD-gate upstream; consequence is the service will transcribe silence and does not defend itself. **★ IMAGE PINNED BY DIGEST, not `:latest-cuda`** — the VAD flag is an upstream-declared *unstable* var name, so a floating bump could rename it, restore VAD, and move gate semantics with **no error and no log line**; bumping = deliberate + re-run the checks. ⚠ Two gotchas: `PRELOAD_MODELS` **does not download** (only loads already-cached — use `POST /v1/models/{id}`), and the bind-mounted cache needs a `hub/` subdir or **every** `/v1/models` 500s `CacheNotFound` **while `/health` still returns 200**. ⚠ docker `device_ids: ["1"]` = A6000 (native CUDA inverts this — sees A6000 as `cuda:0`). ⏳ **NOT DONE by design: no LiteLLM gateway alias** — agreed sequencing was direct-first; adding it means editing the shared gateway + re-running fidelity. ⏳ Eyra's **diarization workload** (GPU pyannote, gated weights, own HF token) is a **future VRAM claim on this same A6000** — arrives at their diarize milestone. Canonical `stacks/speaches/`, commit `aa5863c`. - **🟢 ESH INTERNAL IPv6 — LIVE on two LANs 2026-08-21 (operator-directed), and the CGNAT static is a CARRIER problem.** Enabled `ipv6_interface_type: none → pd` + `ipv6_pd_interface: wan` on **`esh-userland`** (VLAN 10, SSID `PVC`) and **`esh-server`** (10.0.50.0/24) on ESH-UDMPM. Cityside delegates **`2607:73c0:402:1d00::/56`**; per-VLAN /64s userland `:1d00`/`:1d01`, esh-server `:1d02`. **Verified: `esh-docker-vm` egresses over v6 as its OWN address (`2607:73c0:402:1d02:be24:11ff:fe13:3a8`) — no NAT** — vs v4 which egresses as the shared CGNAT pool `154.50.58.1`. Zero disruption through both writes (all subsystems ok, 54 wireless clients, Open WebUI 200, DNS resolving). Originals backed up. **⚠ NH3 stays v6-OFF deliberately — its single `/64` is RESERVED FOR NETWORK MESHING** (operator). **★ THE v4 STATIC IS NOT PROVISIONED — proven, stop testing it.** A full gateway **reboot** (verified by observed down-transition + uptime reset 140,846s→183s) forced a fresh DHCP **DISCOVER** and Cityside returned the *identical* `100.104.3.250`. Also tried: physical cable pull (missed — WAN is **`eth8` = Port 9**, 2.5GE copper; ports 1–7 have **never carried a byte**), `renew-dhcp` (**not a valid devmgr verb**), `force-provision` (no WAN effect). **A DHCP *renew* could never have worked anyway — renewal PRESERVES the lease by design; only release/rebind or link-down forces a new DISCOVER.** Next step is a **Cityside ticket**: is the static applied or merely billed, and will it arrive as a DHCP reservation (a bounce picks it up) or a manual assignment (needs IP/mask/gw → `wan_type=static`)? **v6 firewall AUDITED and CORRECT** — default-deny WAN→internal for BOTH versions, only 4 hand-written policies (all IoT-scoped, none WAN-facing); my earlier "v6 strips CGNAT's accidental shielding" warning was **overstated and is retracted**. ⚠ audit was config-based, **never probed from outside** (no v6 vantage exists — NH3 has none by design). Flat-zone lateral-movement finding **parked, id 44** `esh-split-cameras-mgmt-out-of-the-flat-internal`. Cleanup: the dead `Cable Internet` UCI modem (last seen 08-18, replaced by Cityside; failover is the 5G on WAN2) was **deleted from the controller**, which cleared the lingering `wlan`/`lan` **error** states to ok. `vpn: error` persists and predates all of this. → auto-memory `reference_fleet_vpn_topology_ipv6`