memory: DFlash2 spec-decode measured; sec running on it (experimental, confounded)

Records the 2026-08-22 session with measured results, hypotheses, and wrong
turns explicitly separated -- the operator held this back while it was in
flight specifically so conjecture would not enter the record as fact.

MEASURED. DFlash2 works on an abliterated + NVFP4 compressed-tensors target
on Blackwell sm_120, which the model card does not claim (it tests stock
BF16 on H200). gen 2.753 -> 3.254 accepted tok/forward and 114.9 -> 131.9
tok/s; sec 2.676 -> 3.252 and 110.5 -> 130.0. The drafter is model-agnostic
across two different finetunes to 0.06%, but is EAGLE3-style coupled to its
target's hidden states, so the weights file is shareable while the 3.85 GB
of VRAM is per-seat.

The k=7 MTP control is the load-bearing result: raising MTP depth improves
acceptance and collapses throughput to 74.0 tok/s, because a single-module
head run autoregressively costs one forward pass per draft token. Without
that control the obvious recommendation would have been wrong.

CONFOUNDED. sec no longer degenerates at 2k, but the engine advanced 259
commits and the drafter changed at the same time. Isolating it means running
MTP k=3 on the new build. Also recorded: #51113 is present in both builds
and is therefore necessary but insufficient, since sec ran it and still
degenerated.

HYPOTHESES, labelled as such: that sec's stock-graft MTP head causes the
degeneration, and that NVFP4 explains the gap against published acceptance
figures. Neither is proven.

WRONG TURNS, recorded so they are not re-derived: version strings are not
lineage, Docker Hub push timestamps are not source freshness, and the claim
that 1M context needs YaRN absent from config is false for the sec quant.

Operationally important: sec is serving from a standalone container rather
than its compose stack, which is stopped but unmodified. Rollback is two
commands and is written down.

persistent-memory.d/2026-08-22-dflash2-spec-decode.md

@@ -0,0 +1,149 @@
+# DFlash2 speculative decoding — measured on our own stack (2026-08-22)
+
+Operator-driven session. **Read the epistemic labels.** During the chase we generalised from
+observations that later proved wrong; this file separates what was *measured* from what remains
+*hypothesis*, and records the wrong turns so nobody re-derives them.
+
+## What DFlash2 is
+
+A **2B draft model** (3.85 GB bf16) for speculative decoding against Qwen3.8-27B —
+`incoai/Qwen3.8-27B-DFlash2`, Apache-2.0, blog `inco.ai/blog/dflash2`, upstream `z-lab/dflash`.
+Block diffusion: drafts a whole 8-token block in one pass, with a candidate selector tracing a
+path through per-slot top-K. Lossless (greedy matches the target).
+
+vLLM support merged **2026-08-21 05:27 UTC** as PR **#52816** (`b389ac29`). Method string is
+**`"dflash"`**, not `dflash2`.
+
+## ✅ MEASURED — throughput and acceptance
+
+Single instrument (`specbench.py`, 8 fixed prompts, temp 0, max_tokens 256), delta against
+vLLM's own `spec_decode` counters. The MTP k=3 numbers reproduce our recorded 58.4% / 55.3%
+figures exactly, which is what validates the instrument.
+
+| seat | config | accepted tok/forward | throughput |
+|---|---|---|---|
+| gen (orcarouter) | MTP k=3 *(production)* | 2.753 | 114.9 tok/s |
+| gen | MTP k=7 *(control)* | 3.041 | **74.0 tok/s** |
+| gen | **DFlash2 k=7** | **3.254** | **131.9 tok/s** |
+| sec (M.O.G.-SEC) | MTP k=3 *(production)* | 2.676 | 110.5 tok/s |
+| sec | **DFlash2 k=7** | **3.252** | **130.0 tok/s** |
+
+**⭐ The k=7 MTP control was essential and inverted the obvious read.** Going deeper on MTP
+*improves acceptance* (2.753 → 3.041) while **destroying throughput** (114.9 → 74.0). Our MTP
+head is a single module (`mtp_num_hidden_layers=1`, only `mtp.layers.0`, 15 tensors) run
+autoregressively, so k draft tokens cost k sequential forward passes. **"Just raise
+num_speculative_tokens" is a trap** — without the control I would have recommended it.
+
+DFlash2's win is therefore **not better per-token acceptance** — our MTP is actually *better* at
+position 0 (79.6% vs 75.4%). It is that block drafting makes depth nearly free.
+
+**⭐ The drafter is model-agnostic across finetunes — 3.254 (gen) vs 3.252 (sec), a 0.06%
+difference**, with superimposable per-position curves. One drafter file on `/tank` serves both.
+
+## ✅ MEASURED — how DFlash2 runs (answers "can one drafter serve both seats?")
+
+**EAGLE3-style coupled, not standalone.** In vLLM: `load_model(self, target_model)` binds it to a
+specific target object; `pass_hidden_states_to_model=True`; `gpu_model_runner` reads
+`dflash_config.target_layer_ids` → `[i+1 …]` to register auxiliary hidden-state capture on the
+target at layers **5, 19, 33, 47, 61**. It even reads the target's RoPE style at load.
+
+Consequences:
+- **Weights file is shareable** (one download, both seats mount it) — gen and sec are
+  architecturally identical on every dimension the drafter needs: 64 layers (deepest tap 61),
+  hidden 5120, intermediate 17408, vocab 248,320 > mask token 248,070.
+- **VRAM is NOT shareable — 3.85 GB per seat.** The drafter lives inside the target's engine
+  process, consuming hidden states mid-forward. Two seats are two processes; there is no
+  cross-process sharing mechanism and there could not be.
+
+## ✅ MEASURED — it works on our stack, which the card does not claim
+
+The card tests stock BF16 on an H200 with FlashAttention 3. Verified here instead:
+**abliterated + NVFP4 `compressed-tensors` target ✓, Blackwell sm_120 ✓, DFlash2 CUDA graphs
+captured ✓.** None of that was documented anywhere.
+
+## 🔶 HYPOTHESIS — why our acceptance trails the published numbers
+
+Both our targets land at ~3.25 accepted length against the card's 4.10–5.46 on stock BF16.
+**Finetune drift is ruled out** — two *different* finetunes gave identical results to three
+decimals. The shared variable is **NVFP4 quantization of the target**, which is mechanically
+plausible (the drafter reads quantized hidden states at its five taps). Second candidate:
+prompt distribution (ours general-purpose, theirs GSM8K/MATH/HumanEval/MBPP/MT-Bench).
+**Neither is confirmed.** Settling it needs a BF16 target seat (~56 GB) — a real GPU window.
+
+## 🔶 HYPOTHESIS — why sec degenerated and gen did not
+
+Operator confirmed **sec degenerates at ~2k tokens as served**, gen does not, on identical
+engines. Config was eliminated as a variable:
+
+- **Same vLLM image ID** `sha256:bd3236cff208…`, same live version `0.27.2rc1.dev150+g311b3513a`
+  read from inside both running processes (tag equality alone would not prove this).
+- Same `qwen3_5_mtp` k=3, same `--enable-prefix-caching`, same `fp8` KV, same `float32` mamba
+  cache. Only deltas were `gpu-memory-utilization` 0.43 vs 0.44 and the served name.
+
+**The standing hypothesis is the MTP head.** Per our own provenance (verified with
+`compare_mtp_head.py`): sec's head is **byte-identical to `qwen38-27b-uncensored-bf16`, all 15
+tensors** — a stock head against a security-finetuned body, because the
+`Qwen3_5ForConditionalGeneration` wrapper never loads the head so the finetuning could not reach
+it. gen's orcarouter head was **abliterated in-band by the author**, matched to its body.
+Acceptance corroborates (gen 58.4% vs sec 55.9%).
+
+**⚠ This is consistent with everything measured but is NOT proven.** Nobody has shown the head
+mismatch *causes* the degeneration.
+
+## ⚠️ CONFOUNDED — what fixed sec is not yet known
+
+sec now runs DFlash2 on a newer build and the operator reports **coherent to 10k tokens with
+adversarial nonsense prompts**, where it degenerated at 2k before. **Two variables changed at
+once:**
+
+1. **Engine**: `311b3513` → `e9d1398d`, **+259 commits, `behind_by=0`** (a strict superset),
+   including GDN spec-decode fix **#53077** (merged 2026-08-20) that production is **172 commits
+   behind**.
+2. **Drafter**: frozen MTP head → DFlash2 reading live hidden states.
+
+**Isolating it = run MTP k=3 on the same new build.** Not yet done.
+
+**#51113 is present in BOTH builds** (verified by ancestry, `behind_by=0` each) — so the
+"proper upstream fix" our compose comment credits is **necessary but insufficient**; sec ran it
+and still degenerated. Related open upstream: **#53180** (quantized Qwen3.8-27B hybrid GDN + MTP
+producing *silent* degenerate output, no fix), **#41884** (DFlash + prefix caching on hybrid,
+IndexError, workaround is disabling one).
+
+## ❌ WRONG TURNS — do not repeat
+
+- **Version strings are not lineage.** The DFlash2 build reports `0.26.1rc1.dev1048` and our
+  production nightly `0.27.2rc1.dev150`, which *looks* like a regression. It is a setuptools_scm
+  tag-reachability artifact. **Use the GitHub compare API and check `behind_by`.**
+- **Docker Hub push timestamps lie about source freshness.** `nightly-ba07e4a4` was *pushed*
+  06:12 UTC, comfortably after the 05:27 merge — but *cut* from a 03:46 commit that predates it.
+  **Grep the image for the symbols you need.** Believing the timestamp would have cost an RP-seat
+  outage to serve a model the engine could not instantiate.
+- **`--max-num-batched-tokens` was not the image truncation.** Raising it 16,384 → 32,768 on that
+  theory changed nothing and cost ~3 GiB of peak activation, which came straight out of the KV
+  pool. The cap was the tokenizer (§3.14 of the playbook).
+- **"1M needs YaRN, absent from config" is FALSE for the sec quant.** It is fully present:
+  `rope_type: yarn`, `factor: 4.0`, `original_max_position_embeddings: 262144`,
+  `max_position_embeddings: 1000000`. Context is a KV-memory choice, not a model limit.
+
+## Live state — sec is NOT running from its compose stack
+
+`vllm-sec-dflash2`, a **standalone container** on sec's port with sec's served names, so the
+`sec` / `sec-reasoning` gateway aliases work unchanged. `/opt/docker/compose/mog-sec` is
+**stopped but unmodified**.
+
+| | production sec | current |
+|---|---|---|
+| image | `nightly-311b3513` | `nightly-e9d1398d` |
+| speculation | MTP k=3 | **DFlash2 k=7**, drafter `/tank/aimodels/qwen38-27b-dflash2-drafter` |
+| max-model-len | 262,144 | **480,000** |
+| KV pool | 418,218 (1.60×) | **526,617 (1.10×)** |
+| images | 4096² → 16,384 tok | **2048² → ~5,125 tok** (`--mm-processor-kwargs` size cap) |
+
+**ROLLBACK is two commands:** `docker rm -f vllm-sec-dflash2` then `docker compose up -d` in
+`/opt/docker/compose/mog-sec`.
+
+⚠ **`--gpu-memory-utilization 0.55` is the stable ceiling** while GPU1's other tenants are up.
+0.58 sized KV at 594,172 then **OOM'd during CUDA graph capture** — the process reached 57.49 GiB
+against ~57.6 free. Real 1M context needs ~49 GiB of KV and therefore evicting most of GPU1.
+
+Launcher: `/tmp/run_sec_dflash2.sh` on ana-ml2 (ephemeral — re-derive from this table if lost).

persistent-memory.md

@@ -123,6 +123,8 @@ _As of 2026-08-21 (late) — **the big AI-seat overhaul session; three seats set
 - **FLEET CLEANUP:** LFM2.5 retired; reranker consolidated (**nevermore had been silently failing 8 days** — pinned to the retired `granite-4.1-8b` alias; repointed to `summarizer`/`reranker`); `:8002`/`:8014`/granite service retired; A3 promoted to `stacks/vllm`.
 - ⏳ **OPEN:** file the upstream vLLM issue (operator); OWUI image-tag drift (`:main` vs pinned v0.11.0); Cold-Fusion NVFP4 quants delete/keep; Brokkr duplicate `reranker-a3-bge-v2-m3` alias; `/tank` DEGRADED **70+ days** (parked, operator→colo w/ cold spare); **MANY commits unpushed** (push is operator's call)._
 
+- **🟡 DFLASH2 SPEC-DECODE — MEASURED, and `sec` IS CURRENTLY RUNNING ON IT (2026-08-22, experimental).** `incoai/Qwen3.8-27B-DFlash2` (2B block-diffusion drafter, Apache-2.0) merged into vLLM as **#52816** on 2026-08-21; method string is **`"dflash"`**. **✅ MEASURED on our stack** (the card only claims stock BF16 on H200): works with an **abliterated + NVFP4 `compressed-tensors` target on Blackwell sm_120**. gen: MTP k=3 **2.753** tok/forward @ 114.9 tok/s → DFlash2 k=7 **3.254** @ 131.9. sec: **2.676** @ 110.5 → **3.252** @ 130.0. **⭐ THE k=7 MTP CONTROL INVERTED THE OBVIOUS READ — do not "just raise num_speculative_tokens":** MTP k=7 improves acceptance (3.041) but **collapses throughput to 74.0 tok/s**, because our head is a single module run autoregressively so k tokens cost k sequential passes. DFlash2 wins by making depth cheap, NOT by drafting better (our MTP is *better* at position 0, 79.6% vs 75.4%). **⭐ The drafter is model-agnostic — 3.254 vs 3.252 across two different finetunes (0.06%)** — so ONE weights file serves both seats, but it is **EAGLE3-style coupled** (`load_model(target_model)`, taps target hidden states at layers 5/19/33/47/61) so **VRAM is 3.85 GB PER SEAT, never shared**. **⚠️ `sec` NOW RUNS FROM A STANDALONE CONTAINER `vllm-sec-dflash2`, NOT its compose stack** (stopped, unmodified) — on `nightly-e9d1398d`, DFlash2 k=7, **480,000 ctx / 526,617 KV**, 2048² vision. **ROLLBACK = `docker rm -f vllm-sec-dflash2` + `compose up -d`.** Operator reports **coherent to 10k** where it degenerated at 2k — but **⚠️ CONFOUNDED: engine (+259 commits, incl. GDN fix #53077 that production is 172 behind) and drafter both changed; isolate by running MTP k=3 on the same build.** **#51113 is in BOTH builds — necessary but INSUFFICIENT.** 🔶 Hypotheses NOT proven: that sec's stock-graft MTP head (byte-identical to the uncensored base, vs gen's in-band-abliterated one) *causes* the degeneration; that NVFP4 explains our ~3.25 vs the card's 4.10–5.46. ❌ Wrong turns recorded: version strings ≠ lineage (use compare API `behind_by`), Docker Hub push timestamps ≠ source freshness (grep the image), and **"1M needs YaRN, absent from config" is FALSE** — YaRN is fully present on the sec quant (`factor 4.0`, orig 262144). Full arc + epistemic labels → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md`
+
 - **🟢 SPEACHES ASR — LIVE on irv-ml1 A6000 :8204 (2026-08-21, operator-approved).** OpenAI-compatible faster-whisper for **Eyra** (meeting recorder, `eyra-dev`). `Systran/faster-whisper-large-v3` + `Systran/faster-distil-whisper-large-v3`, fp16, both resident (`STT_MODEL_TTL=-1`), ~5.9 GB VRAM / 20 GB still free. **Deliberately co-exists with `parakeet` (:8765)** — parakeet is a TDT/transducer returning bare `{"text":…}` with **no `no_speech_prob` concept**, so it structurally cannot serve this consumer. **★ THE MEASURED FINDING IS WORTH MORE THAN THE DEPLOY: `no_speech_prob` ALONE IS A WEAK HALLUCINATION GATE.** Silence and pink room tone both produced the classic Whisper `"Thank you."` hallucination while `no_speech_prob` stayed **under 0.11** — a conventional `>0.6` threshold passes both through. `avg_logprob` separates ~6× better (−0.114 speech vs −0.650/−0.724 non-speech); `compression_ratio` 1.141 vs 0.556. **Consumers must gate on a composite.** (Synthetic inputs — shape of the separation, not calibration constants.) **VAD pinned OFF** (`_UNSTABLE_VAD_FILTER=False`) at consumer request — they VAD-gate upstream; consequence is the service will transcribe silence and does not defend itself. **★ IMAGE PINNED BY DIGEST, not `:latest-cuda`** — the VAD flag is an upstream-declared *unstable* var name, so a floating bump could rename it, restore VAD, and move gate semantics with **no error and no log line**; bumping = deliberate + re-run the checks. ⚠ Two gotchas: `PRELOAD_MODELS` **does not download** (only loads already-cached — use `POST /v1/models/{id}`), and the bind-mounted cache needs a `hub/` subdir or **every** `/v1/models` 500s `CacheNotFound` **while `/health` still returns 200**. ⚠ docker `device_ids: ["1"]` = A6000 (native CUDA inverts this — sees A6000 as `cuda:0`). ⏳ **NOT DONE by design: no LiteLLM gateway alias** — agreed sequencing was direct-first; adding it means editing the shared gateway + re-running fidelity. ⏳ Eyra's **diarization workload** (GPU pyannote, gated weights, own HF token) is a **future VRAM claim on this same A6000** — arrives at their diarize milestone. Canonical `stacks/speaches/`, commit `aa5863c`.
 
 - **🟢 ESH INTERNAL IPv6 — LIVE on two LANs 2026-08-21 (operator-directed), and the CGNAT static is a CARRIER problem.** Enabled `ipv6_interface_type: none → pd` + `ipv6_pd_interface: wan` on **`esh-userland`** (VLAN 10, SSID `PVC`) and **`esh-server`** (10.0.50.0/24) on ESH-UDMPM. Cityside delegates **`2607:73c0:402:1d00::/56`**; per-VLAN /64s userland `:1d00`/`:1d01`, esh-server `:1d02`. **Verified: `esh-docker-vm` egresses over v6 as its OWN address (`2607:73c0:402:1d02:be24:11ff:fe13:3a8`) — no NAT** — vs v4 which egresses as the shared CGNAT pool `154.50.58.1`. Zero disruption through both writes (all subsystems ok, 54 wireless clients, Open WebUI 200, DNS resolving). Originals backed up. **⚠ NH3 stays v6-OFF deliberately — its single `/64` is RESERVED FOR NETWORK MESHING** (operator). **★ THE v4 STATIC IS NOT PROVISIONED — proven, stop testing it.** A full gateway **reboot** (verified by observed down-transition + uptime reset 140,846s→183s) forced a fresh DHCP **DISCOVER** and Cityside returned the *identical* `100.104.3.250`. Also tried: physical cable pull (missed — WAN is **`eth8` = Port 9**, 2.5GE copper; ports 1–7 have **never carried a byte**), `renew-dhcp` (**not a valid devmgr verb**), `force-provision` (no WAN effect). **A DHCP *renew* could never have worked anyway — renewal PRESERVES the lease by design; only release/rebind or link-down forces a new DISCOVER.** Next step is a **Cityside ticket**: is the static applied or merely billed, and will it arrive as a DHCP reservation (a bounce picks it up) or a manual assignment (needs IP/mask/gw → `wan_type=static`)? **v6 firewall AUDITED and CORRECT** — default-deny WAN→internal for BOTH versions, only 4 hand-written policies (all IoT-scoped, none WAN-facing); my earlier "v6 strips CGNAT's accidental shielding" warning was **overstated and is retracted**. ⚠ audit was config-based, **never probed from outside** (no v6 vantage exists — NH3 has none by design). Flat-zone lateral-movement finding **parked, id 44** `esh-split-cameras-mgmt-out-of-the-flat-internal`. Cleanup: the dead `Cable Internet` UCI modem (last seen 08-18, replaced by Cityside; failover is the 5G on WAN2) was **deleted from the controller**, which cleared the lingering `wlan`/`lan` **error** states to ok. `vpn: error` persists and predates all of this. → auto-memory `reference_fleet_vpn_topology_ipv6`