feat(mog-sec): promote the DFlash2 configuration into the compose stack

Operator approved after real-use testing. The experimental standalone
container is retired and stacks/mog-sec is canonical again, with
restart: unless-stopped so the configuration survives a reboot.

Cutover verified against the container it replaces: KV pool 526,617 tokens
at 1.10x concurrency, identical; zero restarts; both gateway aliases
serving; DFlash2 confirmed drafting at k=7 with 231 draft tokens over 33
drafts; vision working at 2048x2048.

One variable was deliberately dropped rather than carried over. The previous
stack hardcoded PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, the
validated container never set it, and the quant playbook records
expandable_segments corrupting retained tensors in another context. The
compose now defaults it empty via MOG_ALLOC_CONF. Promoting the stack as it
stood would have shipped a variable the tested configuration did not have.

The speculative config moves into a single MOG_SPEC_CONFIG carrying the
whole JSON, because the two shapes are not interchangeable: dflash requires
a model pointing at the drafter and MTP must not have one, so a
method-plus-tokens template cannot express both. Also parameterised:
MOG_DRAFT_MODEL, MOG_MM_PROCESSOR_KWARGS, MOG_MAX_NUM_BATCHED_TOKENS.

The mm-processor image cap is now mandatory rather than incidental. The
model's own preprocessor declares 4096x4096, which expands to 16384 image
tokens and kills startup on builds that enforce the image-token count check.

Adds the .env.example this stack never had, carrying the measured rationale
for each value and the one-line rollback.

persistent-memory.d/2026-08-22-dflash2-spec-decode.md

@@ -154,11 +154,28 @@ IndexError, workaround is disabling one).
   `rope_type: yarn`, `factor: 4.0`, `original_max_position_embeddings: 262144`,
   `max_position_embeddings: 1000000`. Context is a KV-memory choice, not a model limit.
 
-## Live state — sec is NOT running from its compose stack
+## Live state — PROMOTED to the compose stack 2026-08-22
 
-`vllm-sec-dflash2`, a **standalone container** on sec's port with sec's served names, so the
+**Operator-approved after real-use testing** ("performing very well"). The experimental
-`sec` / `sec-reasoning` gateway aliases work unchanged. `/opt/docker/compose/mog-sec` is
+standalone container is gone; `stacks/mog-sec/` is canonical and `restart: unless-stopped` means
-**stopped but unmodified**.
+it survives reboots. Cutover verified: **KV pool 526,617 / 1.10x — identical to the container it
+replaced**, restarts 0, both gateway aliases serving, DFlash2 confirmed drafting at k=7
+(231 draft tokens over 33 drafts), vision working.
+
+⚠ **One variable was deliberately REMOVED, not carried over.** The old stack hardcoded
+`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`; the validated DFlash2 container never set it,
+and playbook §3.10 records expandable_segments corrupting retained tensors elsewhere. The compose
+now defaults it EMPTY (`MOG_ALLOC_CONF`). Promoting it as-was would have shipped a variable the
+tested configuration did not have.
+
+**Compose is now parameterised for the shapes that differ:** `MOG_SPEC_CONFIG` carries the whole
+speculative JSON (dflash needs `"model": "/drafter"`, MTP must not have one — a method+tokens
+template cannot express both), plus `MOG_MM_PROCESSOR_KWARGS`, `MOG_DRAFT_MODEL`,
+`MOG_MAX_NUM_BATCHED_TOKENS`, `MOG_ALLOC_CONF`.
+
+**ROLLBACK:** `.env.bak-pre-dflash2-20260822` and `compose.yaml.bak-pre-dflash2-20260822` on the
+host; or one line — `MOG_SPEC_CONFIG={"method": "qwen3_5_mtp", "num_speculative_tokens": 3}` plus
+the old `MOG_IMAGE`.
 
 | | production sec | current |
 |---|---|---|
@@ -168,11 +185,8 @@ IndexError, workaround is disabling one).
 | KV pool | 418,218 (1.60×) | **526,617 (1.10×)** |
 | images | 4096² → 16,384 tok | **2048² → ~5,125 tok** (`--mm-processor-kwargs` size cap) |
 
-**ROLLBACK is two commands:** `docker rm -f vllm-sec-dflash2` then `docker compose up -d` in
-`/opt/docker/compose/mog-sec`.
-
 ⚠ **`--gpu-memory-utilization 0.55` is the stable ceiling** while GPU1's other tenants are up.
 0.58 sized KV at 594,172 then **OOM'd during CUDA graph capture** — the process reached 57.49 GiB
 against ~57.6 free. Real 1M context needs ~49 GiB of KV and therefore evicting most of GPU1.
 
-Launcher: `/tmp/run_sec_dflash2.sh` on ana-ml2 (ephemeral — re-derive from this table if lost).
+Canonical config: `stacks/mog-sec/{compose.yaml,.env.example}` in this repo.

persistent-memory.md

@@ -123,7 +123,7 @@ _As of 2026-08-21 (late) — **the big AI-seat overhaul session; three seats set
 - **FLEET CLEANUP:** LFM2.5 retired; reranker consolidated (**nevermore had been silently failing 8 days** — pinned to the retired `granite-4.1-8b` alias; repointed to `summarizer`/`reranker`); `:8002`/`:8014`/granite service retired; A3 promoted to `stacks/vllm`.
 - ⏳ **OPEN:** file the upstream vLLM issue (operator); OWUI image-tag drift (`:main` vs pinned v0.11.0); Cold-Fusion NVFP4 quants delete/keep; Brokkr duplicate `reranker-a3-bge-v2-m3` alias; `/tank` DEGRADED **70+ days** (parked, operator→colo w/ cold spare); **MANY commits unpushed** (push is operator's call)._
 
-- **🟡 DFLASH2 SPEC-DECODE — MEASURED, and `sec` IS CURRENTLY RUNNING ON IT (2026-08-22, experimental).** `incoai/Qwen3.8-27B-DFlash2` (2B block-diffusion drafter, Apache-2.0) merged into vLLM as **#52816** on 2026-08-21; method string is **`"dflash"`**. **✅ MEASURED on our stack** (the card only claims stock BF16 on H200): works with an **abliterated + NVFP4 `compressed-tensors` target on Blackwell sm_120**. gen: MTP k=3 **2.753** tok/forward @ 114.9 tok/s → DFlash2 k=7 **3.254** @ 131.9. sec: **2.676** @ 110.5 → **3.252** @ 130.0. **⭐ THE k=7 MTP CONTROL INVERTED THE OBVIOUS READ — do not "just raise num_speculative_tokens":** MTP k=7 improves acceptance (3.041) but **collapses throughput to 74.0 tok/s**, because our head is a single module run autoregressively so k tokens cost k sequential passes. DFlash2 wins by making depth cheap, NOT by drafting better (our MTP is *better* at position 0, 79.6% vs 75.4%). **⭐ The drafter is model-agnostic — 3.254 vs 3.252 across two different finetunes (0.06%)** — so ONE weights file serves both seats, but it is **EAGLE3-style coupled** (`load_model(target_model)`, taps target hidden states at layers 5/19/33/47/61) so **VRAM is 3.85 GB PER SEAT, never shared**. **⚠️ `sec` NOW RUNS FROM A STANDALONE CONTAINER `vllm-sec-dflash2`, NOT its compose stack** (stopped, unmodified) — on `nightly-e9d1398d`, DFlash2 k=7, **480,000 ctx / 526,617 KV**, 2048² vision. **ROLLBACK = `docker rm -f vllm-sec-dflash2` + `compose up -d`.** **⚠️ DEGENERATION: operator ruling 2026-08-22 — it lives in the UN-FIXED vLLM, not the weights. My "sec's stock-graft MTP head causes it" hypothesis is RETRACTED** (false dichotomy: an engine bug in BOTH seats is not exonerated by the seats being identical — it just can't explain a *difference*). **⭐⭐ AND THE UNDERLYING OBSERVATION IS NOT EVIDENCE: n=1 taken during a session with many concurrent changes.** Same caveat applies to the "coherent to 10k" sighting — same n, same conditions, opposite direction; the comparison is weak at BOTH ends. Production is **172 commits behind GDN spec-decode fix #53077**; **#51113 is in both builds — necessary but INSUFFICIENT.** Any real conclusion needs a held-still system: MTP k=3 on the new build, controlled. 🔶 Still unproven: that NVFP4 explains our ~3.25 vs the card's 4.10–5.46. ❌ Wrong turns recorded: version strings ≠ lineage (use compare API `behind_by`), Docker Hub push timestamps ≠ source freshness (grep the image), and **"1M needs YaRN, absent from config" is FALSE** — YaRN is fully present on the sec quant (`factor 4.0`, orig 262144). Full arc + epistemic labels → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md`
+- **🟡 DFLASH2 SPEC-DECODE — MEASURED, and `sec` IS CURRENTLY RUNNING ON IT (2026-08-22, experimental).** `incoai/Qwen3.8-27B-DFlash2` (2B block-diffusion drafter, Apache-2.0) merged into vLLM as **#52816** on 2026-08-21; method string is **`"dflash"`**. **✅ MEASURED on our stack** (the card only claims stock BF16 on H200): works with an **abliterated + NVFP4 `compressed-tensors` target on Blackwell sm_120**. gen: MTP k=3 **2.753** tok/forward @ 114.9 tok/s → DFlash2 k=7 **3.254** @ 131.9. sec: **2.676** @ 110.5 → **3.252** @ 130.0. **⭐ THE k=7 MTP CONTROL INVERTED THE OBVIOUS READ — do not "just raise num_speculative_tokens":** MTP k=7 improves acceptance (3.041) but **collapses throughput to 74.0 tok/s**, because our head is a single module run autoregressively so k tokens cost k sequential passes. DFlash2 wins by making depth cheap, NOT by drafting better (our MTP is *better* at position 0, 79.6% vs 75.4%). **⭐ The drafter is model-agnostic — 3.254 vs 3.252 across two different finetunes (0.06%)** — so ONE weights file serves both seats, but it is **EAGLE3-style coupled** (`load_model(target_model)`, taps target hidden states at layers 5/19/33/47/61) so **VRAM is 3.85 GB PER SEAT, never shared**. **✅ PROMOTED TO THE COMPOSE STACK 2026-08-22** after operator real-use testing — `stacks/mog-sec/` canonical, `restart: unless-stopped`, survives reboot. `nightly-e9d1398d`, DFlash2 k=7, **480,000 ctx / 526,617 KV** (identical to the container it replaced), 2048² vision. ⚠ **`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` was DELIBERATELY DROPPED** — the validated container never had it and playbook §3.10 flags it; `MOG_ALLOC_CONF` now defaults empty. Speculative shape lives in ONE var `MOG_SPEC_CONFIG` (dflash needs `"model"`, MTP must not — no template expresses both). **ROLLBACK: `.env.bak-pre-dflash2-20260822` on the host, or swap `MOG_SPEC_CONFIG` + `MOG_IMAGE`.** **⚠️ DEGENERATION: operator ruling 2026-08-22 — it lives in the UN-FIXED vLLM, not the weights. My "sec's stock-graft MTP head causes it" hypothesis is RETRACTED** (false dichotomy: an engine bug in BOTH seats is not exonerated by the seats being identical — it just can't explain a *difference*). **⭐⭐ AND THE UNDERLYING OBSERVATION IS NOT EVIDENCE: n=1 taken during a session with many concurrent changes.** Same caveat applies to the "coherent to 10k" sighting — same n, same conditions, opposite direction; the comparison is weak at BOTH ends. Production is **172 commits behind GDN spec-decode fix #53077**; **#51113 is in both builds — necessary but INSUFFICIENT.** Any real conclusion needs a held-still system: MTP k=3 on the new build, controlled. 🔶 Still unproven: that NVFP4 explains our ~3.25 vs the card's 4.10–5.46. ❌ Wrong turns recorded: version strings ≠ lineage (use compare API `behind_by`), Docker Hub push timestamps ≠ source freshness (grep the image), and **"1M needs YaRN, absent from config" is FALSE** — YaRN is fully present on the sec quant (`factor 4.0`, orig 262144). Full arc + epistemic labels → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md`
 
 - **🟢 SPEACHES ASR — LIVE on irv-ml1 A6000 :8204 (2026-08-21, operator-approved).** OpenAI-compatible faster-whisper for **Eyra** (meeting recorder, `eyra-dev`). `Systran/faster-whisper-large-v3` + `Systran/faster-distil-whisper-large-v3`, fp16, both resident (`STT_MODEL_TTL=-1`), ~5.9 GB VRAM / 20 GB still free. **Deliberately co-exists with `parakeet` (:8765)** — parakeet is a TDT/transducer returning bare `{"text":…}` with **no `no_speech_prob` concept**, so it structurally cannot serve this consumer. **★ THE MEASURED FINDING IS WORTH MORE THAN THE DEPLOY: `no_speech_prob` ALONE IS A WEAK HALLUCINATION GATE.** Silence and pink room tone both produced the classic Whisper `"Thank you."` hallucination while `no_speech_prob` stayed **under 0.11** — a conventional `>0.6` threshold passes both through. `avg_logprob` separates ~6× better (−0.114 speech vs −0.650/−0.724 non-speech); `compression_ratio` 1.141 vs 0.556. **Consumers must gate on a composite.** (Synthetic inputs — shape of the separation, not calibration constants.) **VAD pinned OFF** (`_UNSTABLE_VAD_FILTER=False`) at consumer request — they VAD-gate upstream; consequence is the service will transcribe silence and does not defend itself. **★ IMAGE PINNED BY DIGEST, not `:latest-cuda`** — the VAD flag is an upstream-declared *unstable* var name, so a floating bump could rename it, restore VAD, and move gate semantics with **no error and no log line**; bumping = deliberate + re-run the checks. ⚠ Two gotchas: `PRELOAD_MODELS` **does not download** (only loads already-cached — use `POST /v1/models/{id}`), and the bind-mounted cache needs a `hub/` subdir or **every** `/v1/models` 500s `CacheNotFound` **while `/health` still returns 200**. ⚠ docker `device_ids: ["1"]` = A6000 (native CUDA inverts this — sees A6000 as `cuda:0`). ⏳ **NOT DONE by design: no LiteLLM gateway alias** — agreed sequencing was direct-first; adding it means editing the shared gateway + re-running fidelity. ⏳ Eyra's **diarization workload** (GPU pyannote, gated weights, own HF token) is a **future VRAM claim on this same A6000** — arrives at their diarize milestone. Canonical `stacks/speaches/`, commit `aa5863c`.
 

stacks/mog-sec/.env.example

@@ -1,20 +1,66 @@
-# mog-sec tunables — pen-test seat (ana-ml2 GPU 1, :8019). Edit the real .env on
+# mog-sec — pen-test seat (ana-ml2 GPU1, :8019). Copy to .env on the host.
-# the server, never commit it.
+#
-MOG_IMAGE=vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0
+# Values below are the configuration VALIDATED 2026-08-22: DFlash2 speculative
+# decoding on a newer vLLM, 480K context, 2048x2048 vision. Promoted from a
+# standalone experimental container after real-use testing.
+
+# ── Image ───────────────────────────────────────────────────────────────────
+# Contains DFlash2 (#52816) AND GDN spec-decode fix #53077. Verified by
+# ancestry, not by version string: this reports 0.26.1rc1.dev1102, which LOOKS
+# older than the 0.27.2rc1.dev150 it replaced — that is a setuptools_scm
+# tag-reachability artifact, not a downgrade. It is +259 commits, behind_by=0.
+# ⚠ Bumping this is a CONFIG-COMPATIBILITY event, not just a version change.
+#   Newer transformers enforces model-config constraints older ones ignored.
+MOG_IMAGE=vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013
+
 API_KEY=
 MOG_GPU_ID=1
-
 MOG_CONTAINER_NAME=vllm-mog-sec
 MOG_PORT=8019
 MOG_SERVED_NAME=mog-sec-27b
 MOG_SERVED_NAME_THINK=mog-sec-27b-thinking
 MOG_MODEL=/tank/aimodels/mog-sec-27b-nvfp4-mixed
 MOG_QUANT=compressed-tensors
-MOG_GPU_MEM_UTIL=0.44
+
-MOG_MAX_MODEL_LEN=262144
+# ── Speculative decoding — DFlash2 ──────────────────────────────────────────
+# 2B block-diffusion drafter, drafts a whole 8-token block in one pass. Unlike
+# the MTP head it reads the TARGET's live hidden states (layers 5/19/33/47/61),
+# so it is model-agnostic — the same weights measured identically against a
+# different finetune (3.254 vs 3.252 accepted tok/forward, a 0.06% delta).
+# Measured on this seat: 2.676 -> 3.252 tok/forward, 110.5 -> 130.0 tok/s.
+# ⚠ The drafter is COUPLED to its target and lives in that engine's process:
+#   the weights file is shareable across seats, the 3.85 GB of VRAM is NOT.
+MOG_DRAFT_MODEL=/tank/aimodels/qwen38-27b-dflash2-drafter
+MOG_SPEC_CONFIG={"method": "dflash", "model": "/drafter", "num_speculative_tokens": 7}
+# ROLLBACK to the previous behaviour — one line, plus MOG_IMAGE above:
+# MOG_SPEC_CONFIG={"method": "qwen3_5_mtp", "num_speculative_tokens": 3}
+
+# ── Context and memory ──────────────────────────────────────────────────────
+# The model carries a complete YaRN config (rope_type yarn, factor 4.0,
+# original_max_position_embeddings 262144, max_position_embeddings 1000000),
+# so context is a KV-MEMORY choice, not a model limit.
+# ⚠ max-model-len must stay UNDER the KV pool or vLLM refuses to start.
+#   At 0.55 the pool is ~526,617 tokens -> 480,000 gives 1.10x concurrency.
+#   262144 instead would give ~1.86x. Straight trade: context vs concurrency.
+MOG_MAX_MODEL_LEN=480000
+# ⚠ 0.55 IS THE STABLE CEILING while GPU1's other tenants are up. 0.58 sized a
+#   bigger pool and then OOM'd during CUDA graph capture (process reached
+#   57.49 GiB against ~57.6 free). Real 1M context needs ~49 GiB of KV and so
+#   requires evicting most of GPU1 — a fleet decision, not a flag.
+MOG_GPU_MEM_UTIL=0.55
 MOG_MAX_NUM_SEQS=16
+MOG_MAX_NUM_BATCHED_TOKENS=16384
 MOG_KV_CACHE_DTYPE=fp8
+
+# ── Vision ──────────────────────────────────────────────────────────────────
+# 4194304 px = 2048x2048 -> ~5125 image tokens. See the compose comment: the
+# model's own preprocessor declares 4096x4096, which is both wasteful and fatal
+# on builds that enforce the image-token count check.
+MOG_MM_PROCESSOR_KWARGS={"size": {"longest_edge": 4194304, "shortest_edge": 65536}}
+MOG_LIMIT_MM={"image": 4}
+
+# ── Misc ────────────────────────────────────────────────────────────────────
 MOG_REASONING_PARSER=qwen3
 MOG_REASONING_EFFORT=medium
-MOG_SPEC_METHOD=qwen3_5_mtp
+# Intentionally EMPTY — the validated config ran without expandable_segments.
-MOG_SPEC_TOKENS=3
+MOG_ALLOC_CONF=

stacks/mog-sec/compose.yaml

@@ -31,11 +31,20 @@ services:
     volumes:
       - /tank/aimodels/huggingface:/hfcache
       - ${MOG_MODEL:-/tank/aimodels/mog-sec-27b-nvfp4-mixed}:/model:ro
+      # DFlash2 speculative drafter. Mounted unconditionally — it is inert if
+      # MOG_SPEC_CONFIG selects an MTP method that does not reference /drafter.
+      - ${MOG_DRAFT_MODEL:-/tank/aimodels/qwen38-27b-dflash2-drafter}:/drafter:ro
     environment:
       - HF_HOME=/hfcache
       - HF_HUB_CACHE=/hfcache/hub
       - VLLM_API_KEY=${API_KEY:-}
-      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
+      # ⚠ DEFAULTS TO UNSET, deliberately. This seat previously hardcoded
+      # PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. The DFlash2 config
+      # validated 2026-08-22 ran WITHOUT it, and the quant playbook §3.10
+      # records expandable_segments corrupting retained tensors in another
+      # context. Do not re-enable it casually — that would ship a variable the
+      # tested configuration did not have.
+      - PYTORCH_CUDA_ALLOC_CONF=${MOG_ALLOC_CONF:-}
     command:
       - /model
       - --served-model-name
@@ -54,7 +63,9 @@ services:
       - --max-num-seqs
       - ${MOG_MAX_NUM_SEQS:-16}
       - --max-num-batched-tokens
-      - "16384"
+      # ⚠ Raising this costs peak-activation VRAM straight out of the KV pool
+      # (measured 2026-08-22: 16384 -> 32768 cost ~3 GiB of KV for no benefit).
+      - ${MOG_MAX_NUM_BATCHED_TOKENS:-16384}
       - --trust-remote-code
       - --dtype
       - auto
@@ -65,7 +76,15 @@ services:
       - --enable-prefix-caching
       - --enable-chunked-prefill
       - --limit-mm-per-prompt
-      - '{"image": 4}'
+      - '${MOG_LIMIT_MM:-{"image": 4}}'
+      # ⚠ MANDATORY on a newer vLLM. The model's own preprocessor_config.json
+      # declares size.longest_edge = 16777216 px (4096x4096), which expands to
+      # 16384 image tokens — one image eating 6% of a 262K context, and enough
+      # to kill startup on builds that enforce the text-vs-ids count check.
+      # This caps the dummy profiling image AND real images. Cost scales as
+      # (edge/patch)^2 / merge^2, so 2048x2048 -> ~5125 tokens.
+      - --mm-processor-kwargs
+      - '${MOG_MM_PROCESSOR_KWARGS:-{"size": {"longest_edge": 4194304, "shortest_edge": 65536}}}'
       - --reasoning-parser
       - ${MOG_REASONING_PARSER:-qwen3}
       - --default-chat-template-kwargs
@@ -73,8 +92,16 @@ services:
       - --enable-auto-tool-choice
       - --tool-call-parser
       - qwen3_coder
+      # ONE env var carrying the whole JSON, because the two speculative shapes
+      # are not interchangeable: dflash needs a "model" pointing at the drafter,
+      # MTP must NOT have one. A method+tokens template cannot express both.
+      #   DFlash2 : {"method": "dflash", "model": "/drafter", "num_speculative_tokens": 7}
+      #   MTP     : {"method": "qwen3_5_mtp", "num_speculative_tokens": 3}
+      # ⚠ Comparing the two requires matching num_speculative_tokens — see the
+      # quant playbook §5.1: MTP runs a single-module head autoregressively, so
+      # deeper k improves acceptance and DESTROYS throughput.
       - --speculative-config
-      - '{"method": "${MOG_SPEC_METHOD:-qwen3_5_mtp}", "num_speculative_tokens": ${MOG_SPEC_TOKENS:-3}}'
+      - '${MOG_SPEC_CONFIG:-{"method": "qwen3_5_mtp", "num_speculative_tokens": 3}}'
     deploy:
       resources:
         reservations: