From 8389470898f184ba4edd3cf28f6b62b450e7b30d Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sat, 22 Aug 2026 01:16:27 -0700 Subject: [PATCH] feat(mog-sec): promote the DFlash2 configuration into the compose stack Operator approved after real-use testing. The experimental standalone container is retired and stacks/mog-sec is canonical again, with restart: unless-stopped so the configuration survives a reboot. Cutover verified against the container it replaces: KV pool 526,617 tokens at 1.10x concurrency, identical; zero restarts; both gateway aliases serving; DFlash2 confirmed drafting at k=7 with 231 draft tokens over 33 drafts; vision working at 2048x2048. One variable was deliberately dropped rather than carried over. The previous stack hardcoded PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, the validated container never set it, and the quant playbook records expandable_segments corrupting retained tensors in another context. The compose now defaults it empty via MOG_ALLOC_CONF. Promoting the stack as it stood would have shipped a variable the tested configuration did not have. The speculative config moves into a single MOG_SPEC_CONFIG carrying the whole JSON, because the two shapes are not interchangeable: dflash requires a model pointing at the drafter and MTP must not have one, so a method-plus-tokens template cannot express both. Also parameterised: MOG_DRAFT_MODEL, MOG_MM_PROCESSOR_KWARGS, MOG_MAX_NUM_BATCHED_TOKENS. The mm-processor image cap is now mandatory rather than incidental. The model's own preprocessor declares 4096x4096, which expands to 16384 image tokens and kills startup on builds that enforce the image-token count check. Adds the .env.example this stack never had, carrying the measured rationale for each value and the one-line rollback. --- .../2026-08-22-dflash2-spec-decode.md | 30 ++++++--- persistent-memory.md | 2 +- stacks/mog-sec/.env.example | 62 ++++++++++++++++--- stacks/mog-sec/compose.yaml | 35 +++++++++-- 4 files changed, 108 insertions(+), 21 deletions(-) diff --git a/persistent-memory.d/2026-08-22-dflash2-spec-decode.md b/persistent-memory.d/2026-08-22-dflash2-spec-decode.md index 7d2a931..5ba2c75 100644 --- a/persistent-memory.d/2026-08-22-dflash2-spec-decode.md +++ b/persistent-memory.d/2026-08-22-dflash2-spec-decode.md @@ -154,11 +154,28 @@ IndexError, workaround is disabling one). `rope_type: yarn`, `factor: 4.0`, `original_max_position_embeddings: 262144`, `max_position_embeddings: 1000000`. Context is a KV-memory choice, not a model limit. -## Live state — sec is NOT running from its compose stack +## Live state — PROMOTED to the compose stack 2026-08-22 -`vllm-sec-dflash2`, a **standalone container** on sec's port with sec's served names, so the -`sec` / `sec-reasoning` gateway aliases work unchanged. `/opt/docker/compose/mog-sec` is -**stopped but unmodified**. +**Operator-approved after real-use testing** ("performing very well"). The experimental +standalone container is gone; `stacks/mog-sec/` is canonical and `restart: unless-stopped` means +it survives reboots. Cutover verified: **KV pool 526,617 / 1.10x — identical to the container it +replaced**, restarts 0, both gateway aliases serving, DFlash2 confirmed drafting at k=7 +(231 draft tokens over 33 drafts), vision working. + +⚠ **One variable was deliberately REMOVED, not carried over.** The old stack hardcoded +`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`; the validated DFlash2 container never set it, +and playbook §3.10 records expandable_segments corrupting retained tensors elsewhere. The compose +now defaults it EMPTY (`MOG_ALLOC_CONF`). Promoting it as-was would have shipped a variable the +tested configuration did not have. + +**Compose is now parameterised for the shapes that differ:** `MOG_SPEC_CONFIG` carries the whole +speculative JSON (dflash needs `"model": "/drafter"`, MTP must not have one — a method+tokens +template cannot express both), plus `MOG_MM_PROCESSOR_KWARGS`, `MOG_DRAFT_MODEL`, +`MOG_MAX_NUM_BATCHED_TOKENS`, `MOG_ALLOC_CONF`. + +**ROLLBACK:** `.env.bak-pre-dflash2-20260822` and `compose.yaml.bak-pre-dflash2-20260822` on the +host; or one line — `MOG_SPEC_CONFIG={"method": "qwen3_5_mtp", "num_speculative_tokens": 3}` plus +the old `MOG_IMAGE`. | | production sec | current | |---|---|---| @@ -168,11 +185,8 @@ IndexError, workaround is disabling one). | KV pool | 418,218 (1.60×) | **526,617 (1.10×)** | | images | 4096² → 16,384 tok | **2048² → ~5,125 tok** (`--mm-processor-kwargs` size cap) | -**ROLLBACK is two commands:** `docker rm -f vllm-sec-dflash2` then `docker compose up -d` in -`/opt/docker/compose/mog-sec`. - ⚠ **`--gpu-memory-utilization 0.55` is the stable ceiling** while GPU1's other tenants are up. 0.58 sized KV at 594,172 then **OOM'd during CUDA graph capture** — the process reached 57.49 GiB against ~57.6 free. Real 1M context needs ~49 GiB of KV and therefore evicting most of GPU1. -Launcher: `/tmp/run_sec_dflash2.sh` on ana-ml2 (ephemeral — re-derive from this table if lost). +Canonical config: `stacks/mog-sec/{compose.yaml,.env.example}` in this repo. diff --git a/persistent-memory.md b/persistent-memory.md index 7255afa..4bafcdf 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -123,7 +123,7 @@ _As of 2026-08-21 (late) — **the big AI-seat overhaul session; three seats set - **FLEET CLEANUP:** LFM2.5 retired; reranker consolidated (**nevermore had been silently failing 8 days** — pinned to the retired `granite-4.1-8b` alias; repointed to `summarizer`/`reranker`); `:8002`/`:8014`/granite service retired; A3 promoted to `stacks/vllm`. - ⏳ **OPEN:** file the upstream vLLM issue (operator); OWUI image-tag drift (`:main` vs pinned v0.11.0); Cold-Fusion NVFP4 quants delete/keep; Brokkr duplicate `reranker-a3-bge-v2-m3` alias; `/tank` DEGRADED **70+ days** (parked, operator→colo w/ cold spare); **MANY commits unpushed** (push is operator's call)._ -- **🟡 DFLASH2 SPEC-DECODE — MEASURED, and `sec` IS CURRENTLY RUNNING ON IT (2026-08-22, experimental).** `incoai/Qwen3.8-27B-DFlash2` (2B block-diffusion drafter, Apache-2.0) merged into vLLM as **#52816** on 2026-08-21; method string is **`"dflash"`**. **✅ MEASURED on our stack** (the card only claims stock BF16 on H200): works with an **abliterated + NVFP4 `compressed-tensors` target on Blackwell sm_120**. gen: MTP k=3 **2.753** tok/forward @ 114.9 tok/s → DFlash2 k=7 **3.254** @ 131.9. sec: **2.676** @ 110.5 → **3.252** @ 130.0. **⭐ THE k=7 MTP CONTROL INVERTED THE OBVIOUS READ — do not "just raise num_speculative_tokens":** MTP k=7 improves acceptance (3.041) but **collapses throughput to 74.0 tok/s**, because our head is a single module run autoregressively so k tokens cost k sequential passes. DFlash2 wins by making depth cheap, NOT by drafting better (our MTP is *better* at position 0, 79.6% vs 75.4%). **⭐ The drafter is model-agnostic — 3.254 vs 3.252 across two different finetunes (0.06%)** — so ONE weights file serves both seats, but it is **EAGLE3-style coupled** (`load_model(target_model)`, taps target hidden states at layers 5/19/33/47/61) so **VRAM is 3.85 GB PER SEAT, never shared**. **⚠️ `sec` NOW RUNS FROM A STANDALONE CONTAINER `vllm-sec-dflash2`, NOT its compose stack** (stopped, unmodified) — on `nightly-e9d1398d`, DFlash2 k=7, **480,000 ctx / 526,617 KV**, 2048² vision. **ROLLBACK = `docker rm -f vllm-sec-dflash2` + `compose up -d`.** **⚠️ DEGENERATION: operator ruling 2026-08-22 — it lives in the UN-FIXED vLLM, not the weights. My "sec's stock-graft MTP head causes it" hypothesis is RETRACTED** (false dichotomy: an engine bug in BOTH seats is not exonerated by the seats being identical — it just can't explain a *difference*). **⭐⭐ AND THE UNDERLYING OBSERVATION IS NOT EVIDENCE: n=1 taken during a session with many concurrent changes.** Same caveat applies to the "coherent to 10k" sighting — same n, same conditions, opposite direction; the comparison is weak at BOTH ends. Production is **172 commits behind GDN spec-decode fix #53077**; **#51113 is in both builds — necessary but INSUFFICIENT.** Any real conclusion needs a held-still system: MTP k=3 on the new build, controlled. 🔶 Still unproven: that NVFP4 explains our ~3.25 vs the card's 4.10–5.46. ❌ Wrong turns recorded: version strings ≠ lineage (use compare API `behind_by`), Docker Hub push timestamps ≠ source freshness (grep the image), and **"1M needs YaRN, absent from config" is FALSE** — YaRN is fully present on the sec quant (`factor 4.0`, orig 262144). Full arc + epistemic labels → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md` +- **🟡 DFLASH2 SPEC-DECODE — MEASURED, and `sec` IS CURRENTLY RUNNING ON IT (2026-08-22, experimental).** `incoai/Qwen3.8-27B-DFlash2` (2B block-diffusion drafter, Apache-2.0) merged into vLLM as **#52816** on 2026-08-21; method string is **`"dflash"`**. **✅ MEASURED on our stack** (the card only claims stock BF16 on H200): works with an **abliterated + NVFP4 `compressed-tensors` target on Blackwell sm_120**. gen: MTP k=3 **2.753** tok/forward @ 114.9 tok/s → DFlash2 k=7 **3.254** @ 131.9. sec: **2.676** @ 110.5 → **3.252** @ 130.0. **⭐ THE k=7 MTP CONTROL INVERTED THE OBVIOUS READ — do not "just raise num_speculative_tokens":** MTP k=7 improves acceptance (3.041) but **collapses throughput to 74.0 tok/s**, because our head is a single module run autoregressively so k tokens cost k sequential passes. DFlash2 wins by making depth cheap, NOT by drafting better (our MTP is *better* at position 0, 79.6% vs 75.4%). **⭐ The drafter is model-agnostic — 3.254 vs 3.252 across two different finetunes (0.06%)** — so ONE weights file serves both seats, but it is **EAGLE3-style coupled** (`load_model(target_model)`, taps target hidden states at layers 5/19/33/47/61) so **VRAM is 3.85 GB PER SEAT, never shared**. **✅ PROMOTED TO THE COMPOSE STACK 2026-08-22** after operator real-use testing — `stacks/mog-sec/` canonical, `restart: unless-stopped`, survives reboot. `nightly-e9d1398d`, DFlash2 k=7, **480,000 ctx / 526,617 KV** (identical to the container it replaced), 2048² vision. ⚠ **`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` was DELIBERATELY DROPPED** — the validated container never had it and playbook §3.10 flags it; `MOG_ALLOC_CONF` now defaults empty. Speculative shape lives in ONE var `MOG_SPEC_CONFIG` (dflash needs `"model"`, MTP must not — no template expresses both). **ROLLBACK: `.env.bak-pre-dflash2-20260822` on the host, or swap `MOG_SPEC_CONFIG` + `MOG_IMAGE`.** **⚠️ DEGENERATION: operator ruling 2026-08-22 — it lives in the UN-FIXED vLLM, not the weights. My "sec's stock-graft MTP head causes it" hypothesis is RETRACTED** (false dichotomy: an engine bug in BOTH seats is not exonerated by the seats being identical — it just can't explain a *difference*). **⭐⭐ AND THE UNDERLYING OBSERVATION IS NOT EVIDENCE: n=1 taken during a session with many concurrent changes.** Same caveat applies to the "coherent to 10k" sighting — same n, same conditions, opposite direction; the comparison is weak at BOTH ends. Production is **172 commits behind GDN spec-decode fix #53077**; **#51113 is in both builds — necessary but INSUFFICIENT.** Any real conclusion needs a held-still system: MTP k=3 on the new build, controlled. 🔶 Still unproven: that NVFP4 explains our ~3.25 vs the card's 4.10–5.46. ❌ Wrong turns recorded: version strings ≠ lineage (use compare API `behind_by`), Docker Hub push timestamps ≠ source freshness (grep the image), and **"1M needs YaRN, absent from config" is FALSE** — YaRN is fully present on the sec quant (`factor 4.0`, orig 262144). Full arc + epistemic labels → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md` - **🟢 SPEACHES ASR — LIVE on irv-ml1 A6000 :8204 (2026-08-21, operator-approved).** OpenAI-compatible faster-whisper for **Eyra** (meeting recorder, `eyra-dev`). `Systran/faster-whisper-large-v3` + `Systran/faster-distil-whisper-large-v3`, fp16, both resident (`STT_MODEL_TTL=-1`), ~5.9 GB VRAM / 20 GB still free. **Deliberately co-exists with `parakeet` (:8765)** — parakeet is a TDT/transducer returning bare `{"text":…}` with **no `no_speech_prob` concept**, so it structurally cannot serve this consumer. **★ THE MEASURED FINDING IS WORTH MORE THAN THE DEPLOY: `no_speech_prob` ALONE IS A WEAK HALLUCINATION GATE.** Silence and pink room tone both produced the classic Whisper `"Thank you."` hallucination while `no_speech_prob` stayed **under 0.11** — a conventional `>0.6` threshold passes both through. `avg_logprob` separates ~6× better (−0.114 speech vs −0.650/−0.724 non-speech); `compression_ratio` 1.141 vs 0.556. **Consumers must gate on a composite.** (Synthetic inputs — shape of the separation, not calibration constants.) **VAD pinned OFF** (`_UNSTABLE_VAD_FILTER=False`) at consumer request — they VAD-gate upstream; consequence is the service will transcribe silence and does not defend itself. **★ IMAGE PINNED BY DIGEST, not `:latest-cuda`** — the VAD flag is an upstream-declared *unstable* var name, so a floating bump could rename it, restore VAD, and move gate semantics with **no error and no log line**; bumping = deliberate + re-run the checks. ⚠ Two gotchas: `PRELOAD_MODELS` **does not download** (only loads already-cached — use `POST /v1/models/{id}`), and the bind-mounted cache needs a `hub/` subdir or **every** `/v1/models` 500s `CacheNotFound` **while `/health` still returns 200**. ⚠ docker `device_ids: ["1"]` = A6000 (native CUDA inverts this — sees A6000 as `cuda:0`). ⏳ **NOT DONE by design: no LiteLLM gateway alias** — agreed sequencing was direct-first; adding it means editing the shared gateway + re-running fidelity. ⏳ Eyra's **diarization workload** (GPU pyannote, gated weights, own HF token) is a **future VRAM claim on this same A6000** — arrives at their diarize milestone. Canonical `stacks/speaches/`, commit `aa5863c`. diff --git a/stacks/mog-sec/.env.example b/stacks/mog-sec/.env.example index 61e29a8..3269abe 100644 --- a/stacks/mog-sec/.env.example +++ b/stacks/mog-sec/.env.example @@ -1,20 +1,66 @@ -# mog-sec tunables — pen-test seat (ana-ml2 GPU 1, :8019). Edit the real .env on -# the server, never commit it. -MOG_IMAGE=vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0 +# mog-sec — pen-test seat (ana-ml2 GPU1, :8019). Copy to .env on the host. +# +# Values below are the configuration VALIDATED 2026-08-22: DFlash2 speculative +# decoding on a newer vLLM, 480K context, 2048x2048 vision. Promoted from a +# standalone experimental container after real-use testing. + +# ── Image ─────────────────────────────────────────────────────────────────── +# Contains DFlash2 (#52816) AND GDN spec-decode fix #53077. Verified by +# ancestry, not by version string: this reports 0.26.1rc1.dev1102, which LOOKS +# older than the 0.27.2rc1.dev150 it replaced — that is a setuptools_scm +# tag-reachability artifact, not a downgrade. It is +259 commits, behind_by=0. +# ⚠ Bumping this is a CONFIG-COMPATIBILITY event, not just a version change. +# Newer transformers enforces model-config constraints older ones ignored. +MOG_IMAGE=vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013 + API_KEY= MOG_GPU_ID=1 - MOG_CONTAINER_NAME=vllm-mog-sec MOG_PORT=8019 MOG_SERVED_NAME=mog-sec-27b MOG_SERVED_NAME_THINK=mog-sec-27b-thinking MOG_MODEL=/tank/aimodels/mog-sec-27b-nvfp4-mixed MOG_QUANT=compressed-tensors -MOG_GPU_MEM_UTIL=0.44 -MOG_MAX_MODEL_LEN=262144 + +# ── Speculative decoding — DFlash2 ────────────────────────────────────────── +# 2B block-diffusion drafter, drafts a whole 8-token block in one pass. Unlike +# the MTP head it reads the TARGET's live hidden states (layers 5/19/33/47/61), +# so it is model-agnostic — the same weights measured identically against a +# different finetune (3.254 vs 3.252 accepted tok/forward, a 0.06% delta). +# Measured on this seat: 2.676 -> 3.252 tok/forward, 110.5 -> 130.0 tok/s. +# ⚠ The drafter is COUPLED to its target and lives in that engine's process: +# the weights file is shareable across seats, the 3.85 GB of VRAM is NOT. +MOG_DRAFT_MODEL=/tank/aimodels/qwen38-27b-dflash2-drafter +MOG_SPEC_CONFIG={"method": "dflash", "model": "/drafter", "num_speculative_tokens": 7} +# ROLLBACK to the previous behaviour — one line, plus MOG_IMAGE above: +# MOG_SPEC_CONFIG={"method": "qwen3_5_mtp", "num_speculative_tokens": 3} + +# ── Context and memory ────────────────────────────────────────────────────── +# The model carries a complete YaRN config (rope_type yarn, factor 4.0, +# original_max_position_embeddings 262144, max_position_embeddings 1000000), +# so context is a KV-MEMORY choice, not a model limit. +# ⚠ max-model-len must stay UNDER the KV pool or vLLM refuses to start. +# At 0.55 the pool is ~526,617 tokens -> 480,000 gives 1.10x concurrency. +# 262144 instead would give ~1.86x. Straight trade: context vs concurrency. +MOG_MAX_MODEL_LEN=480000 +# ⚠ 0.55 IS THE STABLE CEILING while GPU1's other tenants are up. 0.58 sized a +# bigger pool and then OOM'd during CUDA graph capture (process reached +# 57.49 GiB against ~57.6 free). Real 1M context needs ~49 GiB of KV and so +# requires evicting most of GPU1 — a fleet decision, not a flag. +MOG_GPU_MEM_UTIL=0.55 MOG_MAX_NUM_SEQS=16 +MOG_MAX_NUM_BATCHED_TOKENS=16384 MOG_KV_CACHE_DTYPE=fp8 + +# ── Vision ────────────────────────────────────────────────────────────────── +# 4194304 px = 2048x2048 -> ~5125 image tokens. See the compose comment: the +# model's own preprocessor declares 4096x4096, which is both wasteful and fatal +# on builds that enforce the image-token count check. +MOG_MM_PROCESSOR_KWARGS={"size": {"longest_edge": 4194304, "shortest_edge": 65536}} +MOG_LIMIT_MM={"image": 4} + +# ── Misc ──────────────────────────────────────────────────────────────────── MOG_REASONING_PARSER=qwen3 MOG_REASONING_EFFORT=medium -MOG_SPEC_METHOD=qwen3_5_mtp -MOG_SPEC_TOKENS=3 +# Intentionally EMPTY — the validated config ran without expandable_segments. +MOG_ALLOC_CONF= diff --git a/stacks/mog-sec/compose.yaml b/stacks/mog-sec/compose.yaml index b1b6046..79d1549 100644 --- a/stacks/mog-sec/compose.yaml +++ b/stacks/mog-sec/compose.yaml @@ -31,11 +31,20 @@ services: volumes: - /tank/aimodels/huggingface:/hfcache - ${MOG_MODEL:-/tank/aimodels/mog-sec-27b-nvfp4-mixed}:/model:ro + # DFlash2 speculative drafter. Mounted unconditionally — it is inert if + # MOG_SPEC_CONFIG selects an MTP method that does not reference /drafter. + - ${MOG_DRAFT_MODEL:-/tank/aimodels/qwen38-27b-dflash2-drafter}:/drafter:ro environment: - HF_HOME=/hfcache - HF_HUB_CACHE=/hfcache/hub - VLLM_API_KEY=${API_KEY:-} - - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True + # ⚠ DEFAULTS TO UNSET, deliberately. This seat previously hardcoded + # PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. The DFlash2 config + # validated 2026-08-22 ran WITHOUT it, and the quant playbook §3.10 + # records expandable_segments corrupting retained tensors in another + # context. Do not re-enable it casually — that would ship a variable the + # tested configuration did not have. + - PYTORCH_CUDA_ALLOC_CONF=${MOG_ALLOC_CONF:-} command: - /model - --served-model-name @@ -54,7 +63,9 @@ services: - --max-num-seqs - ${MOG_MAX_NUM_SEQS:-16} - --max-num-batched-tokens - - "16384" + # ⚠ Raising this costs peak-activation VRAM straight out of the KV pool + # (measured 2026-08-22: 16384 -> 32768 cost ~3 GiB of KV for no benefit). + - ${MOG_MAX_NUM_BATCHED_TOKENS:-16384} - --trust-remote-code - --dtype - auto @@ -65,7 +76,15 @@ services: - --enable-prefix-caching - --enable-chunked-prefill - --limit-mm-per-prompt - - '{"image": 4}' + - '${MOG_LIMIT_MM:-{"image": 4}}' + # ⚠ MANDATORY on a newer vLLM. The model's own preprocessor_config.json + # declares size.longest_edge = 16777216 px (4096x4096), which expands to + # 16384 image tokens — one image eating 6% of a 262K context, and enough + # to kill startup on builds that enforce the text-vs-ids count check. + # This caps the dummy profiling image AND real images. Cost scales as + # (edge/patch)^2 / merge^2, so 2048x2048 -> ~5125 tokens. + - --mm-processor-kwargs + - '${MOG_MM_PROCESSOR_KWARGS:-{"size": {"longest_edge": 4194304, "shortest_edge": 65536}}}' - --reasoning-parser - ${MOG_REASONING_PARSER:-qwen3} - --default-chat-template-kwargs @@ -73,8 +92,16 @@ services: - --enable-auto-tool-choice - --tool-call-parser - qwen3_coder + # ONE env var carrying the whole JSON, because the two speculative shapes + # are not interchangeable: dflash needs a "model" pointing at the drafter, + # MTP must NOT have one. A method+tokens template cannot express both. + # DFlash2 : {"method": "dflash", "model": "/drafter", "num_speculative_tokens": 7} + # MTP : {"method": "qwen3_5_mtp", "num_speculative_tokens": 3} + # ⚠ Comparing the two requires matching num_speculative_tokens — see the + # quant playbook §5.1: MTP runs a single-module head autoregressively, so + # deeper k improves acceptance and DESTROYS throughput. - --speculative-config - - '{"method": "${MOG_SPEC_METHOD:-qwen3_5_mtp}", "num_speculative_tokens": ${MOG_SPEC_TOKENS:-3}}' + - '${MOG_SPEC_CONFIG:-{"method": "qwen3_5_mtp", "num_speculative_tokens": 3}}' deploy: resources: reservations: