From 9d36c74572498a8eec720f9196e22999707c1fed Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 15 Sep 2026 10:08:57 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20refresh=20=E2=80=94=20bree?= =?UTF-8?q?ze=20settled,=20util=20does=20not=20predict=20residency,=20hand?= =?UTF-8?q?off=20defect?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Incremental over 653f7fb, covering the three turns since. breeze-tts stays on irv-ml1 and the TTS-stack move is parked at id 75. Adds the full fv-ml1 per-seat residency table and the shuffle arithmetic behind the recommendation: every arrangement that makes real room on GPU 1 spends the GPU 3 reserve anyway, so if breeze ever moves it should go straight to GPU 3 rather than evacuating three working seats for the same cost plus a migration. The finding worth keeping is that --gpu-memory-utilization does not predict resident VRAM, and is wrong in both directions: cyberprev at util 0.40 holds 8 GB more than its fraction, gen-small at 0.48 holds 10 GB less. An hour earlier in the session I reasoned about placement from those fractions and would have been 8 GB out. Read nvidia-smi --query-compute-apps instead. Also records a characterized defect in this skill's own handoff generator. Across two runs in one session it converted operator-deferred items into an imperative next-steps list and twice invited the next session to commit files marked as predating it. It fails in the blind spot of the documented fail-loud posture: the output is structurally valid and exits 0 while inverting the intent. That matters more than an ordinary bad summary because the handoff is the one artifact a fresh context reads as instruction. Both runs were corrected in-session; not yet reported to the skill's owner. --- .../2026-09-15-breeze-placement-sizing.md | 37 +++++++++++++++++++ .../2026-09-15-silent-wrong-answer-pattern.md | 28 ++++++++++++++ persistent-memory.md | 16 +++++--- 3 files changed, 75 insertions(+), 6 deletions(-) diff --git a/persistent-memory.d/2026-09-15-breeze-placement-sizing.md b/persistent-memory.d/2026-09-15-breeze-placement-sizing.md index 8609d1f..e5177f3 100644 --- a/persistent-memory.d/2026-09-15-breeze-placement-sizing.md +++ b/persistent-memory.d/2026-09-15-breeze-placement-sizing.md @@ -73,6 +73,43 @@ GPU 1 is the constrained card at **0.975 committed / 4,336 MiB free**; GPU 0 has and `tts-gateway` reach each other by name only through `extra_hosts` pins. Those pins travel with them and need re-pointing at the new host. +## fv-ml1 per-seat residency, measured 2026-09-15 (97,887 MiB per card) + +| GPU | seats (resident MiB) | resident | free | +|---|---|---|---| +| **0** | cyberprev 47,124 · gen-small 36,942 · parakeet 1,178 | 85,244 | **11,982** | +| **1** | meromero 44,056 · erp 27,748 · reward 9,192 · coder 6,292 · embed 3,472 · rerank 2,114 | 92,874 | **4,336** | +| **2** | flash-next 95,344 | 95,344 | 1,767 | +| **3** | — (deliberate reserve) | 2 | **97,247** | + +## ⚠⚠ `--gpu-memory-utilization` DOES NOT PREDICT RESIDENCY — measure, don't compute + +Capacity planning off the configured util fractions is wrong **in both directions**: + + vllm-cyberprev util 0.40 -> expect ~39,155 MiB ACTUAL 47,124 (+8.0 GB OVER) + vllm-gen-small util 0.48 -> expect ~46,986 MiB ACTUAL 36,942 (-10.0 GB UNDER) + +An hour earlier in this session I reasoned about placement from the util fractions and +would have been ~8 GB wrong on one card. ⭐ **`util` is a ceiling the engine may not +reach and evidently can exceed** (weights, activations and non-KV allocations sit +outside the KV budget the fraction governs). Always read +`nvidia-smi --query-compute-apps` for the real number. + +## Does a shuffle make room for breeze? — worked through + +- **GPU 0 today: YES, no shuffle.** 11,982 free vs breeze 10,316 → **1.7 GB margin**, + against a tenant that grew +1.1 GB in an hour, on the live chat path. Fits; thin. +- **GPU 1 today: NO.** 4,336 free. +- **Evacuate embed+rerank+reward (14,778 MiB) → GPU 1 has 19,114 free**, breeze fits + with ~8.8 GB headroom. **But there is nowhere to put those three except GPU 3** — + 14,778 does not fit GPU 0's 11,982, and GPU 2 has 1,767. +- **Partial shuffle fails:** embed+rerank (5,586) to GPU 0 leaves GPU 1 at 9,922 free — + **~400 MB short** of breeze, and breeze is growing. + +⭐ **Conclusion: every arrangement that makes real room on GPU 1 spends the GPU 3 +reserve anyway.** So if breeze ever goes to FV, put it **straight on GPU 3** rather than +evacuating three working seats to achieve the same cost plus a migration. + ## If consolidation onto FV is the goal anyway **GPU 3** fits it comfortably (97,247 MiB free) and has none of the margin problem — but diff --git a/persistent-memory.d/2026-09-15-silent-wrong-answer-pattern.md b/persistent-memory.d/2026-09-15-silent-wrong-answer-pattern.md index c1d6bf8..d53a525 100644 --- a/persistent-memory.d/2026-09-15-silent-wrong-answer-pattern.md +++ b/persistent-memory.d/2026-09-15-silent-wrong-answer-pattern.md @@ -111,6 +111,34 @@ possible answer, and it was the right one here. problem; a refusal naming permission rules or auto-mode is a harness one. Different fixes, and neither applies when nothing failed. +## ⚠ CHARACTERIZED DEFECT: `/snapshot`'s handoff generator turns deferred items into orders + +**n=2, same session, reproducible.** `snapshot_handoff.py` (gen-small) reliably converts +"open, operator-deferred, not blocking" into an imperative **Next steps** list, and twice +invited the next session to commit files explicitly marked as predating the session. + + run 1: "Execute deferred operator tasks: AI-tab Dormant regrouping, nconnect=8, + fused MoE (park id 47)" + "Commit graphify-out/… if they are ready" + run 2: six next-steps, FIVE of them deferred/parked items presented as actions, + + the same commit invitation + +⚠ **It fails silently in the skill's blind spot.** The documented failure posture is +fail-loud-fall-back — unreachable gateway, timeout, truncation, missing section → write +nothing, exit non-zero. **A structurally valid handoff whose content inverts the +operator's intent passes every one of those checks** and exits 0. + +⚠ **And this is the one artifact a fresh context inherits as instruction.** It is read +immediately after `/clear`, before any other framing, and its Next steps read as a +mandate. A wrong one here is not a bad summary; it is a fresh session going and doing +belayed work. + +**Mitigation until fixed: read the generated handoff before accepting it**, and invert +any deferred item into an explicit *do NOT*. Both runs this session were corrected +in-session. Owner is the `/snapshot` skill (galdrabok); not reported yet. + +⭐ Same family as everything above — the instrument produced a plausible artifact and +the plausibility is exactly what makes it dangerous. + ## Related `2026-09-15-talk-v10-deploy.md` (#2, and the gate built for it), diff --git a/persistent-memory.md b/persistent-memory.md index f98c601..fd42b29 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-15 ~09:30 PT (Parakeet STT live on fv-ml1 GPU 0 + LiteLLM `ext-stt`; `svos_miranda` LIVE in Hermes; talk v10 deployed; irv-ml1 dead-address sweep COMPLETE; secrets-broker concurrency bug fixed. Nothing blocked, nothing mid-flight.)_ +_Last updated: 2026-09-15 ~09:45 PT (Parakeet STT live on fv-ml1 GPU 0; `svos_miranda` LIVE in Hermes; talk v10 deployed; irv-ml1 dead-address sweep COMPLETE; secrets-broker concurrency fixed; breeze stays on irv-ml1, TTS-stack move parked at id 75. Nothing blocked, nothing mid-flight.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -115,24 +115,28 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-09-15 ~09:30 PT._ +_As of 2026-09-15 ~09:45 PT._ -### Nothing is blocked and nothing is mid-flight. Both of the previous session's named jobs closed, plus six unplanned pieces of work. +### Nothing is in flight and nothing is blocked. Both of the previous session's named jobs closed, plus six unplanned pieces of work. **Closed this session:** -1. **Parakeet STT live** — fv-ml1 **GPU 0** (not GPU 3), port 8300, v3 int8 25-language model, behind LiteLLM `ext-stt` / `whisper-1`. ⚠ Placed on GPU 3 first; operator corrected it — a ~800 MiB seat belongs on the card with the most uncommitted headroom, not on the one pristine 96 GB card, because vLLM sizes KV against TOTAL VRAM. **GPU 3 is now a deliberate reserve at 2 MiB.** -2. **`svos_miranda` LIVE in Hermes** — gateway restarted, 29 toolsets, Miranda scoped to exactly 8 tools, operator's own surface intact at 46. ⚠ `agent.disabled_toolsets` is DELETED and stays out (operator: "i dont want the tools disabled everywhere"); svos-dev fixed their roster check at `c9d2a96`. SVOS restarted itself; both roster lines verified. +1. **Parakeet STT live** — fv-ml1 **GPU 0**, port 8300, v3 int8 25-language model, behind LiteLLM `ext-stt` / `whisper-1`. ⚠ Placed on GPU 3 first; operator corrected it — a ~800 MiB seat belongs on the card with the most uncommitted headroom, not on the one pristine 96 GB card, because vLLM sizes KV against TOTAL VRAM. **GPU 3 is a deliberate reserve at 2 MiB.** +2. **`svos_miranda` LIVE in Hermes** — gateway restarted, 29 toolsets, Miranda scoped to exactly 8, operator's own surface intact at 46. ⚠ `agent.disabled_toolsets` is DELETED and stays out (operator ruling); svos-dev fixed their roster check at `c9d2a96`. SVOS restarted itself; both roster lines verified. 3. **talk v10 deployed** on nh3-dev :8092 — push-to-talk STT through `ext-stt`, barge-in. First consumer of the Parakeet seat. 4. **irv-ml1 dead-address sweep DONE** — 0 of 112 Homepage cards on `10.100.79.3`, was 9. Found and fixed four live breakages on OTHER hosts (Open WebUI TTS, asset-engine, skaldsong x2). 5. **`secret` concurrency bug fixed** — parallel `secret get` returned empty with exit 0. Command-level lock + empty-value guard + `find()` no longer coercing empty stdout to `[]`. `~/.local/bin/secret` is now a symlink, was a stale copy. 6. **Retired:** irv-ml1 parakeet (lost tts-dev's bench) and voice-studio (dots obsoleted by Breeze). -**Open, all operator-deferred, none blocking:** the AI-tab Dormant regrouping (belayed), `nconnect=8` on /mnt/smithy (deferred), fused MoE kernel path (park id 47). `speaches`'s label claims `:8204`, which is breeze-tts's live port — a latent conflict if anyone starts it. +**Settled at the end of the session:** breeze-tts **stays on irv-ml1**; moving it plus `bragi` and `tts-gateway` to fv-ml1 is **parked at id 75**, triggered on evacuating embed/rerank/reward. Full fv-ml1 per-seat residency measured — see the breeze detail file. + +**Open, all operator-deferred, none blocking:** the AI-tab Dormant regrouping (belayed), `nconnect=8` on /mnt/smithy (deferred), fused MoE kernel path (park id 47), TTS-stack move (park id 75). `speaches`'s label claims `:8204`, which is breeze-tts's live port — a latent conflict if anyone starts it. **Uncommitted:** `graphify-out/GRAPH_REPORT.md` and `scripts/seat-inventory.py` were modified before this session began; untouched and deliberately not committed. ## Recent decisions +- `[2026-09-15]` ⚠⚠ **`--gpu-memory-utilization` DOES NOT PREDICT RESIDENT VRAM — measure it, never compute it.** Wrong in **both** directions on fv-ml1: `vllm-cyberprev` util 0.40 (expect ~39,155 MiB) holds **47,124** (+8 GB over); `vllm-gen-small` util 0.48 (expect ~46,986) holds **36,942** (−10 GB under). Planning a placement off the fractions would have been 8 GB wrong. Read `nvidia-smi --query-compute-apps`. Full per-seat residency table + the breeze shuffle arithmetic → `persistent-memory.d/2026-09-15-breeze-placement-sizing.md` + - `[2026-09-15]` **breeze-tts stays on irv-ml1; the TTS-stack move to fv-ml1 is PARKED (park id 75, `move-the-tts-stack-breeze-tts-bragi-tts-gateway`), triggered on evacuating embed/rerank/reward.** ⚠ Trigger as stated says "gpu0" but those three are on **GPU 1** (~0.16 util, ~15.7 GB; GPU 1 is the tight card at 0.975 / 4,336 MiB free) — confirm which he meant before executing. All three services move together because only `breeze-tts` is GPU-resident (~10.3 GiB, **growing**) while `bragi` and `tts-gateway` are CPU proxies, and co-location is what avoids a cross-site hop per TTS call. **breeze-tts sizing — original recommendation NOT to move it.** ~**10.3 GiB** measured under load at 53 min uptime, **up from 9.2 GiB** shortly after warm-up (it grows; n=2, plateau unmeasured) — so GPU 0's 11,982 MiB free is a **1.7 GB margin and shrinking**, on the live chat serving path. ⚠ Two measurement traps: it reports **nothing at idle on the wrong card** (`BREEZE_GPU_DEVICES=0` = the **3090**, not the A6000), and an early reading understates it. ⭐ The real objection is **topology**: `tts-gateway` is on irv-ml1 and reaches it same-box, so moving breeze alone adds a cross-site hop to every TTS call against a 478 ms first-sample budget. GPU 3 would fit it but spends the reserve. → `persistent-memory.d/2026-09-15-breeze-placement-sizing.md` - `[2026-09-15]` **Parakeet STT live on fv-ml1 GPU 0, behind LiteLLM `ext-stt` / `whisper-1`.** ⚠ **Placed on GPU 3 first, which was wrong — operator caught it.** A ~800 MiB seat should ride the card with the most uncommitted headroom (GPU 0, util 0.88, ~13 GB spare), not put the first fingerprint on the one pristine 96 GB card: vLLM sizes KV cache against TOTAL VRAM, so any tenant on an empty card eats a future full-size seat's profiling margin (flash-next needs 93 of 96 GiB). **GPU 3 is now a deliberate reserve at 2 MiB.** Retargeted the existing `stacks/parakeet/` (sherpa-onnx + our own FastAPI wrapper) from irv-ml1; v3 int8, 25 languages. ⚠ **ORT's CUDA EP compiles kernels lazily and the first decode on sm_120 took 45.7 s** — every later call ~0.5 s; a startup warmup in `app.py` now absorbs it, so the first real request is 0.65 s instead of a 45 s hang that no client would wait through. GPU use was **verified by a process on GPU 3 (922 MiB), not by the `provider=cuda` log line**, because ORT falls back to CPU silently and still returns correct text. Silence → `""` (null control), known sentence → near-exact (positive control). → `persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md`