From 7a33bd9f09cf6621ab2cbfa268bb2fe74abe3680 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 15 Sep 2026 09:31:12 -0700 Subject: [PATCH] memory: breeze stays put; TTS-stack move to fv-ml1 parked at id 75 Operator ruling: leave breeze-tts on irv-ml1 and park moving it, bragi and tts-gateway to fv-ml1 until the embedder, reranker and reward seats are evacuated. Parked as move-the-tts-stack-breeze-tts-bragi-tts-gateway (id 75) with the trigger, the footprints and the migration gotchas, so it resurfaces with everything needed rather than as a bare line. Two things worth having recorded against the trigger. All three services move as a set because only breeze is GPU-resident at ~10.3 GiB and growing, while bragi and tts-gateway are CPU-only proxies - co-location with the gateway is the entire reason not to move breeze alone, since that is what puts a cross-site hop on every TTS call. And the trigger as stated names gpu0, but vllm-embed, vllm-rerank-a3 and vllm-reward are all pinned to GPU 1. GPU 1 is the constrained card at 0.975 committed with 4,336 MiB free, while GPU 0 has 11,982 MiB free and carries the live chat path, so evacuating those three relieves GPU 1 rather than GPU 0. Recorded as a confirm-before-executing rather than silently corrected, since it changes where the TTS stack would land. Also notes that bragi and tts-gateway reach each other by name only through extra_hosts pins, because containers on irv-ml1 cannot resolve nh3.internal - those pins travel with them and need re-pointing at the new host. --- .../2026-09-15-breeze-placement-sizing.md | 29 +++++++++++++++++++ persistent-memory.md | 2 +- 2 files changed, 30 insertions(+), 1 deletion(-) diff --git a/persistent-memory.d/2026-09-15-breeze-placement-sizing.md b/persistent-memory.d/2026-09-15-breeze-placement-sizing.md index c6e7959..8609d1f 100644 --- a/persistent-memory.d/2026-09-15-breeze-placement-sizing.md +++ b/persistent-memory.d/2026-09-15-breeze-placement-sizing.md @@ -44,6 +44,35 @@ placement tweak. **3. It is not constrained where it is.** The 3090 still has **10,099 MiB free**. +## ✅ RESOLVED — operator 2026-09-15: leave it, park the move + +> "leave it where it is, park moving breeze, bragi, and the tts-gateway to fv1 when we +> move the embedder, reranker, and reward models off gpu0" + +Parked at **`move-the-tts-stack-breeze-tts-bragi-tts-gateway` (park id 75)**. + +**All three move together, and only one is GPU-resident:** + +| service | footprint | +|---|---| +| `breeze-tts` | **~10.3 GiB VRAM**, growing | +| `bragi` | **CPU only** — proxy, polls tts-gateway `/health` | +| `tts-gateway` | **CPU only** — the `ext-tts` routing gateway | + +⭐ Co-location is the whole reason to move them as a set: `tts-gateway` reaches breeze +**same-box** today, and moving breeze alone is what puts the cross-site hop on every call. + +⚠ **The trigger as stated has the card wrong, and it is worth confirming before +executing.** The operator said "off gpu0", but `vllm-embed`, `vllm-rerank-a3` and +`vllm-reward` are all pinned to **GPU 1** (util 0.03 + 0.03 + 0.10 ≈ 0.16, ~15.7 GB). +GPU 1 is the constrained card at **0.975 committed / 4,336 MiB free**; GPU 0 has +11,982 MiB free and carries the live chat path. So evacuating those three relieves +**GPU 1**, not GPU 0 — which changes where the TTS stack would land. + +⚠ **Migration gotcha:** containers on irv-ml1 cannot resolve `*.nh3.internal`; `bragi` +and `tts-gateway` reach each other by name only through `extra_hosts` pins. Those pins +travel with them and need re-pointing at the new host. + ## If consolidation onto FV is the goal anyway **GPU 3** fits it comfortably (97,247 MiB free) and has none of the margin problem — but diff --git a/persistent-memory.md b/persistent-memory.md index ce7d717..f98c601 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -133,7 +133,7 @@ _As of 2026-09-15 ~09:30 PT._ ## Recent decisions -- `[2026-09-15]` **breeze-tts sizing / fv-ml1 GPU 0 placement — RECOMMEND NOT MOVING IT.** ~**10.3 GiB** measured under load at 53 min uptime, **up from 9.2 GiB** shortly after warm-up (it grows; n=2, plateau unmeasured) — so GPU 0's 11,982 MiB free is a **1.7 GB margin and shrinking**, on the live chat serving path. ⚠ Two measurement traps: it reports **nothing at idle on the wrong card** (`BREEZE_GPU_DEVICES=0` = the **3090**, not the A6000), and an early reading understates it. ⭐ The real objection is **topology**: `tts-gateway` is on irv-ml1 and reaches it same-box, so moving breeze alone adds a cross-site hop to every TTS call against a 478 ms first-sample budget. GPU 3 would fit it but spends the reserve. → `persistent-memory.d/2026-09-15-breeze-placement-sizing.md` +- `[2026-09-15]` **breeze-tts stays on irv-ml1; the TTS-stack move to fv-ml1 is PARKED (park id 75, `move-the-tts-stack-breeze-tts-bragi-tts-gateway`), triggered on evacuating embed/rerank/reward.** ⚠ Trigger as stated says "gpu0" but those three are on **GPU 1** (~0.16 util, ~15.7 GB; GPU 1 is the tight card at 0.975 / 4,336 MiB free) — confirm which he meant before executing. All three services move together because only `breeze-tts` is GPU-resident (~10.3 GiB, **growing**) while `bragi` and `tts-gateway` are CPU proxies, and co-location is what avoids a cross-site hop per TTS call. **breeze-tts sizing — original recommendation NOT to move it.** ~**10.3 GiB** measured under load at 53 min uptime, **up from 9.2 GiB** shortly after warm-up (it grows; n=2, plateau unmeasured) — so GPU 0's 11,982 MiB free is a **1.7 GB margin and shrinking**, on the live chat serving path. ⚠ Two measurement traps: it reports **nothing at idle on the wrong card** (`BREEZE_GPU_DEVICES=0` = the **3090**, not the A6000), and an early reading understates it. ⭐ The real objection is **topology**: `tts-gateway` is on irv-ml1 and reaches it same-box, so moving breeze alone adds a cross-site hop to every TTS call against a 478 ms first-sample budget. GPU 3 would fit it but spends the reserve. → `persistent-memory.d/2026-09-15-breeze-placement-sizing.md` - `[2026-09-15]` **Parakeet STT live on fv-ml1 GPU 0, behind LiteLLM `ext-stt` / `whisper-1`.** ⚠ **Placed on GPU 3 first, which was wrong — operator caught it.** A ~800 MiB seat should ride the card with the most uncommitted headroom (GPU 0, util 0.88, ~13 GB spare), not put the first fingerprint on the one pristine 96 GB card: vLLM sizes KV cache against TOTAL VRAM, so any tenant on an empty card eats a future full-size seat's profiling margin (flash-next needs 93 of 96 GiB). **GPU 3 is now a deliberate reserve at 2 MiB.** Retargeted the existing `stacks/parakeet/` (sherpa-onnx + our own FastAPI wrapper) from irv-ml1; v3 int8, 25 languages. ⚠ **ORT's CUDA EP compiles kernels lazily and the first decode on sm_120 took 45.7 s** — every later call ~0.5 s; a startup warmup in `app.py` now absorbs it, so the first real request is 0.65 s instead of a 45 s hang that no client would wait through. GPU use was **verified by a process on GPU 3 (922 MiB), not by the `provider=cuda` log line**, because ORT falls back to CPU silently and still returns correct text. Silence → `""` (null control), known sentence → near-exact (positive control). → `persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md`