diff --git a/persistent-memory.d/2026-09-15-breeze-placement-sizing.md b/persistent-memory.d/2026-09-15-breeze-placement-sizing.md new file mode 100644 index 0000000..c6e7959 --- /dev/null +++ b/persistent-memory.d/2026-09-15-breeze-placement-sizing.md @@ -0,0 +1,53 @@ +# breeze-tts sizing and the fv-ml1 GPU 0 placement question (2026-09-15) + +Operator asked how big breeze-tts is and whether fv-ml1 GPU 0 has room for it. +**Recommendation: do not move it.** Numbers and reasoning below. + +## ⚠ Measure it UNDER LOAD and measure it LATE — it does both badly at idle and early + +| when | breeze VRAM | +|---|---| +| shortly after warm-up | **9,218 MiB** | +| after 53 min uptime | **10,316 MiB** | +| at idle on the wrong card | **invisible** — shows nothing | + +Two traps stacked: + +1. ⚠ **It reports nothing when you query the wrong GPU.** `BREEZE_GPU_DEVICES=0` and on + irv-ml1 that is the **RTX 3090**, not the A6000. An idle query of GPU 1 shows breeze + absent entirely — I read that as "released its memory" before checking the device map. +2. ⚠ **The footprint GROWS with use** — +1.1 GiB over 53 minutes. Consistent with + PyTorch's caching allocator not returning memory (already hypothesised in this repo + for this host), so probably caching rather than a leak — but it is resident either way + and it counts against any neighbour. **n=2 points is a trend, not a characterised + curve**; whether it plateaus is unmeasured. + +Other facts: image `local/breeze-tts:v3` is **26.5 GB** on disk; weights bind-mounted from +`/home/lkraven/breeze/weights`; container pid 2048660, CUDA pid 2048873. + +## Why NOT to move it to fv-ml1 GPU 0 + +**1. The margin is thinner than it first looked, and trending the wrong way.** + + fv-ml1 GPU 0 free : 11,982 MiB (two vLLM seats at util 0.88, + parakeet ~800 MiB) + breeze at 9.2 GiB -> 2.8 GB margin <- the number I first quoted + breeze at 10.3 GiB -> 1.7 GB margin <- the number after an hour, still climbing + +GPU 0 is the **live chat serving path**. Putting a growing tenant in shrinking headroom +there is the wrong direction. + +**2. ⭐ The real objection is topology, not VRAM.** `tts-gateway` runs on **irv-ml1** and +reaches breeze **on the same box**. Moving breeze alone to Fountain Valley puts an +Irvine↔FV hop on **every TTS call**, against a budget tts-dev measured at **478 ms to +first sample**. Moving it properly means moving the gateway too — a project, not a +placement tweak. + +**3. It is not constrained where it is.** The 3090 still has **10,099 MiB free**. + +## If consolidation onto FV is the goal anyway + +**GPU 3** fits it comfortably (97,247 MiB free) and has none of the margin problem — but +it **spends the reserve**, and the reserve exists because vLLM sizes KV against TOTAL +VRAM and `flash-next` needs 93 of 96 GiB. Breeze at ~10 GiB leaves ~87 GiB, which no +longer takes a full-card seat. That is the trade; it is the operator's call, and the +gateway should move with it either way. diff --git a/persistent-memory.md b/persistent-memory.md index d4c2e51..ce7d717 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -133,6 +133,8 @@ _As of 2026-09-15 ~09:30 PT._ ## Recent decisions +- `[2026-09-15]` **breeze-tts sizing / fv-ml1 GPU 0 placement — RECOMMEND NOT MOVING IT.** ~**10.3 GiB** measured under load at 53 min uptime, **up from 9.2 GiB** shortly after warm-up (it grows; n=2, plateau unmeasured) — so GPU 0's 11,982 MiB free is a **1.7 GB margin and shrinking**, on the live chat serving path. ⚠ Two measurement traps: it reports **nothing at idle on the wrong card** (`BREEZE_GPU_DEVICES=0` = the **3090**, not the A6000), and an early reading understates it. ⭐ The real objection is **topology**: `tts-gateway` is on irv-ml1 and reaches it same-box, so moving breeze alone adds a cross-site hop to every TTS call against a 478 ms first-sample budget. GPU 3 would fit it but spends the reserve. → `persistent-memory.d/2026-09-15-breeze-placement-sizing.md` + - `[2026-09-15]` **Parakeet STT live on fv-ml1 GPU 0, behind LiteLLM `ext-stt` / `whisper-1`.** ⚠ **Placed on GPU 3 first, which was wrong — operator caught it.** A ~800 MiB seat should ride the card with the most uncommitted headroom (GPU 0, util 0.88, ~13 GB spare), not put the first fingerprint on the one pristine 96 GB card: vLLM sizes KV cache against TOTAL VRAM, so any tenant on an empty card eats a future full-size seat's profiling margin (flash-next needs 93 of 96 GiB). **GPU 3 is now a deliberate reserve at 2 MiB.** Retargeted the existing `stacks/parakeet/` (sherpa-onnx + our own FastAPI wrapper) from irv-ml1; v3 int8, 25 languages. ⚠ **ORT's CUDA EP compiles kernels lazily and the first decode on sm_120 took 45.7 s** — every later call ~0.5 s; a startup warmup in `app.py` now absorbs it, so the first real request is 0.65 s instead of a 45 s hang that no client would wait through. GPU use was **verified by a process on GPU 3 (922 MiB), not by the `provider=cuda` log line**, because ORT falls back to CPU silently and still returns correct text. Silence → `""` (null control), known sentence → near-exact (positive control). → `persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md` - `[2026-09-15]` ⭐⭐⭐ **THE FLEET'S CHARACTERISTIC FAILURE, named: a confident answer from a broken instrument.** Nine instances in one night, every one of which PASSED A CHECK — `provider=cuda` while ORT ran on CPU; `node --check` green on a file whose SERVED script was dead; `secret get` returning `""` with exit 0; `find()` turning a failed listing into an authoritative "not found"; a 401 rendering as "0 toolsets"; `compat` ✓ on a typo'd path; `doctor` exit 0 on ERROR; `ss | grep python` missing a listener named `hermes`; SIGTERM freeing a port 35 s before the process died. ⚠ **The tell: whenever "broken" and "legitimately empty/absent/off" produce the same output.** Remedies: measure the output not the input, positive AND true-negative controls, refuse to emit the ambiguous value, and never declare victory on a plausible fix. → `persistent-memory.d/2026-09-15-silent-wrong-answer-pattern.md`