memory: breeze-tts sizing and the fv-ml1 GPU 0 placement recommendation

The operator asked this mid-sweep and the answer never reached durable memory -
caught only because he asked again after the snapshot. Recommendation is not to
move it.

Re-measured rather than reciting the earlier figure, which was right when taken and
is now wrong: breeze holds 10,316 MiB after 53 minutes of uptime against 9,218 MiB
shortly after warm-up. The footprint grows with use, consistent with PyTorch's
caching allocator not returning memory - probably caching rather than a leak, but
resident either way and counting against any neighbour. Two points is a trend, not
a curve; whether it plateaus is unmeasured and stated as such.

That changes the placement answer. fv-ml1 GPU 0 has 11,982 MiB free, so the margin
is 1.7 GB and shrinking rather than the 2.8 GB the earlier number implied, on the
card carrying the live chat serving path.

The stronger objection is topology rather than VRAM: tts-gateway runs on irv-ml1
and reaches breeze on the same box, so moving breeze alone puts a cross-site hop on
every TTS call against a 478 ms to-first-sample budget. Moving it properly means
moving the gateway too. It is also not constrained where it sits - the 3090 still
has 10 GB free.

Also records the trap that nearly produced a wrong number: breeze reports nothing
at idle when queried on the wrong GPU, because BREEZE_GPU_DEVICES=0 is the 3090
rather than the A6000. An idle query of the A6000 shows it absent entirely.

persistent-memory.d/2026-09-15-breeze-placement-sizing.md

@@ -0,0 +1,53 @@
+# breeze-tts sizing and the fv-ml1 GPU 0 placement question (2026-09-15)
+
+Operator asked how big breeze-tts is and whether fv-ml1 GPU 0 has room for it.
+**Recommendation: do not move it.** Numbers and reasoning below.
+
+## ⚠ Measure it UNDER LOAD and measure it LATE — it does both badly at idle and early
+
+| when | breeze VRAM |
+|---|---|
+| shortly after warm-up | **9,218 MiB** |
+| after 53 min uptime | **10,316 MiB** |
+| at idle on the wrong card | **invisible** — shows nothing |
+
+Two traps stacked:
+
+1. ⚠ **It reports nothing when you query the wrong GPU.** `BREEZE_GPU_DEVICES=0` and on
+   irv-ml1 that is the **RTX 3090**, not the A6000. An idle query of GPU 1 shows breeze
+   absent entirely — I read that as "released its memory" before checking the device map.
+2. ⚠ **The footprint GROWS with use** — +1.1 GiB over 53 minutes. Consistent with
+   PyTorch's caching allocator not returning memory (already hypothesised in this repo
+   for this host), so probably caching rather than a leak — but it is resident either way
+   and it counts against any neighbour. **n=2 points is a trend, not a characterised
+   curve**; whether it plateaus is unmeasured.
+
+Other facts: image `local/breeze-tts:v3` is **26.5 GB** on disk; weights bind-mounted from
+`/home/lkraven/breeze/weights`; container pid 2048660, CUDA pid 2048873.
+
+## Why NOT to move it to fv-ml1 GPU 0
+
+**1. The margin is thinner than it first looked, and trending the wrong way.**
+
+    fv-ml1 GPU 0 free : 11,982 MiB   (two vLLM seats at util 0.88, + parakeet ~800 MiB)
+    breeze at 9.2 GiB -> 2.8 GB margin   <- the number I first quoted
+    breeze at 10.3 GiB -> 1.7 GB margin  <- the number after an hour, still climbing
+
+GPU 0 is the **live chat serving path**. Putting a growing tenant in shrinking headroom
+there is the wrong direction.
+
+**2. ⭐ The real objection is topology, not VRAM.** `tts-gateway` runs on **irv-ml1** and
+reaches breeze **on the same box**. Moving breeze alone to Fountain Valley puts an
+Irvine↔FV hop on **every TTS call**, against a budget tts-dev measured at **478 ms to
+first sample**. Moving it properly means moving the gateway too — a project, not a
+placement tweak.
+
+**3. It is not constrained where it is.** The 3090 still has **10,099 MiB free**.
+
+## If consolidation onto FV is the goal anyway
+
+**GPU 3** fits it comfortably (97,247 MiB free) and has none of the margin problem — but
+it **spends the reserve**, and the reserve exists because vLLM sizes KV against TOTAL
+VRAM and `flash-next` needs 93 of 96 GiB. Breeze at ~10 GiB leaves ~87 GiB, which no
+longer takes a full-card seat. That is the trade; it is the operator's call, and the
+gateway should move with it either way.

persistent-memory.md

@@ -133,6 +133,8 @@ _As of 2026-09-15 ~09:30 PT._
 
 ## Recent decisions
 
+- `[2026-09-15]` **breeze-tts sizing / fv-ml1 GPU 0 placement — RECOMMEND NOT MOVING IT.** ~**10.3 GiB** measured under load at 53 min uptime, **up from 9.2 GiB** shortly after warm-up (it grows; n=2, plateau unmeasured) — so GPU 0's 11,982 MiB free is a **1.7 GB margin and shrinking**, on the live chat serving path. ⚠ Two measurement traps: it reports **nothing at idle on the wrong card** (`BREEZE_GPU_DEVICES=0` = the **3090**, not the A6000), and an early reading understates it. ⭐ The real objection is **topology**: `tts-gateway` is on irv-ml1 and reaches it same-box, so moving breeze alone adds a cross-site hop to every TTS call against a 478 ms first-sample budget. GPU 3 would fit it but spends the reserve. → `persistent-memory.d/2026-09-15-breeze-placement-sizing.md`
+
 - `[2026-09-15]` **Parakeet STT live on fv-ml1 GPU 0, behind LiteLLM `ext-stt` / `whisper-1`.** ⚠ **Placed on GPU 3 first, which was wrong — operator caught it.** A ~800 MiB seat should ride the card with the most uncommitted headroom (GPU 0, util 0.88, ~13 GB spare), not put the first fingerprint on the one pristine 96 GB card: vLLM sizes KV cache against TOTAL VRAM, so any tenant on an empty card eats a future full-size seat's profiling margin (flash-next needs 93 of 96 GiB). **GPU 3 is now a deliberate reserve at 2 MiB.** Retargeted the existing `stacks/parakeet/` (sherpa-onnx + our own FastAPI wrapper) from irv-ml1; v3 int8, 25 languages. ⚠ **ORT's CUDA EP compiles kernels lazily and the first decode on sm_120 took 45.7 s** — every later call ~0.5 s; a startup warmup in `app.py` now absorbs it, so the first real request is 0.65 s instead of a 45 s hang that no client would wait through. GPU use was **verified by a process on GPU 3 (922 MiB), not by the `provider=cuda` log line**, because ORT falls back to CPU silently and still returns correct text. Silence → `""` (null control), known sentence → near-exact (positive control). → `persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md`
 
 - `[2026-09-15]` ⭐⭐⭐ **THE FLEET'S CHARACTERISTIC FAILURE, named: a confident answer from a broken instrument.** Nine instances in one night, every one of which PASSED A CHECK — `provider=cuda` while ORT ran on CPU; `node --check` green on a file whose SERVED script was dead; `secret get` returning `""` with exit 0; `find()` turning a failed listing into an authoritative "not found"; a 401 rendering as "0 toolsets"; `compat` ✓ on a typo'd path; `doctor` exit 0 on ERROR; `ss | grep python` missing a listener named `hermes`; SIGTERM freeing a port 35 s before the process died. ⚠ **The tell: whenever "broken" and "legitimately empty/absent/off" produce the same output.** Remedies: measure the output not the input, positive AND true-negative controls, refuse to emit the ambiguous value, and never declare victory on a plausible fix. → `persistent-memory.d/2026-09-15-silent-wrong-answer-pattern.md`