memory: breeze-tts sizing and the fv-ml1 GPU 0 placement recommendation

The operator asked this mid-sweep and the answer never reached durable memory - caught only because he asked again after the snapshot. Recommendation is not to move it. Re-measured rather than reciting the earlier figure, which was right when taken and is now wrong: breeze holds 10,316 MiB after 53 minutes of uptime against 9,218 MiB shortly after warm-up. The footprint grows with use, consistent with PyTorch's caching allocator not returning memory - probably caching rather than a leak, but resident either way and counting against any neighbour. Two points is a trend, not a curve; whether it plateaus is unmeasured and stated as such. That changes the placement answer. fv-ml1 GPU 0 has 11,982 MiB free, so the margin is 1.7 GB and shrinking rather than the 2.8 GB the earlier number implied, on the card carrying the live chat serving path. The stronger objection is topology rather than VRAM: tts-gateway runs on irv-ml1 and reaches breeze on the same box, so moving breeze alone puts a cross-site hop on every TTS call against a 478 ms to-first-sample budget. Moving it properly means moving the gateway too. It is also not constrained where it sits - the 3090 still has 10 GB free. Also records the trap that nearly produced a wrong number: breeze reports nothing at idle when queried on the wrong GPU, because BREEZE_GPU_DEVICES=0 is the 3090 rather than the A6000. An idle query of the A6000 shows it absent entirely.
2026-09-15 09:27:59 -07:00
parent 653f7fb939
commit b0e7b408d9
2 changed files with 55 additions and 0 deletions
@@ -0,0 +1,53 @@
+# breeze-tts sizing and the fv-ml1 GPU 0 placement question (2026-09-15)
+
+Operator asked how big breeze-tts is and whether fv-ml1 GPU 0 has room for it.
+**Recommendation: do not move it.** Numbers and reasoning below.
+
+## ⚠ Measure it UNDER LOAD and measure it LATE — it does both badly at idle and early
+
+| when | breeze VRAM |
+|---|---|
+| shortly after warm-up | **9,218 MiB** |
+| after 53 min uptime | **10,316 MiB** |
+| at idle on the wrong card | **invisible** — shows nothing |
+
+Two traps stacked:
+
+1. ⚠ **It reports nothing when you query the wrong GPU.** `BREEZE_GPU_DEVICES=0` and on
+   irv-ml1 that is the **RTX 3090**, not the A6000. An idle query of GPU 1 shows breeze
+   absent entirely — I read that as "released its memory" before checking the device map.
+2. ⚠ **The footprint GROWS with use** — +1.1 GiB over 53 minutes. Consistent with
+   PyTorch's caching allocator not returning memory (already hypothesised in this repo
+   for this host), so probably caching rather than a leak — but it is resident either way
+   and it counts against any neighbour. **n=2 points is a trend, not a characterised
+   curve**; whether it plateaus is unmeasured.
+
+Other facts: image `local/breeze-tts:v3` is **26.5 GB** on disk; weights bind-mounted from
+`/home/lkraven/breeze/weights`; container pid 2048660, CUDA pid 2048873.
+
+## Why NOT to move it to fv-ml1 GPU 0
+
+**1. The margin is thinner than it first looked, and trending the wrong way.**
+
+    fv-ml1 GPU 0 free : 11,982 MiB   (two vLLM seats at util 0.88, + parakeet ~800 MiB)
+    breeze at 9.2 GiB -> 2.8 GB margin   <- the number I first quoted
+    breeze at 10.3 GiB -> 1.7 GB margin  <- the number after an hour, still climbing
+
+GPU 0 is the **live chat serving path**. Putting a growing tenant in shrinking headroom
+there is the wrong direction.
+
+**2. ⭐ The real objection is topology, not VRAM.** `tts-gateway` runs on **irv-ml1** and
+reaches breeze **on the same box**. Moving breeze alone to Fountain Valley puts an
+Irvine↔FV hop on **every TTS call**, against a budget tts-dev measured at **478 ms to
+first sample**. Moving it properly means moving the gateway too — a project, not a
+placement tweak.
+
+**3. It is not constrained where it is.** The 3090 still has **10,099 MiB free**.
+
+## If consolidation onto FV is the goal anyway
+
+**GPU 3** fits it comfortably (97,247 MiB free) and has none of the margin problem — but
+it **spends the reserve**, and the reserve exists because vLLM sizes KV against TOTAL
+VRAM and `flash-next` needs 93 of 96 GiB. Breeze at ~10 GiB leaves ~87 GiB, which no
+longer takes a full-card seat. That is the trade; it is the operator's call, and the
+gateway should move with it either way.