fix(parakeet): move the seat from the empty GPU 3 to GPU 0

Placed on GPU 3 first because it was the empty card. That was the wrong read: the seat is ~800 MiB, under 1% of a 96 GB card, so the question was never "where does it fit" but "whose headroom is cheapest to spend". vLLM sizes its KV cache as a fraction of TOTAL VRAM, not free VRAM. A resident tenant on an otherwise-clean card therefore does not cost its own megabytes — it costs the profiling margin of whatever full-size seat lands there later, and flash-next needs 93 GiB of 96. A 96 GB card at 2 MiB can still take that; the same card at 922 MiB is one where the next big seat needs its utilization hand-trimmed, which this repo's flash-next history shows is both thin and silent when it goes wrong. Committed utilization per card is the number that governs, not free bytes: GPU 0 0.40 + 0.48 = 0.88 ~13 GB spare <- moved here GPU 1 0.52+0.24+0.10+0.055+0.03+0.03 = 0.975 ~4.3 GB GPU 2 0.96 ~1.8 GB GPU 3 - kept empty as reserve GPU 3 is back to 2 MiB / 97,247 MiB free and is now documented as a deliberate reserve rather than a spare. Post-move n=5 on the same clip: 0.68 / 0.54 / 0.54 / 0.52 / 0.53 s, median 0.54 s against 0.50 s on GPU 3. The spreads overlap at this sample size and no difference is claimed; the GPU 3 figure was taken on an idle card and is now noted as a best case, since the seat shares GPU 0 with the hot serving path. Silence control and the gateway round-trip both re-verified after the move. Also records both Parakeet endpoints (FV v3 on Blackwell, IRV v2 on a 3090) and the four confounds that make them not an A/B pair, sent to tts-dev for the bench.
2026-09-15 01:50:25 -07:00
parent b9b14b5baf
commit caa04801f3
6 changed files with 117 additions and 42 deletions
@@ -6,24 +6,35 @@ wrapper over [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx)

 **Server:** fv-ml1 (Fountain Valley, `10.251.50.54`) — moved from irv-ml1 2026-09-15
 **Port:** 8300 (container 8000)
-**GPU:** **3**, pinned explicitly via `device_ids` — the utility card
+**GPU:** **0**, pinned explicitly via `device_ids` — shares the card with two vLLM seats
 **Image:** `local/parakeet:sherpa-onnx-v4` — built from `Dockerfile` +
 `app.py` + `entrypoint.sh` in this directory; **we own all the code**
 **Model:** `parakeet-tdt-0.6b-v3` int8, 25 European languages (~464 MiB)

-## Why GPU 3
+## Why GPU 0 and not the empty card

-fv-ml1 has four RTX PRO 6000 Blackwell Max-Q (96 GB each). Three carry the
-vLLM serving seats and run 85–98 % full; GPU 3 is the utility card and was
-empty (2 MiB) at placement time. A 0.6 B int8 ASR model is a rounding error
-next to those seats, but it still has to go somewhere that is not fighting
-them for VRAM.
+fv-ml1 has four RTX PRO 6000 Blackwell Max-Q (96 GB each). This seat holds
+**~800 MiB — under 1 %** of one card, so the question is not "where does it
+fit" but "whose headroom can it spend most cheaply".

-⚠ The pin is `deploy.resources.reservations.devices[].device_ids: ["3"]`,
-the fleet convention — **not** `count: all`, which is what the dead on-host
-stub used and which would have handed this seat all four cards. Inside the
-container the pinned card presents as `cuda:0`, which is what sherpa-onnx's
-CUDA execution provider takes by default.
+| GPU | committed `--gpu-memory-utilization` | spare |
+|---|---|---|
+| **0** | 0.40 + 0.48 = **0.88** | ~13 GB ← here |
+| 1 | 0.52+0.24+0.10+0.055+0.03+0.03 = **0.975** | ~4.3 GB |
+| 2 | **0.96** | ~1.8 GB |
+| 3 | — | *kept empty* |
+
+It was first placed on the empty GPU 3, which was wrong: **vLLM sizes its KV
+cache as a fraction of TOTAL VRAM, not free VRAM**, so any resident tenant on an
+otherwise-clean card eats directly into the profiling margin of whatever big seat
+lands there later — and `flash-next` needs 93 GiB of 96. A 96 GB card at 2 MiB is
+worth far more than 800 MiB of it. Moved to GPU 0 the same night.
+
+⚠ The pin is `deploy.resources.reservations.devices[].device_ids`, the fleet
+convention — **not** `count: all`, which is what the dead on-host stub used and
+which would have handed this seat all four cards. Inside the container the pinned
+card presents as `cuda:0`, which is what sherpa-onnx's CUDA execution provider
+takes by default.

 ## Why not the FastAPI community wrappers

@@ -70,13 +81,14 @@ The honest check is to watch the card while a transcription runs:
 ```bash
 # on fv-ml1 — terminal 1
 watch -n0.2 'nvidia-smi --query-compute-apps=pid,process_name,used_memory \
-    --format=csv -i 3'
+    --format=csv -i 0'

 # terminal 2 — send real audio, not silence
 curl -s -F file=@sample.wav http://127.0.0.1:8300/v1/audio/transcriptions
 ```

-A process must appear **on GPU 3** for the duration. If GPU 3 stays empty, the
+A process must appear **on GPU 0** for the duration — ~800 MiB alongside the two
+much larger `VLLM::EngineCore` entries. If it never appears, the
 CUDA EP did not initialise and you are on CPU regardless of what `.env` says.
 Confirm with the container's own startup log, which names the providers ORT
 actually registered: