vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(parakeet): move the seat from the empty GPU 3 to GPU 0

Browse Source 
Placed on GPU 3 first because it was the empty card. That was the wrong read:
the seat is ~800 MiB, under 1% of a 96 GB card, so the question was never "where
does it fit" but "whose headroom is cheapest to spend".

vLLM sizes its KV cache as a fraction of TOTAL VRAM, not free VRAM. A resident
tenant on an otherwise-clean card therefore does not cost its own megabytes — it
costs the profiling margin of whatever full-size seat lands there later, and
flash-next needs 93 GiB of 96. A 96 GB card at 2 MiB can still take that; the
same card at 922 MiB is one where the next big seat needs its utilization
hand-trimmed, which this repo's flash-next history shows is both thin and silent
when it goes wrong.

Committed utilization per card is the number that governs, not free bytes:

    GPU 0   0.40 + 0.48                       = 0.88    ~13 GB spare  <- moved here
    GPU 1   0.52+0.24+0.10+0.055+0.03+0.03    = 0.975   ~4.3 GB
    GPU 2   0.96                                        ~1.8 GB
    GPU 3   -                                           kept empty as reserve

GPU 3 is back to 2 MiB / 97,247 MiB free and is now documented as a deliberate
reserve rather than a spare.

Post-move n=5 on the same clip: 0.68 / 0.54 / 0.54 / 0.52 / 0.53 s, median 0.54 s
against 0.50 s on GPU 3. The spreads overlap at this sample size and no difference
is claimed; the GPU 3 figure was taken on an idle card and is now noted as a best
case, since the seat shares GPU 0 with the hot serving path. Silence control and
the gateway round-trip both re-verified after the move.

Also records both Parakeet endpoints (FV v3 on Blackwell, IRV v2 on a 3090) and
the four confounds that make them not an A/B pair, sent to tts-dev for the bench.
This commit is contained in:
vh
2026-09-15 01:50:25 -07:00
parent b9b14b5baf
commit caa04801f3
6 changed files with 117 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md
+58 -10
View File
@@ -6,19 +6,45 @@ Operator asked for an STT service on fv-ml1's utility GPU plus a LiteLLM alias.
`stacks/parakeet/` — Parakeet-TDT 0.6B **v3** int8 ONNX (25 European languages,
464 MiB) under sherpa-onnx, behind ~90 lines of FastAPI we own. Container
`parakeet`, port **8300**, **GPU 3** pinned by `device_ids`. Image
`parakeet`, port **8300**, **GPU 0** pinned by `device_ids`. Image
`local/parakeet:sherpa-onnx-v4` (5.09 GB).
Not greenfield: the stack already existed, targeting irv-ml1. Retargeted rather
than rewritten — the Ampere→Blackwell move was the only real question.
## Why GPU 3
## ⚠ Placement — got this wrong first, operator caught it
GPU 0 = 84/96 GB, GPU 1 = 92.9/96, GPU 2 = 95.5/96 (the vLLM seats). **GPU 3 was
at 2 MiB.** The dead on-host stub used `count: all`, which would have handed this
seat all four cards; replaced with an explicit `device_ids: ["3"]` per the fleet
convention. Inside the container the pinned card presents as `cuda:0`, which is
what ORT's CUDA EP takes by default.
Placed on the empty **GPU 3** initially, reading "the utility gpu" as "the spare
card". Operator's correction: *"1gb total vram pressure — and you didn't load it on
gpu 0?"* He is right, and the reason is sharper than "it fits anywhere".
**vLLM sizes its KV cache as a fraction of TOTAL VRAM, not free VRAM.** So a
resident tenant on an otherwise-clean card does not cost its own megabytes — it
costs a future full-size seat's profiling margin. `flash-next` needs **93 GiB of
96**. A 96 GB card at 2 MiB is a card that can still take that; the same card at
922 MiB is a card where the next big seat's `--gpu-memory-utilization` has to be
hand-trimmed, and the flash-next history in this repo shows exactly how thin and
how silent that failure gets.
The right question is not "where does 800 MiB fit" but "whose headroom is cheapest
to spend":
| GPU | committed util | spare |
|---|---|---|
| **0** | 0.40 + 0.48 = **0.88** | ~13 GB ← moved here |
| 1 | **0.975** (six small seats) | ~4.3 GB |
| 2 | **0.96** (flash-next) | ~1.8 GB |
| 3 | — | **kept empty as reserve** |
Moved the same night: one env var (`PARAKEET_GPU`) plus `compose up -d`. GPU 3 back
to 2 MiB / 97,247 MiB free. Post-move n=5 on the same clip: 0.68 / 0.54 / 0.54 /
0.52 / 0.53 s, median 0.54 s — **indistinguishable from the GPU 3 median of 0.50 s
at this sample size**; the spreads overlap and no difference is claimed.
The dead on-host stub used `count: all`, which would have handed this seat all four
cards; replaced with an explicit `device_ids` pin per the fleet convention. Inside
the container the pinned card presents as `cuda:0`, which is what ORT's CUDA EP
takes by default.
## ⚠ The finding worth keeping: a 45-second first decode
@@ -30,7 +56,9 @@ at session creation. On sm_120:
| first decode, cold container | **45.7 s** (n=1), reproduced at **45.1 s** on a second container |
| warm, 8.52 s clip | **0.50 s** median (n=5: 0.65 / 0.53 / 0.48 / 0.47 / 0.50) |
≈17× realtime warm, single-stream, one 8.52 s clip, int8, GPU 3 idle otherwise.
≈17x realtime warm, single-stream, one 8.52 s clip, int8. ⚠ Measured on GPU 3 while
it was idle; the seat now lives on GPU 0 beside the hot serving path, so treat that
number as a best case.
That is a smoke measurement with its harness stated, **not** a benchmark — no
concurrency sweep, no length sweep, one clip.
@@ -48,8 +76,8 @@ returns *correct text*, just slowly. Our own log line `loading OfflineRecognizer
The discriminator that actually settles it:
```
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv -i 3
-> 1588301, /opt/venv/bin/python3, 922 MiB
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv -i 0
-> 1594431, /opt/venv/bin/python3, 794 MiB (beside two VLLM::EngineCore entries)
```
Timing is **not** a sufficient check either — the int8 model is fast enough on a
@@ -84,3 +112,23 @@ that is where the `ext-tts` family lives, and it needs no gateway restart.
part of the 5-host normalisation).
- `servers/fv-ml1/README.md` is still broadly stale — it claims 2 GPUs and a
2026-07-22 stack list. Only the parakeet/GPU-3 rows were corrected.
## Two Parakeets, and the bench Vuong asked for
Both are live; endpoints sent to **tts-dev** 2026-09-15 for a head-to-head.
| | FV (new) | IRV (existing, up 2 months) |
|---|---|---|
| endpoint | `http://10.251.50.54:8300/v1/audio/transcriptions` | `http://100.64.0.6:8765/...` or `http://10.6.110.50:8765/...` |
| model | parakeet-tdt-0.6b-**v3** int8, 25 languages | parakeet-tdt-0.6b-**v2** int8, English only |
| GPU | RTX PRO 6000 Blackwell **sm_120**, GPU 0, shares with 2 vLLM seats | RTX 3090 **sm_86**, shares with 4 processes, 4.0 GB free |
| image | `local/parakeet:sherpa-onnx-v4` (has startup warmup) | `local/parakeet:sherpa-onnx-v2` (no warmup) |
⚠ **`10.100.79.3:8765` is DEAD** — the retired wg0 lifeline, still the href on IRV's
Homepage card. Same for `Speaches ASR` at `10.100.79.3:8204`.
⚠ **These were never an A/B pair — four things differ at once** (model version,
GPU architecture, card contention, image). A WER delta is a **v2-vs-v3** result, not
an FV-vs-IRV one. Offered tts-dev a v2 container on FV as a second compose project so
accuracy can be varied one factor at a time; not built unless they take it up.