diff --git a/persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md b/persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md index 0eefdd7..076fc1b 100644 --- a/persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md +++ b/persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md @@ -6,19 +6,45 @@ Operator asked for an STT service on fv-ml1's utility GPU plus a LiteLLM alias. `stacks/parakeet/` — Parakeet-TDT 0.6B **v3** int8 ONNX (25 European languages, 464 MiB) under sherpa-onnx, behind ~90 lines of FastAPI we own. Container -`parakeet`, port **8300**, **GPU 3** pinned by `device_ids`. Image +`parakeet`, port **8300**, **GPU 0** pinned by `device_ids`. Image `local/parakeet:sherpa-onnx-v4` (5.09 GB). Not greenfield: the stack already existed, targeting irv-ml1. Retargeted rather than rewritten — the Ampere→Blackwell move was the only real question. -## Why GPU 3 +## ⚠ Placement — got this wrong first, operator caught it -GPU 0 = 84/96 GB, GPU 1 = 92.9/96, GPU 2 = 95.5/96 (the vLLM seats). **GPU 3 was -at 2 MiB.** The dead on-host stub used `count: all`, which would have handed this -seat all four cards; replaced with an explicit `device_ids: ["3"]` per the fleet -convention. Inside the container the pinned card presents as `cuda:0`, which is -what ORT's CUDA EP takes by default. +Placed on the empty **GPU 3** initially, reading "the utility gpu" as "the spare +card". Operator's correction: *"1gb total vram pressure — and you didn't load it on +gpu 0?"* He is right, and the reason is sharper than "it fits anywhere". + +**vLLM sizes its KV cache as a fraction of TOTAL VRAM, not free VRAM.** So a +resident tenant on an otherwise-clean card does not cost its own megabytes — it +costs a future full-size seat's profiling margin. `flash-next` needs **93 GiB of +96**. A 96 GB card at 2 MiB is a card that can still take that; the same card at +922 MiB is a card where the next big seat's `--gpu-memory-utilization` has to be +hand-trimmed, and the flash-next history in this repo shows exactly how thin and +how silent that failure gets. + +The right question is not "where does 800 MiB fit" but "whose headroom is cheapest +to spend": + +| GPU | committed util | spare | +|---|---|---| +| **0** | 0.40 + 0.48 = **0.88** | ~13 GB ← moved here | +| 1 | **0.975** (six small seats) | ~4.3 GB | +| 2 | **0.96** (flash-next) | ~1.8 GB | +| 3 | — | **kept empty as reserve** | + +Moved the same night: one env var (`PARAKEET_GPU`) plus `compose up -d`. GPU 3 back +to 2 MiB / 97,247 MiB free. Post-move n=5 on the same clip: 0.68 / 0.54 / 0.54 / +0.52 / 0.53 s, median 0.54 s — **indistinguishable from the GPU 3 median of 0.50 s +at this sample size**; the spreads overlap and no difference is claimed. + +The dead on-host stub used `count: all`, which would have handed this seat all four +cards; replaced with an explicit `device_ids` pin per the fleet convention. Inside +the container the pinned card presents as `cuda:0`, which is what ORT's CUDA EP +takes by default. ## ⚠ The finding worth keeping: a 45-second first decode @@ -30,7 +56,9 @@ at session creation. On sm_120: | first decode, cold container | **45.7 s** (n=1), reproduced at **45.1 s** on a second container | | warm, 8.52 s clip | **0.50 s** median (n=5: 0.65 / 0.53 / 0.48 / 0.47 / 0.50) | -≈17× realtime warm, single-stream, one 8.52 s clip, int8, GPU 3 idle otherwise. +≈17x realtime warm, single-stream, one 8.52 s clip, int8. ⚠ Measured on GPU 3 while +it was idle; the seat now lives on GPU 0 beside the hot serving path, so treat that +number as a best case. That is a smoke measurement with its harness stated, **not** a benchmark — no concurrency sweep, no length sweep, one clip. @@ -48,8 +76,8 @@ returns *correct text*, just slowly. Our own log line `loading OfflineRecognizer The discriminator that actually settles it: ``` -nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv -i 3 --> 1588301, /opt/venv/bin/python3, 922 MiB +nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv -i 0 +-> 1594431, /opt/venv/bin/python3, 794 MiB (beside two VLLM::EngineCore entries) ``` Timing is **not** a sufficient check either — the int8 model is fast enough on a @@ -84,3 +112,23 @@ that is where the `ext-tts` family lives, and it needs no gateway restart. part of the 5-host normalisation). - `servers/fv-ml1/README.md` is still broadly stale — it claims 2 GPUs and a 2026-07-22 stack list. Only the parakeet/GPU-3 rows were corrected. + + +## Two Parakeets, and the bench Vuong asked for + +Both are live; endpoints sent to **tts-dev** 2026-09-15 for a head-to-head. + +| | FV (new) | IRV (existing, up 2 months) | +|---|---|---| +| endpoint | `http://10.251.50.54:8300/v1/audio/transcriptions` | `http://100.64.0.6:8765/...` or `http://10.6.110.50:8765/...` | +| model | parakeet-tdt-0.6b-**v3** int8, 25 languages | parakeet-tdt-0.6b-**v2** int8, English only | +| GPU | RTX PRO 6000 Blackwell **sm_120**, GPU 0, shares with 2 vLLM seats | RTX 3090 **sm_86**, shares with 4 processes, 4.0 GB free | +| image | `local/parakeet:sherpa-onnx-v4` (has startup warmup) | `local/parakeet:sherpa-onnx-v2` (no warmup) | + +⚠ **`10.100.79.3:8765` is DEAD** — the retired wg0 lifeline, still the href on IRV's +Homepage card. Same for `Speaches ASR` at `10.100.79.3:8204`. + +⚠ **These were never an A/B pair — four things differ at once** (model version, +GPU architecture, card contention, image). A WER delta is a **v2-vs-v3** result, not +an FV-vs-IRV one. Offered tts-dev a v2 container on FV as a second compose project so +accuracy can be varied one factor at a time; not built unless they take it up. diff --git a/persistent-memory.md b/persistent-memory.md index 862b46f..277b328 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -175,7 +175,7 @@ hardened for ha-dev (`d1769ed` ff); `kb` KB-search tool (`68fa80f`). ## Recent decisions -- `[2026-09-15]` **Parakeet STT live on fv-ml1 GPU 3, behind LiteLLM `ext-stt` / `whisper-1`.** Retargeted the existing `stacks/parakeet/` (sherpa-onnx + our own FastAPI wrapper) from irv-ml1; v3 int8, 25 languages. ⚠ **ORT's CUDA EP compiles kernels lazily and the first decode on sm_120 took 45.7 s** — every later call ~0.5 s; a startup warmup in `app.py` now absorbs it, so the first real request is 0.65 s instead of a 45 s hang that no client would wait through. GPU use was **verified by a process on GPU 3 (922 MiB), not by the `provider=cuda` log line**, because ORT falls back to CPU silently and still returns correct text. Silence → `""` (null control), known sentence → near-exact (positive control). → `persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md` +- `[2026-09-15]` **Parakeet STT live on fv-ml1 GPU 0, behind LiteLLM `ext-stt` / `whisper-1`.** ⚠ **Placed on GPU 3 first, which was wrong — operator caught it.** A ~800 MiB seat should ride the card with the most uncommitted headroom (GPU 0, util 0.88, ~13 GB spare), not put the first fingerprint on the one pristine 96 GB card: vLLM sizes KV cache against TOTAL VRAM, so any tenant on an empty card eats a future full-size seat's profiling margin (flash-next needs 93 of 96 GiB). **GPU 3 is now a deliberate reserve at 2 MiB.** Retargeted the existing `stacks/parakeet/` (sherpa-onnx + our own FastAPI wrapper) from irv-ml1; v3 int8, 25 languages. ⚠ **ORT's CUDA EP compiles kernels lazily and the first decode on sm_120 took 45.7 s** — every later call ~0.5 s; a startup warmup in `app.py` now absorbs it, so the first real request is 0.65 s instead of a 45 s hang that no client would wait through. GPU use was **verified by a process on GPU 3 (922 MiB), not by the `provider=cuda` log line**, because ORT falls back to CPU silently and still returns correct text. Silence → `""` (null control), known sentence → near-exact (positive control). → `persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md` - `[2026-09-15]` **`svos_miranda` Hermes plugin validated; found its load blocker.** Absolute intra-package imports (`from hermes_plugin.x`) could not resolve at the documented install name — fixed by svos-dev at `c964e64`. ⚠ **`hermes plugins validate` and `doctor` can NEVER pass this plugin**, by construction: validate's probe stub is config-blind AND returns `None` from `register_tool` (which the plugin's guard reads as a collision), and doctor runs under a temp `HERMES_HOME` with no config. ⚠ `doctor` exits **0** on ERROR (use `--ci`); `compat` reads a **nonexistent path as a pass**. Roster verified 8/7 by a probe supplying real settings. → `persistent-memory.d/2026-09-15-svos-miranda-plugin-validation.md` diff --git a/servers/fv-ml1/README.md b/servers/fv-ml1/README.md index cc91281..3b05aa9 100644 --- a/servers/fv-ml1/README.md +++ b/servers/fv-ml1/README.md @@ -110,14 +110,18 @@ seats, safe to leave: `mistral-medium-3.5`, `mistral-small-4(-heretic)`, `qwen36-27b-aeon`, `qwen-image-bench`, `vibevoice`, `comfyui`, `kokoro`, `vllm-qwen3`. -**GPU 3 — utility card:** +**Also on GPU 0 (non-vLLM):** | Container | Port | Serves | Notes | |-----------|------|--------|-------| -| `parakeet` | 8300 | Parakeet-TDT 0.6B v3 int8 (25 languages) | ASR via sherpa-onnx, LiteLLM `ext-stt` / `whisper-1`. Relocated from irv-ml1 2026-09-15. `stacks/parakeet/`. | +| `parakeet` | 8300 | Parakeet-TDT 0.6B v3 int8 (25 languages) | ASR via sherpa-onnx, LiteLLM `ext-stt` / `whisper-1`. Relocated from irv-ml1 2026-09-15. ~800 MiB. `stacks/parakeet/`. | -⚠ The other three cards run 85-98% full, so GPU 3 is where a new small seat goes -until something bigger claims it. +⚠ **GPU 3 is deliberately kept EMPTY (2 MiB).** It is the only card that can still +take a full-size seat — `flash-next` needs 93 GiB of 96 — and vLLM sizes its KV +cache against *total* VRAM rather than free VRAM, so even a sub-1 GB tenant there +eats into a future big seat's profiling margin. Small seats go on GPU 0, which has +the most uncommitted headroom (its seats commit util 0.88; GPU 1 is at 0.975 and +GPU 2 at 0.96). **Retired:** - `llama-swap` (former GGUF multiplexer on :9292) — replaced by dedicated diff --git a/stacks/parakeet/.env.example b/stacks/parakeet/.env.example index ab7339f..7b3d990 100644 --- a/stacks/parakeet/.env.example +++ b/stacks/parakeet/.env.example @@ -9,10 +9,14 @@ # cleanly. PARAKEET_TAG=sherpa-onnx-v4 -# Which GPU to pin. fv-ml1 GPU 3 is the utility card — 0/1/2 carry the vLLM -# serving seats and sit at 85-98% VRAM, so this is the only one with room. -# The container sees whichever card this names as cuda:0 internally. -PARAKEET_GPU=3 +# Which GPU to pin. The container sees whichever card this names as cuda:0. +# +# ⚠ GPU 0, deliberately, NOT the empty GPU 3. This seat is ~800 MiB and GPU 0 has +# the most uncommitted headroom of the three working cards (its seats commit +# util 0.88, ~13 GB spare; GPU 1 is at 0.975, GPU 2 at 0.96). Leaving GPU 3 +# untouched keeps a full 96 GB card available for a real seat — vLLM sizes KV +# cache against TOTAL VRAM, so even a 1 GB tenant eats into a big seat's margin. +PARAKEET_GPU=0 # Host port for the FastAPI server (container listens on 8000). 8300 is # fv-ml1's established parakeet port; the 80xx range belongs to the vLLM seats. diff --git a/stacks/parakeet/README.md b/stacks/parakeet/README.md index 91f245a..06caf14 100644 --- a/stacks/parakeet/README.md +++ b/stacks/parakeet/README.md @@ -6,24 +6,35 @@ wrapper over [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) **Server:** fv-ml1 (Fountain Valley, `10.251.50.54`) — moved from irv-ml1 2026-09-15 **Port:** 8300 (container 8000) -**GPU:** **3**, pinned explicitly via `device_ids` — the utility card +**GPU:** **0**, pinned explicitly via `device_ids` — shares the card with two vLLM seats **Image:** `local/parakeet:sherpa-onnx-v4` — built from `Dockerfile` + `app.py` + `entrypoint.sh` in this directory; **we own all the code** **Model:** `parakeet-tdt-0.6b-v3` int8, 25 European languages (~464 MiB) -## Why GPU 3 +## Why GPU 0 and not the empty card -fv-ml1 has four RTX PRO 6000 Blackwell Max-Q (96 GB each). Three carry the -vLLM serving seats and run 85–98 % full; GPU 3 is the utility card and was -empty (2 MiB) at placement time. A 0.6 B int8 ASR model is a rounding error -next to those seats, but it still has to go somewhere that is not fighting -them for VRAM. +fv-ml1 has four RTX PRO 6000 Blackwell Max-Q (96 GB each). This seat holds +**~800 MiB — under 1 %** of one card, so the question is not "where does it +fit" but "whose headroom can it spend most cheaply". -⚠ The pin is `deploy.resources.reservations.devices[].device_ids: ["3"]`, -the fleet convention — **not** `count: all`, which is what the dead on-host -stub used and which would have handed this seat all four cards. Inside the -container the pinned card presents as `cuda:0`, which is what sherpa-onnx's -CUDA execution provider takes by default. +| GPU | committed `--gpu-memory-utilization` | spare | +|---|---|---| +| **0** | 0.40 + 0.48 = **0.88** | ~13 GB ← here | +| 1 | 0.52+0.24+0.10+0.055+0.03+0.03 = **0.975** | ~4.3 GB | +| 2 | **0.96** | ~1.8 GB | +| 3 | — | *kept empty* | + +It was first placed on the empty GPU 3, which was wrong: **vLLM sizes its KV +cache as a fraction of TOTAL VRAM, not free VRAM**, so any resident tenant on an +otherwise-clean card eats directly into the profiling margin of whatever big seat +lands there later — and `flash-next` needs 93 GiB of 96. A 96 GB card at 2 MiB is +worth far more than 800 MiB of it. Moved to GPU 0 the same night. + +⚠ The pin is `deploy.resources.reservations.devices[].device_ids`, the fleet +convention — **not** `count: all`, which is what the dead on-host stub used and +which would have handed this seat all four cards. Inside the container the pinned +card presents as `cuda:0`, which is what sherpa-onnx's CUDA execution provider +takes by default. ## Why not the FastAPI community wrappers @@ -70,13 +81,14 @@ The honest check is to watch the card while a transcription runs: ```bash # on fv-ml1 — terminal 1 watch -n0.2 'nvidia-smi --query-compute-apps=pid,process_name,used_memory \ - --format=csv -i 3' + --format=csv -i 0' # terminal 2 — send real audio, not silence curl -s -F file=@sample.wav http://127.0.0.1:8300/v1/audio/transcriptions ``` -A process must appear **on GPU 3** for the duration. If GPU 3 stays empty, the +A process must appear **on GPU 0** for the duration — ~800 MiB alongside the two +much larger `VLLM::EngineCore` entries. If it never appears, the CUDA EP did not initialise and you are on CPU regardless of what `.env` says. Confirm with the container's own startup log, which names the providers ORT actually registered: diff --git a/stacks/parakeet/compose.yaml b/stacks/parakeet/compose.yaml index b078fe8..8c116c1 100644 --- a/stacks/parakeet/compose.yaml +++ b/stacks/parakeet/compose.yaml @@ -6,16 +6,23 @@ # prebuilt int8 quantized Parakeet-TDT from k2-fsa — and wrote our own ~50-line # wrapper we own end-to-end. # -# HOST: fv-ml1, GPU 3 (relocated from irv-ml1 2026-09-15). GPU 3 is the utility -# card — the other three carry the vLLM serving seats and run 85-98% full, so a -# seat placed anywhere else would fight them for VRAM. +# HOST: fv-ml1, GPU 0 (relocated from irv-ml1 2026-09-15). # -# ⚠ GPU pin is `deploy.resources.reservations.devices[].device_ids`, the fleet +# ⚠ GPU 0, NOT the empty GPU 3. This seat holds ~800 MiB — under 1% of a 96 GB +# card — so it rides on the card with the most uncommitted headroom rather than +# putting the first fingerprint on a pristine one. GPU 0's seats commit +# util 0.40 + 0.48 = 0.88, leaving ~13 GB; GPU 1 is at 0.975 and GPU 2 at 0.96, +# both too tight. Keeping GPU 3 at 2 MiB means it can still take a full-card seat +# (flash-next needs 93 GiB) without a neighbour eating its profiling margin — +# vLLM sizes KV cache as a fraction of TOTAL VRAM, not free VRAM, so a resident +# tenant on an otherwise-empty card is worth more than its megabytes suggest. +# +# ⚠ The pin is `deploy.resources.reservations.devices[].device_ids`, the fleet # convention — NOT `runtime: nvidia` + NVIDIA_VISIBLE_DEVICES, and NOT # `count: all` (which is what the dead on-host stub did, and would have let this -# tiny ASR seat see all four cards including the three that are full). -# device_ids ["3"] presents that card as cuda:0 INSIDE the container, which is -# what sherpa-onnx's CUDAExecutionProvider takes by default. +# tiny ASR seat see all four cards). device_ids presents the pinned card as +# cuda:0 INSIDE the container, which is what sherpa-onnx's +# CUDAExecutionProvider takes by default. # # Model weights (~460 MB int8) download on first run via the entrypoint to # ${PARAKEET_MODELS_DIR}/ (persistent host bind mount). Subsequent starts skip @@ -51,7 +58,7 @@ services: reservations: devices: - driver: nvidia - device_ids: ["${PARAKEET_GPU:-3}"] + device_ids: ["${PARAKEET_GPU:-0}"] capabilities: [gpu] networks: - tnet @@ -67,7 +74,7 @@ services: - homepage.group=AI - Audio Tools - homepage.name=Parakeet ASR - homepage.icon=mdi-microphone - - homepage.description=Parakeet-TDT speech-to-text via sherpa-onnx (fv-ml1 GPU 3) + - homepage.description=Parakeet-TDT speech-to-text via sherpa-onnx (fv-ml1 GPU 0) - homepage.href=http://10.251.50.54:${PARAKEET_PORT} networks: