vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(parakeet): move the seat from the empty GPU 3 to GPU 0

Browse Source 
Placed on GPU 3 first because it was the empty card. That was the wrong read:
the seat is ~800 MiB, under 1% of a 96 GB card, so the question was never "where
does it fit" but "whose headroom is cheapest to spend".

vLLM sizes its KV cache as a fraction of TOTAL VRAM, not free VRAM. A resident
tenant on an otherwise-clean card therefore does not cost its own megabytes — it
costs the profiling margin of whatever full-size seat lands there later, and
flash-next needs 93 GiB of 96. A 96 GB card at 2 MiB can still take that; the
same card at 922 MiB is one where the next big seat needs its utilization
hand-trimmed, which this repo's flash-next history shows is both thin and silent
when it goes wrong.

Committed utilization per card is the number that governs, not free bytes:

    GPU 0   0.40 + 0.48                       = 0.88    ~13 GB spare  <- moved here
    GPU 1   0.52+0.24+0.10+0.055+0.03+0.03    = 0.975   ~4.3 GB
    GPU 2   0.96                                        ~1.8 GB
    GPU 3   -                                           kept empty as reserve

GPU 3 is back to 2 MiB / 97,247 MiB free and is now documented as a deliberate
reserve rather than a spare.

Post-move n=5 on the same clip: 0.68 / 0.54 / 0.54 / 0.52 / 0.53 s, median 0.54 s
against 0.50 s on GPU 3. The spreads overlap at this sample size and no difference
is claimed; the GPU 3 figure was taken on an idle card and is now noted as a best
case, since the seat shares GPU 0 with the hot serving path. Silence control and
the gateway round-trip both re-verified after the move.

Also records both Parakeet endpoints (FV v3 on Blackwell, IRV v2 on a 3090) and
the four confounds that make them not an A/B pair, sent to tts-dev for the bench.
This commit is contained in:
Vuong Hoang
2026-09-15 01:50:25 -07:00
parent b9b14b5baf
commit caa04801f3
6 changed files with 117 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md
+58 -10
View File
@@ -6,19 +6,45 @@ Operator asked for an STT service on fv-ml1's utility GPU plus a LiteLLM alias.
`stacks/parakeet/` — Parakeet-TDT 0.6B **v3** int8 ONNX (25 European languages,
464 MiB) under sherpa-onnx, behind ~90 lines of FastAPI we own. Container
`parakeet`, port **8300**, **GPU 3** pinned by `device_ids`. Image
`parakeet`, port **8300**, **GPU 0** pinned by `device_ids`. Image
`local/parakeet:sherpa-onnx-v4` (5.09 GB).
Not greenfield: the stack already existed, targeting irv-ml1. Retargeted rather
than rewritten — the Ampere→Blackwell move was the only real question.
## Why GPU 3
## ⚠ Placement — got this wrong first, operator caught it
GPU 0 = 84/96 GB, GPU 1 = 92.9/96, GPU 2 = 95.5/96 (the vLLM seats). **GPU 3 was
at 2 MiB.** The dead on-host stub used `count: all`, which would have handed this
seat all four cards; replaced with an explicit `device_ids: ["3"]` per the fleet
convention. Inside the container the pinned card presents as `cuda:0`, which is
what ORT's CUDA EP takes by default.
Placed on the empty **GPU 3** initially, reading "the utility gpu" as "the spare
card". Operator's correction: *"1gb total vram pressure — and you didn't load it on
gpu 0?"* He is right, and the reason is sharper than "it fits anywhere".
**vLLM sizes its KV cache as a fraction of TOTAL VRAM, not free VRAM.** So a
resident tenant on an otherwise-clean card does not cost its own megabytes — it
costs a future full-size seat's profiling margin. `flash-next` needs **93 GiB of
96**. A 96 GB card at 2 MiB is a card that can still take that; the same card at
922 MiB is a card where the next big seat's `--gpu-memory-utilization` has to be
hand-trimmed, and the flash-next history in this repo shows exactly how thin and
how silent that failure gets.
The right question is not "where does 800 MiB fit" but "whose headroom is cheapest
to spend":
| GPU | committed util | spare |
|---|---|---|
| **0** | 0.40 + 0.48 = **0.88** | ~13 GB ← moved here |
| 1 | **0.975** (six small seats) | ~4.3 GB |
| 2 | **0.96** (flash-next) | ~1.8 GB |
| 3 | — | **kept empty as reserve** |
Moved the same night: one env var (`PARAKEET_GPU`) plus `compose up -d`. GPU 3 back
to 2 MiB / 97,247 MiB free. Post-move n=5 on the same clip: 0.68 / 0.54 / 0.54 /
0.52 / 0.53 s, median 0.54 s — **indistinguishable from the GPU 3 median of 0.50 s
at this sample size**; the spreads overlap and no difference is claimed.
The dead on-host stub used `count: all`, which would have handed this seat all four
cards; replaced with an explicit `device_ids` pin per the fleet convention. Inside
the container the pinned card presents as `cuda:0`, which is what ORT's CUDA EP
takes by default.
## ⚠ The finding worth keeping: a 45-second first decode
@@ -30,7 +56,9 @@ at session creation. On sm_120:
| first decode, cold container | **45.7 s** (n=1), reproduced at **45.1 s** on a second container |
| warm, 8.52 s clip | **0.50 s** median (n=5: 0.65 / 0.53 / 0.48 / 0.47 / 0.50) |
≈17× realtime warm, single-stream, one 8.52 s clip, int8, GPU 3 idle otherwise.
≈17x realtime warm, single-stream, one 8.52 s clip, int8. ⚠ Measured on GPU 3 while
it was idle; the seat now lives on GPU 0 beside the hot serving path, so treat that
number as a best case.
That is a smoke measurement with its harness stated, **not** a benchmark — no
concurrency sweep, no length sweep, one clip.
@@ -48,8 +76,8 @@ returns *correct text*, just slowly. Our own log line `loading OfflineRecognizer
The discriminator that actually settles it:
```
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv -i 3
-> 1588301, /opt/venv/bin/python3, 922 MiB
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv -i 0
-> 1594431, /opt/venv/bin/python3, 794 MiB (beside two VLLM::EngineCore entries)
```
Timing is **not** a sufficient check either — the int8 model is fast enough on a
@@ -84,3 +112,23 @@ that is where the `ext-tts` family lives, and it needs no gateway restart.
part of the 5-host normalisation).
- `servers/fv-ml1/README.md` is still broadly stale — it claims 2 GPUs and a
2026-07-22 stack list. Only the parakeet/GPU-3 rows were corrected.
## Two Parakeets, and the bench Vuong asked for
Both are live; endpoints sent to **tts-dev** 2026-09-15 for a head-to-head.
| | FV (new) | IRV (existing, up 2 months) |
|---|---|---|
| endpoint | `http://10.251.50.54:8300/v1/audio/transcriptions` | `http://100.64.0.6:8765/...` or `http://10.6.110.50:8765/...` |
| model | parakeet-tdt-0.6b-**v3** int8, 25 languages | parakeet-tdt-0.6b-**v2** int8, English only |
| GPU | RTX PRO 6000 Blackwell **sm_120**, GPU 0, shares with 2 vLLM seats | RTX 3090 **sm_86**, shares with 4 processes, 4.0 GB free |
| image | `local/parakeet:sherpa-onnx-v4` (has startup warmup) | `local/parakeet:sherpa-onnx-v2` (no warmup) |
⚠ **`10.100.79.3:8765` is DEAD** — the retired wg0 lifeline, still the href on IRV's
Homepage card. Same for `Speaches ASR` at `10.100.79.3:8204`.
⚠ **These were never an A/B pair — four things differ at once** (model version,
GPU architecture, card contention, image). A WER delta is a **v2-vs-v3** result, not
an FV-vs-IRV one. Offered tts-dev a v2 container on FV as a second compose project so
accuracy can be varied one factor at a time; not built unless they take it up.
persistent-memory.md
+1 -1
View File
@@ -175,7 +175,7 @@ hardened for ha-dev (`d1769ed` ff); `kb` KB-search tool (`68fa80f`).
## Recent decisions
- `[2026-09-15]` **Parakeet STT live on fv-ml1 GPU 3, behind LiteLLM `ext-stt` / `whisper-1`.** Retargeted the existing `stacks/parakeet/` (sherpa-onnx + our own FastAPI wrapper) from irv-ml1; v3 int8, 25 languages. ⚠ **ORT's CUDA EP compiles kernels lazily and the first decode on sm_120 took 45.7 s** — every later call ~0.5 s; a startup warmup in `app.py` now absorbs it, so the first real request is 0.65 s instead of a 45 s hang that no client would wait through. GPU use was **verified by a process on GPU 3 (922 MiB), not by the `provider=cuda` log line**, because ORT falls back to CPU silently and still returns correct text. Silence → `""` (null control), known sentence → near-exact (positive control). → `persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md`
- `[2026-09-15]` **Parakeet STT live on fv-ml1 GPU 0, behind LiteLLM `ext-stt` / `whisper-1`.** ⚠ **Placed on GPU 3 first, which was wrong — operator caught it.** A ~800 MiB seat should ride the card with the most uncommitted headroom (GPU 0, util 0.88, ~13 GB spare), not put the first fingerprint on the one pristine 96 GB card: vLLM sizes KV cache against TOTAL VRAM, so any tenant on an empty card eats a future full-size seat's profiling margin (flash-next needs 93 of 96 GiB). **GPU 3 is now a deliberate reserve at 2 MiB.** Retargeted the existing `stacks/parakeet/` (sherpa-onnx + our own FastAPI wrapper) from irv-ml1; v3 int8, 25 languages. ⚠ **ORT's CUDA EP compiles kernels lazily and the first decode on sm_120 took 45.7 s** — every later call ~0.5 s; a startup warmup in `app.py` now absorbs it, so the first real request is 0.65 s instead of a 45 s hang that no client would wait through. GPU use was **verified by a process on GPU 3 (922 MiB), not by the `provider=cuda` log line**, because ORT falls back to CPU silently and still returns correct text. Silence → `""` (null control), known sentence → near-exact (positive control). → `persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md`
- `[2026-09-15]` **`svos_miranda` Hermes plugin validated; found its load blocker.** Absolute intra-package imports (`from hermes_plugin.x`) could not resolve at the documented install name — fixed by svos-dev at `c964e64`. ⚠ **`hermes plugins validate` and `doctor` can NEVER pass this plugin**, by construction: validate's probe stub is config-blind AND returns `None` from `register_tool` (which the plugin's guard reads as a collision), and doctor runs under a temp `HERMES_HOME` with no config. ⚠ `doctor` exits **0** on ERROR (use `--ci`); `compat` reads a **nonexistent path as a pass**. Roster verified 8/7 by a probe supplying real settings. → `persistent-memory.d/2026-09-15-svos-miranda-plugin-validation.md`
servers/fv-ml1/README.md
+8 -4
View File
@@ -110,14 +110,18 @@ seats, safe to leave: `mistral-medium-3.5`, `mistral-small-4(-heretic)`,
`qwen36-27b-aeon`, `qwen-image-bench`, `vibevoice`, `comfyui`, `kokoro`,
`vllm-qwen3`.
**GPU 3 — utility card:**
**Also on GPU 0 (non-vLLM):**
| Container | Port | Serves | Notes |
|-----------|------|--------|-------|
| `parakeet` | 8300 | Parakeet-TDT 0.6B v3 int8 (25 languages) | ASR via sherpa-onnx, LiteLLM `ext-stt` / `whisper-1`. Relocated from irv-ml1 2026-09-15. `stacks/parakeet/`. |
| `parakeet` | 8300 | Parakeet-TDT 0.6B v3 int8 (25 languages) | ASR via sherpa-onnx, LiteLLM `ext-stt` / `whisper-1`. Relocated from irv-ml1 2026-09-15. ~800 MiB. `stacks/parakeet/`. |
⚠ The other three cards run 85-98% full, so GPU 3 is where a new small seat goes
until something bigger claims it.
⚠ **GPU 3 is deliberately kept EMPTY (2 MiB).** It is the only card that can still
take a full-size seat — `flash-next` needs 93 GiB of 96 — and vLLM sizes its KV
cache against *total* VRAM rather than free VRAM, so even a sub-1 GB tenant there
eats into a future big seat's profiling margin. Small seats go on GPU 0, which has
the most uncommitted headroom (its seats commit util 0.88; GPU 1 is at 0.975 and
GPU 2 at 0.96).
**Retired:**
- `llama-swap` (former GGUF multiplexer on :9292) — replaced by dedicated
stacks/parakeet/.env.example
+8 -4
View File
@@ -9,10 +9,14 @@
# cleanly.
PARAKEET_TAG=sherpa-onnx-v4
# Which GPU to pin. fv-ml1 GPU 3 is the utility card — 0/1/2 carry the vLLM
# serving seats and sit at 85-98% VRAM, so this is the only one with room.
# The container sees whichever card this names as cuda:0 internally.
PARAKEET_GPU=3
# Which GPU to pin. The container sees whichever card this names as cuda:0.
#
# ⚠ GPU 0, deliberately, NOT the empty GPU 3. This seat is ~800 MiB and GPU 0 has
# the most uncommitted headroom of the three working cards (its seats commit
# util 0.88, ~13 GB spare; GPU 1 is at 0.975, GPU 2 at 0.96). Leaving GPU 3
# untouched keeps a full 96 GB card available for a real seat — vLLM sizes KV
# cache against TOTAL VRAM, so even a 1 GB tenant eats into a big seat's margin.
PARAKEET_GPU=0
# Host port for the FastAPI server (container listens on 8000). 8300 is
# fv-ml1's established parakeet port; the 80xx range belongs to the vLLM seats.
stacks/parakeet/README.md
+26 -14
View File
@@ -6,24 +6,35 @@ wrapper over [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx)
**Server:** fv-ml1 (Fountain Valley, `10.251.50.54`) — moved from irv-ml1 2026-09-15
**Port:** 8300 (container 8000)
**GPU:** **3**, pinned explicitly via `device_ids` — the utility card
**GPU:** **0**, pinned explicitly via `device_ids` — shares the card with two vLLM seats
**Image:** `local/parakeet:sherpa-onnx-v4` — built from `Dockerfile` +
`app.py` + `entrypoint.sh` in this directory; **we own all the code**
**Model:** `parakeet-tdt-0.6b-v3` int8, 25 European languages (~464 MiB)
## Why GPU 3
## Why GPU 0 and not the empty card
fv-ml1 has four RTX PRO 6000 Blackwell Max-Q (96 GB each). Three carry the
vLLM serving seats and run 85–98 % full; GPU 3 is the utility card and was
empty (2 MiB) at placement time. A 0.6 B int8 ASR model is a rounding error
next to those seats, but it still has to go somewhere that is not fighting
them for VRAM.
fv-ml1 has four RTX PRO 6000 Blackwell Max-Q (96 GB each). This seat holds
**~800 MiB — under 1 %** of one card, so the question is not "where does it
fit" but "whose headroom can it spend most cheaply".
⚠ The pin is `deploy.resources.reservations.devices[].device_ids: ["3"]`,
the fleet convention — **not** `count: all`, which is what the dead on-host
stub used and which would have handed this seat all four cards. Inside the
container the pinned card presents as `cuda:0`, which is what sherpa-onnx's
CUDA execution provider takes by default.
| GPU | committed `--gpu-memory-utilization` | spare |
|---|---|---|
| **0** | 0.40 + 0.48 = **0.88** | ~13 GB ← here |
| 1 | 0.52+0.24+0.10+0.055+0.03+0.03 = **0.975** | ~4.3 GB |
| 2 | **0.96** | ~1.8 GB |
| 3 | — | *kept empty* |
It was first placed on the empty GPU 3, which was wrong: **vLLM sizes its KV
cache as a fraction of TOTAL VRAM, not free VRAM**, so any resident tenant on an
otherwise-clean card eats directly into the profiling margin of whatever big seat
lands there later — and `flash-next` needs 93 GiB of 96. A 96 GB card at 2 MiB is
worth far more than 800 MiB of it. Moved to GPU 0 the same night.
⚠ The pin is `deploy.resources.reservations.devices[].device_ids`, the fleet
convention — **not** `count: all`, which is what the dead on-host stub used and
which would have handed this seat all four cards. Inside the container the pinned
card presents as `cuda:0`, which is what sherpa-onnx's CUDA execution provider
takes by default.
## Why not the FastAPI community wrappers
@@ -70,13 +81,14 @@ The honest check is to watch the card while a transcription runs:
```bash
# on fv-ml1 — terminal 1
watch -n0.2 'nvidia-smi --query-compute-apps=pid,process_name,used_memory \
--format=csv -i 3'
--format=csv -i 0'
# terminal 2 — send real audio, not silence
curl -s -F file=@sample.wav http://127.0.0.1:8300/v1/audio/transcriptions
```
A process must appear **on GPU 3** for the duration. If GPU 3 stays empty, the
A process must appear **on GPU 0** for the duration — ~800 MiB alongside the two
much larger `VLLM::EngineCore` entries. If it never appears, the
CUDA EP did not initialise and you are on CPU regardless of what `.env` says.
Confirm with the container's own startup log, which names the providers ORT
actually registered:
stacks/parakeet/compose.yaml
+16 -9
View File
@@ -6,16 +6,23 @@
# prebuilt int8 quantized Parakeet-TDT from k2-fsa — and wrote our own ~50-line
# wrapper we own end-to-end.
#
# HOST: fv-ml1, GPU 3 (relocated from irv-ml1 2026-09-15). GPU 3 is the utility
# card — the other three carry the vLLM serving seats and run 85-98% full, so a
# seat placed anywhere else would fight them for VRAM.
# HOST: fv-ml1, GPU 0 (relocated from irv-ml1 2026-09-15).
#
# ⚠ GPU pin is `deploy.resources.reservations.devices[].device_ids`, the fleet
# ⚠ GPU 0, NOT the empty GPU 3. This seat holds ~800 MiB — under 1% of a 96 GB
# card — so it rides on the card with the most uncommitted headroom rather than
# putting the first fingerprint on a pristine one. GPU 0's seats commit
# util 0.40 + 0.48 = 0.88, leaving ~13 GB; GPU 1 is at 0.975 and GPU 2 at 0.96,
# both too tight. Keeping GPU 3 at 2 MiB means it can still take a full-card seat
# (flash-next needs 93 GiB) without a neighbour eating its profiling margin —
# vLLM sizes KV cache as a fraction of TOTAL VRAM, not free VRAM, so a resident
# tenant on an otherwise-empty card is worth more than its megabytes suggest.
#
# ⚠ The pin is `deploy.resources.reservations.devices[].device_ids`, the fleet
# convention — NOT `runtime: nvidia` + NVIDIA_VISIBLE_DEVICES, and NOT
# `count: all` (which is what the dead on-host stub did, and would have let this
# tiny ASR seat see all four cards including the three that are full).
# device_ids ["3"] presents that card as cuda:0 INSIDE the container, which is
# what sherpa-onnx's CUDAExecutionProvider takes by default.
# tiny ASR seat see all four cards). device_ids presents the pinned card as
# cuda:0 INSIDE the container, which is what sherpa-onnx's
# CUDAExecutionProvider takes by default.
#
# Model weights (~460 MB int8) download on first run via the entrypoint to
# ${PARAKEET_MODELS_DIR}/ (persistent host bind mount). Subsequent starts skip
@@ -51,7 +58,7 @@ services:
reservations:
devices:
- driver: nvidia
device_ids: ["${PARAKEET_GPU:-3}"]
device_ids: ["${PARAKEET_GPU:-0}"]
capabilities: [gpu]
networks:
- tnet
@@ -67,7 +74,7 @@ services:
- homepage.group=AI - Audio Tools
- homepage.name=Parakeet ASR
- homepage.icon=mdi-microphone
- homepage.description=Parakeet-TDT speech-to-text via sherpa-onnx (fv-ml1 GPU 3)
- homepage.description=Parakeet-TDT speech-to-text via sherpa-onnx (fv-ml1 GPU 0)
- homepage.href=http://10.251.50.54:${PARAKEET_PORT}
networks: