vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(parakeet): move the seat from the empty GPU 3 to GPU 0

Browse Source 
Placed on GPU 3 first because it was the empty card. That was the wrong read:
the seat is ~800 MiB, under 1% of a 96 GB card, so the question was never "where
does it fit" but "whose headroom is cheapest to spend".

vLLM sizes its KV cache as a fraction of TOTAL VRAM, not free VRAM. A resident
tenant on an otherwise-clean card therefore does not cost its own megabytes — it
costs the profiling margin of whatever full-size seat lands there later, and
flash-next needs 93 GiB of 96. A 96 GB card at 2 MiB can still take that; the
same card at 922 MiB is one where the next big seat needs its utilization
hand-trimmed, which this repo's flash-next history shows is both thin and silent
when it goes wrong.

Committed utilization per card is the number that governs, not free bytes:

    GPU 0   0.40 + 0.48                       = 0.88    ~13 GB spare  <- moved here
    GPU 1   0.52+0.24+0.10+0.055+0.03+0.03    = 0.975   ~4.3 GB
    GPU 2   0.96                                        ~1.8 GB
    GPU 3   -                                           kept empty as reserve

GPU 3 is back to 2 MiB / 97,247 MiB free and is now documented as a deliberate
reserve rather than a spare.

Post-move n=5 on the same clip: 0.68 / 0.54 / 0.54 / 0.52 / 0.53 s, median 0.54 s
against 0.50 s on GPU 3. The spreads overlap at this sample size and no difference
is claimed; the GPU 3 figure was taken on an idle card and is now noted as a best
case, since the seat shares GPU 0 with the hot serving path. Silence control and
the gateway round-trip both re-verified after the move.

Also records both Parakeet endpoints (FV v3 on Blackwell, IRV v2 on a 3090) and
the four confounds that make them not an A/B pair, sent to tts-dev for the bench.
This commit is contained in:
vh
2026-09-15 01:50:25 -07:00
parent b9b14b5baf
commit caa04801f3
6 changed files with 117 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
servers/fv-ml1/README.md
+8 -4
View File
@@ -110,14 +110,18 @@ seats, safe to leave: `mistral-medium-3.5`, `mistral-small-4(-heretic)`,
`qwen36-27b-aeon`, `qwen-image-bench`, `vibevoice`, `comfyui`, `kokoro`,
`vllm-qwen3`.
**GPU 3 — utility card:**
**Also on GPU 0 (non-vLLM):**
| Container | Port | Serves | Notes |
|-----------|------|--------|-------|
| `parakeet` | 8300 | Parakeet-TDT 0.6B v3 int8 (25 languages) | ASR via sherpa-onnx, LiteLLM `ext-stt` / `whisper-1`. Relocated from irv-ml1 2026-09-15. `stacks/parakeet/`. |
| `parakeet` | 8300 | Parakeet-TDT 0.6B v3 int8 (25 languages) | ASR via sherpa-onnx, LiteLLM `ext-stt` / `whisper-1`. Relocated from irv-ml1 2026-09-15. ~800 MiB. `stacks/parakeet/`. |
⚠ The other three cards run 85-98% full, so GPU 3 is where a new small seat goes
until something bigger claims it.
⚠ **GPU 3 is deliberately kept EMPTY (2 MiB).** It is the only card that can still
take a full-size seat — `flash-next` needs 93 GiB of 96 — and vLLM sizes its KV
cache against *total* VRAM rather than free VRAM, so even a sub-1 GB tenant there
eats into a future big seat's profiling margin. Small seats go on GPU 0, which has
the most uncommitted headroom (its seats commit util 0.88; GPU 1 is at 0.975 and
GPU 2 at 0.96).
**Retired:**
- `llama-swap` (former GGUF multiplexer on :9292) — replaced by dedicated