feat(parakeet): stand up Parakeet STT on fv-ml1 GPU 3 + LiteLLM ext-stt/whisper-1
Retargets the existing sherpa-onnx stack from irv-ml1 to fv-ml1's utility card and puts it behind the gateway. GPU 3 was the only card with room: 0/1/2 carry the vLLM seats at 84-95.5 GB of 96. Changes: - compose: pin GPU via `device_ids: ["3"]` (the dead on-host stub used `count: all`, which would have handed a 0.6B ASR seat all four cards); join traefik-net; port 8300; homepage href to the live FV address. - .env.example: default to the v3 int8 model (25 European languages, 464 MiB) rather than English-only v2; models to /tank/parakeet/models. - app.py: warm the recognizer at startup before uvicorn accepts traffic. The warmup is not an optimisation. ONNX Runtime's CUDA EP compiles and autotunes lazily on the FIRST DECODE, and on sm_120 that measured 45.7s cold (reproduced at 45.1s on a second container) against ~0.50s warm. A 45s first request is indistinguishable from a hang and LiteLLM's default timeout abandons it long before it returns. Decoding 1s of silence at load moves the cost inside the healthcheck's 300s start_period; first real request after restart is now 0.65s. Verification, because "provider=cuda" in the log is only an echo of the env var: ORT falls back to CPU silently and still returns correct text, so the service being up and the transcript being right establishes nothing. The discriminator is a process on GPU 3 (922 MiB), confirmed. Controls both directions — a known TTS sentence transcribes near-exactly (positive), 3s of digital silence returns empty (null). Warm throughput 0.50s median on an 8.52s clip, n=5, spread 0.47-0.65s, single-stream, one clip: a smoke measurement with its harness stated, not a benchmark. Gateway aliases `ext-stt` (engine-neutral, mirrors ext-tts) and `whisper-1` (OpenAI-compatible drop-in) registered via POST /model/new, i.e. LiteLLM's Postgres store where the ext-tts family already lives — no gateway restart, and config.yaml is consequently not a complete picture of what the gateway serves. Both verified end to end. The aliases use a raw IP deliberately: ana-docker resolves no .internal names at all (resolv.conf points at 1.1.1.1), and LiteLLM only reaches irv-ml1 through a hand-pinned extra_hosts entry. A second hosts entry would mean recreating the container and bouncing the gateway for every consumer. Also records the svos_miranda plugin validation pass and its structural findings, and notes that the irv-ml1 parakeet is still running — there are two now, and retiring the old one is the operator's call.
This commit is contained in:
@@ -0,0 +1,86 @@
|
||||
# Parakeet STT on fv-ml1 GPU 3 (2026-09-15)
|
||||
|
||||
Operator asked for an STT service on fv-ml1's utility GPU plus a LiteLLM alias.
|
||||
|
||||
## What it is
|
||||
|
||||
`stacks/parakeet/` — Parakeet-TDT 0.6B **v3** int8 ONNX (25 European languages,
|
||||
464 MiB) under sherpa-onnx, behind ~90 lines of FastAPI we own. Container
|
||||
`parakeet`, port **8300**, **GPU 3** pinned by `device_ids`. Image
|
||||
`local/parakeet:sherpa-onnx-v4` (5.09 GB).
|
||||
|
||||
Not greenfield: the stack already existed, targeting irv-ml1. Retargeted rather
|
||||
than rewritten — the Ampere→Blackwell move was the only real question.
|
||||
|
||||
## Why GPU 3
|
||||
|
||||
GPU 0 = 84/96 GB, GPU 1 = 92.9/96, GPU 2 = 95.5/96 (the vLLM seats). **GPU 3 was
|
||||
at 2 MiB.** The dead on-host stub used `count: all`, which would have handed this
|
||||
seat all four cards; replaced with an explicit `device_ids: ["3"]` per the fleet
|
||||
convention. Inside the container the pinned card presents as `cuda:0`, which is
|
||||
what ORT's CUDA EP takes by default.
|
||||
|
||||
## ⚠ The finding worth keeping: a 45-second first decode
|
||||
|
||||
ONNX Runtime's CUDA EP compiles and autotunes lazily, on the **first decode**, not
|
||||
at session creation. On sm_120:
|
||||
|
||||
| | measured |
|
||||
|---|---|
|
||||
| first decode, cold container | **45.7 s** (n=1), reproduced at **45.1 s** on a second container |
|
||||
| warm, 8.52 s clip | **0.50 s** median (n=5: 0.65 / 0.53 / 0.48 / 0.47 / 0.50) |
|
||||
|
||||
≈17× realtime warm, single-stream, one 8.52 s clip, int8, GPU 3 idle otherwise.
|
||||
That is a smoke measurement with its harness stated, **not** a benchmark — no
|
||||
concurrency sweep, no length sweep, one clip.
|
||||
|
||||
A 45 s first request is indistinguishable from a hang to any caller, and LiteLLM's
|
||||
default timeout would abandon it. `_warm()` in `app.py` now decodes 1 s of silence
|
||||
before uvicorn accepts traffic, so the cost lands inside the healthcheck's 300 s
|
||||
`start_period`. First real request after restart: **0.65 s**.
|
||||
|
||||
## ⚠⚠ "provider=cuda" is not evidence the GPU is being used
|
||||
|
||||
ORT's CUDA EP **falls back to CPU silently** — the process lives, answers 200, and
|
||||
returns *correct text*, just slowly. Our own log line `loading OfflineRecognizer
|
||||
(provider=cuda...)` merely echoes the env var and proves nothing.
|
||||
|
||||
The discriminator that actually settles it:
|
||||
|
||||
```
|
||||
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv -i 3
|
||||
-> 1588301, /opt/venv/bin/python3, 922 MiB
|
||||
```
|
||||
|
||||
Timing is **not** a sufficient check either — the int8 model is fast enough on a
|
||||
96-thread EPYC that a CPU fallback still looks brisk on short clips.
|
||||
|
||||
Controls run, both directions:
|
||||
- **positive** — known TTS sentence in, near-exact transcript out (two word errors,
|
||||
both attributable to the source audio: an inserted "um", "Foun Valley").
|
||||
- **null** — 3 s of digital silence → `{"text": ""}`. The instrument does not
|
||||
manufacture signal.
|
||||
|
||||
## LiteLLM
|
||||
|
||||
Two aliases, both `mode: audio_transcription` → `http://10.251.50.54:8300/v1`:
|
||||
`ext-stt` (engine-neutral fleet name, mirrors `ext-tts`) and `whisper-1`
|
||||
(OpenAI-compatible drop-in). Both verified end-to-end through the gateway.
|
||||
|
||||
Registered via `POST /model/new`, i.e. the **Postgres store**, not `config.yaml` —
|
||||
that is where the `ext-tts` family lives, and it needs no gateway restart.
|
||||
⚠ Corollary: `config.yaml` is NOT a complete picture of what the gateway serves
|
||||
(it lists 35 models; the gateway serves 40, and carries stale entries like
|
||||
`granite-4.1-8b`). Read `/v1/models` or `/model/info`, never just the file.
|
||||
|
||||
⚠ **Raw IP on purpose** — see the ana-docker DNS row in the index.
|
||||
|
||||
## Loose ends
|
||||
|
||||
- The **irv-ml1 parakeet is still running** (healthz 200 on `100.64.0.6:8765`).
|
||||
Two Parakeets now. Retiring the old one is the operator's call — not touched.
|
||||
- `/opt/docker/compose/parakeet` and `/tank/parakeet` normalised to `root:docker
|
||||
2775`; the rest of fv-ml1's deploy tree is still `lkraven:lkraven` (it was not
|
||||
part of the 5-host normalisation).
|
||||
- `servers/fv-ml1/README.md` is still broadly stale — it claims 2 GPUs and a
|
||||
2026-07-22 stack list. Only the parakeet/GPU-3 rows were corrected.
|
||||
@@ -0,0 +1,64 @@
|
||||
# svos_miranda Hermes plugin — validation pass (2026-09-15)
|
||||
|
||||
svos-dev asked infra-ops to run `hermes plugins validate` → `doctor` → `compat`
|
||||
on `/home/lkraven/development/svos/hermes_plugin/` and report before enabling.
|
||||
Hermes Agent v0.21.1 (2026.9.7), local `b88e6776`, on nh3-dev.
|
||||
|
||||
## The blocker (found, fixed by svos-dev at `c964e64`)
|
||||
|
||||
Three absolute intra-package imports — `from hermes_plugin._vendored`, `.forward`,
|
||||
`.jwt` — pinned the package to its **source directory name**. The documented
|
||||
install renames it to `svos_miranda`, and Hermes loads directory plugins under the
|
||||
`hermes_plugins.<dir>` namespace; in neither case does a top-level `hermes_plugin`
|
||||
exist. Fix: three relative imports.
|
||||
|
||||
⚠ **The harness hid this from three different readers.** My first `validate` passed
|
||||
the import only because my cwd was the SVOS repo root. svos-dev's test suite
|
||||
imports `hermes_plugin.*` from that same root, and their editable install resolves
|
||||
the name from anywhere on the box — it only reproduced for them once `sys.path` was
|
||||
stripped. Same class as `feedback_filters_that_silently_narrow_the_window`: the
|
||||
instrument carried the result.
|
||||
|
||||
## ⚠ Two of the three commands CANNOT pass this plugin, ever
|
||||
|
||||
Neither is fixable from the plugin side. Both are now documented in its README.
|
||||
|
||||
- **`validate`** — two independent causes. Its `RecordingContext.get_config`
|
||||
(`hermes_cli/plugin_validate.py:219-222`) returns the **default for every key**,
|
||||
ignoring `config.yaml` entirely, so `dispatch_key` is always `""`. And its
|
||||
`register_tool` returns `None`, which the plugin's INV-P6 guard correctly reads
|
||||
as a name collision — so even with a key supplied it raises on the first tool.
|
||||
- **`doctor`** — runs `register()` under a **temp `HERMES_HOME`** with sockets
|
||||
blocked, so no config exists there either.
|
||||
|
||||
## Tool-level gotchas worth remembering
|
||||
|
||||
- ⚠ **`hermes plugins doctor` exits 0 even when it prints ERROR.** Needs `--ci`.
|
||||
- ⚠ **`hermes plugins compat <nonexistent-path>` prints ✓ and exits 0.** A typo'd
|
||||
path reads as a pass. (The instrument itself is sound — verified with a throwaway
|
||||
plugin importing a real deprecated path, which it flagged with file:line, exit 1.)
|
||||
- ⚠ **`doctor`'s sandbox registry starts EMPTY — 0 entries, no built-ins.** So
|
||||
doctor cannot detect tool-name collisions at all. `validate`'s separate static
|
||||
"built-in tool collisions" check is what covers that.
|
||||
- The real `PluginContext.register_tool` (`hermes_cli/plugins.py:449-491`) returns
|
||||
a truthy `PluginRegistration` on success — confirmed against the live runtime.
|
||||
|
||||
## Roster verified another way
|
||||
|
||||
Since neither command can supply config, a probe mirroring validate's context but
|
||||
returning real settings and a truthy handle gave: **8 tools** with
|
||||
`repo_read_enabled: true`, **7** with false or omitted, names matching
|
||||
`plugin.yaml` exactly, zero hooks/middleware/commands. All nine settings-validation
|
||||
controls (quoted booleans, `"90 s"`, zero/negative timeouts, empty/whitespace
|
||||
strings) raise errors naming their own key.
|
||||
|
||||
## A false finding I caught on myself
|
||||
|
||||
A probe registering `read_file` got back a `PluginRegistration` instead of the
|
||||
expected refusal — which looked like the plugin's collision reading was wrong. It
|
||||
was not: doctor's sandbox holds no built-ins, so nothing was claimed and **my
|
||||
positive case was not positive.** Reported as untested rather than as a finding.
|
||||
|
||||
## Where enabling stands
|
||||
|
||||
Blocked on the operator only. See the OPEN row in the index for the sequence.
|
||||
Reference in New Issue
Block a user