vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(parakeet): stand up Parakeet STT on fv-ml1 GPU 3 + LiteLLM ext-stt/whisper-1

Browse Source 
Retargets the existing sherpa-onnx stack from irv-ml1 to fv-ml1's utility card
and puts it behind the gateway. GPU 3 was the only card with room: 0/1/2 carry
the vLLM seats at 84-95.5 GB of 96.

Changes:

- compose: pin GPU via `device_ids: ["3"]` (the dead on-host stub used
  `count: all`, which would have handed a 0.6B ASR seat all four cards);
  join traefik-net; port 8300; homepage href to the live FV address.
- .env.example: default to the v3 int8 model (25 European languages, 464 MiB)
  rather than English-only v2; models to /tank/parakeet/models.
- app.py: warm the recognizer at startup before uvicorn accepts traffic.

The warmup is not an optimisation. ONNX Runtime's CUDA EP compiles and autotunes
lazily on the FIRST DECODE, and on sm_120 that measured 45.7s cold (reproduced at
45.1s on a second container) against ~0.50s warm. A 45s first request is
indistinguishable from a hang and LiteLLM's default timeout abandons it long
before it returns. Decoding 1s of silence at load moves the cost inside the
healthcheck's 300s start_period; first real request after restart is now 0.65s.

Verification, because "provider=cuda" in the log is only an echo of the env var:
ORT falls back to CPU silently and still returns correct text, so the service
being up and the transcript being right establishes nothing. The discriminator is
a process on GPU 3 (922 MiB), confirmed. Controls both directions — a known TTS
sentence transcribes near-exactly (positive), 3s of digital silence returns
empty (null). Warm throughput 0.50s median on an 8.52s clip, n=5, spread
0.47-0.65s, single-stream, one clip: a smoke measurement with its harness
stated, not a benchmark.

Gateway aliases `ext-stt` (engine-neutral, mirrors ext-tts) and `whisper-1`
(OpenAI-compatible drop-in) registered via POST /model/new, i.e. LiteLLM's
Postgres store where the ext-tts family already lives — no gateway restart, and
config.yaml is consequently not a complete picture of what the gateway serves.
Both verified end to end.

The aliases use a raw IP deliberately: ana-docker resolves no .internal names at
all (resolv.conf points at 1.1.1.1), and LiteLLM only reaches irv-ml1 through a
hand-pinned extra_hosts entry. A second hosts entry would mean recreating the
container and bouncing the gateway for every consumer.

Also records the svos_miranda plugin validation pass and its structural findings,
and notes that the irv-ml1 parakeet is still running — there are two now, and
retiring the old one is the operator's call.
This commit is contained in:
vh
2026-09-15 01:41:41 -07:00
parent c6b6435c52
commit b9b14b5baf
8 changed files with 366 additions and 93 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+28 -6
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-09-15 ~01:15 PT (FV cross-site routing FIXED; mesh membership RETIRED for both fv-ml1 and nh3-dev — six nodes remain, each with a job; fv-ml1 carries a break-glass rejoin instead. Next session: drain svos-dev, then stand up an STT.)_
_Last updated: 2026-09-15 ~01:45 PT (Parakeet STT LIVE on fv-ml1 GPU 3 + LiteLLM `ext-stt`/`whisper-1`; svos_miranda plugin validated and its blocker fixed by svos-dev — enabling is the operator's call.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -117,11 +117,25 @@ no longer deployed sidecars here. See Recent decisions.)
_As of 2026-09-15 ~01:15 PT._
### ⭐ NEXT SESSION'S NAMED WORK (operator, at snapshot time)
1. **Drain svos-dev** — one althing message unread since 00:52 PT, pinged five times,
never read. `/althing:inbox` first.
2. **Stand up an STT** (speech-to-text) service. Nothing started: no placement decided,
no model chosen, no stack authored. Greenfield.
### ⭐ BOTH NAMED JOBS CLOSED (2026-09-15 ~01:45 PT)
1. **svos-dev drained** — four messages, thread closed from their end. Ran
`hermes plugins validate/doctor/compat` on `svos_miranda`; found and A/B-proved a
load blocker (absolute intra-package imports), which svos-dev fixed at `c964e64`.
⚠ **Enabling is still pending and is the OPERATOR'S call** — see the decision row below.
2. **STT stood up** — Parakeet on fv-ml1 GPU 3, `ext-stt` / `whisper-1` in LiteLLM.
Not greenfield after all: `stacks/parakeet/` already existed (sherpa-onnx + our own
FastAPI wrapper, previously on irv-ml1) and was retargeted rather than rewritten.
### ⏳ OPEN — svos_miranda enable (needs the operator)
Everything technical is done. Enabling requires editing the operator's live
`~/.hermes/config.yaml` (this session was guardrailed out of that write, then
unblocked — but the go/no-go is his) and svos-dev restarts `:8770` after, which takes
the SVOS board down until the roster verifies. Sequence when he says go: install to
`~/.hermes/plugins/svos_miranda`; add `plugins.enabled` + `entries.svos_miranda.settings`
(dispatch key from `secret get nh3-dev/svos/bifrost-dispatch-key`, `repo_read_enabled: true`);
set `platform_toolsets["api_server"] = ["svos_miranda"]` and `agent.disabled_toolsets`;
**re-derive that list after registration** from `GET http://127.0.0.1:8765/v1/toolsets`
(28 rows now, `svos_miranda` joins once it registers); hand off to svos-dev for the restart.
### Fleet networking — closed out this session
- **FV cross-site routing FIXED.** fv-ml1 reaches NH3/ESH/ANA/IRV/internet via outbound
@@ -161,6 +175,14 @@ hardened for ha-dev (`d1769ed` ff); `kb` KB-search tool (`68fa80f`).
## Recent decisions
- `[2026-09-15]` **Parakeet STT live on fv-ml1 GPU 3, behind LiteLLM `ext-stt` / `whisper-1`.** Retargeted the existing `stacks/parakeet/` (sherpa-onnx + our own FastAPI wrapper) from irv-ml1; v3 int8, 25 languages. ⚠ **ORT's CUDA EP compiles kernels lazily and the first decode on sm_120 took 45.7 s** — every later call ~0.5 s; a startup warmup in `app.py` now absorbs it, so the first real request is 0.65 s instead of a 45 s hang that no client would wait through. GPU use was **verified by a process on GPU 3 (922 MiB), not by the `provider=cuda` log line**, because ORT falls back to CPU silently and still returns correct text. Silence → `""` (null control), known sentence → near-exact (positive control). → `persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md`
- `[2026-09-15]` **`svos_miranda` Hermes plugin validated; found its load blocker.** Absolute intra-package imports (`from hermes_plugin.x`) could not resolve at the documented install name — fixed by svos-dev at `c964e64`. ⚠ **`hermes plugins validate` and `doctor` can NEVER pass this plugin**, by construction: validate's probe stub is config-blind AND returns `None` from `register_tool` (which the plugin's guard reads as a collision), and doctor runs under a temp `HERMES_HOME` with no config. ⚠ `doctor` exits **0** on ERROR (use `--ci`); `compat` reads a **nonexistent path as a pass**. Roster verified 8/7 by a probe supplying real settings. → `persistent-memory.d/2026-09-15-svos-miranda-plugin-validation.md`
- `[2026-09-15]` ⚠ **ana-docker resolves NO `.internal` names** — its `/etc/resolv.conf` is `1.1.1.1`/`1.0.0.1`, not the fleet AdGuard. LiteLLM only reaches `irv-ml1.nh3.internal` because of a hand-pinned `extra_hosts` in its compose. New gateway aliases therefore use **raw IPs**; adding a hosts entry would mean recreating the container and bouncing the gateway for every consumer. Fleet-wide DNS fix is unowned.
- `[2026-09-15]` ⚠ **Two Homepage ASR cards point at the dead wg0 lifeline `10.100.79.3`** (retired at the headscale cutover): the old irv-ml1 `parakeet` :8765 and `Speaches ASR` :8204. The irv-ml1 parakeet is **genuinely still running** (healthz 200 on `100.64.0.6` and `10.6.110.50`) — only its href is stale. So the fleet now has **two** Parakeets. Retiring the irv-ml1 one is the operator's call; not torn down.
- `[2026-09-15]` **Mesh membership retired for fv-ml1 and nh3-dev — six nodes left, each with a job.** fv-ml1 gets break-glass rejoin instead of standing membership; nh3-dev's retirement also removed the nh3-scale masquerade exception it had required. Exactly one live reusable pre-auth key remains fleet-wide. → `persistent-memory.d/2026-09-15-fv-mesh-watchdog.md`
- `[2026-09-15]` **FV cross-site routing fixed — one OPNsense outbound-NAT rule had been scoped to Anaheim only.** fv-ml1 now reaches NH3/ESH/IRV/ANA/mesh/internet; four rules, all `src=10.251.50.0/24`. The diagnostic signature is the valuable part: every layer looks correct and the discriminator is that *every other site pair works*. → `persistent-memory.d/2026-09-15-fv-cross-site-snat.md`