vh
b9b14b5baf
Retargets the existing sherpa-onnx stack from irv-ml1 to fv-ml1's utility card and puts it behind the gateway. GPU 3 was the only card with room: 0/1/2 carry the vLLM seats at 84-95.5 GB of 96. Changes: - compose: pin GPU via `device_ids: ["3"]` (the dead on-host stub used `count: all`, which would have handed a 0.6B ASR seat all four cards); join traefik-net; port 8300; homepage href to the live FV address. - .env.example: default to the v3 int8 model (25 European languages, 464 MiB) rather than English-only v2; models to /tank/parakeet/models. - app.py: warm the recognizer at startup before uvicorn accepts traffic. The warmup is not an optimisation. ONNX Runtime's CUDA EP compiles and autotunes lazily on the FIRST DECODE, and on sm_120 that measured 45.7s cold (reproduced at 45.1s on a second container) against ~0.50s warm. A 45s first request is indistinguishable from a hang and LiteLLM's default timeout abandons it long before it returns. Decoding 1s of silence at load moves the cost inside the healthcheck's 300s start_period; first real request after restart is now 0.65s. Verification, because "provider=cuda" in the log is only an echo of the env var: ORT falls back to CPU silently and still returns correct text, so the service being up and the transcript being right establishes nothing. The discriminator is a process on GPU 3 (922 MiB), confirmed. Controls both directions — a known TTS sentence transcribes near-exactly (positive), 3s of digital silence returns empty (null). Warm throughput 0.50s median on an 8.52s clip, n=5, spread 0.47-0.65s, single-stream, one clip: a smoke measurement with its harness stated, not a benchmark. Gateway aliases `ext-stt` (engine-neutral, mirrors ext-tts) and `whisper-1` (OpenAI-compatible drop-in) registered via POST /model/new, i.e. LiteLLM's Postgres store where the ext-tts family already lives — no gateway restart, and config.yaml is consequently not a complete picture of what the gateway serves. Both verified end to end. The aliases use a raw IP deliberately: ana-docker resolves no .internal names at all (resolv.conf points at 1.1.1.1), and LiteLLM only reaches irv-ml1 through a hand-pinned extra_hosts entry. A second hosts entry would mean recreating the container and bouncing the gateway for every consumer. Also records the svos_miranda plugin validation pass and its structural findings, and notes that the irv-ml1 parakeet is still running — there are two now, and retiring the old one is the operator's call.
fv-ml1
Primary AI inference host for PFI.
Network
- LAN IP: 10.251.50.54 (in-band, OS-side)
- BMC (OOB): 10.251.250.50 — Supermicro IPMI web UI at https://10.251.250.50 (homepage card: PFI-ANA-ML2 BMC)
- SSH: standard port 22 on 10.251.50.54
Hardware
- Chassis: Supermicro mid-range inferencing server (bare metal, NOT Dell / not the same box as sf-r630 / sfsrv-ana)
- CPU: AMD EPYC 9254 24-core (96 threads)
- RAM: 566 GB
- GPUs: 2x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB VRAM each, cc 12.0 / sm_120, GPU 0 and GPU 1) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores and doubles VRAM.
- Storage: ZFS
zroot(434 GB root) +tankpool (raidz2, 8× NVMe, 8.6 TB at/tank) — drive inventory below - OS: Debian 13 (trixie), kernel 6.12.x
- Docker: 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2
NVMe drive inventory (tank, raidz2-0) — read 2026-09-09 via nvme-cli
All eight are Dell Express Flash PM1725b 1.6 TB SFF (Samsung OEM), PCIe 3.0 x4 behind
a Broadcom PEX switch. Two provenance batches: the S5CU… six (fw 1.2.2) and the
S47V… pair (fw 1.2.0 / 1.2.1) with thousands of prior-life power cycles.
| dev | PCI | serial | fw | pwr-on h | pwr cycles | unsafe shut. | media err | used |
|---|---|---|---|---|---|---|---|---|
| nvme0 | 46:00.0 | S5CUNEUMB05672 | 1.2.2 | 33856 | 196 | 175 | 0 | 0% |
| nvme1 | — | S5CUNEUMB05671 | 1.2.2 | 33856 | 199 | 178 | 0 | 0% |
| nvme2 | — | S5CUNEUMB05694 | 1.2.2 | 15688 | 90 | 75 | 0 | 0% |
| nvme3 | — | S5CUNEUMB05667 | 1.2.2 | 33857 | 197 | 176 | 0 | 0% |
| nvme4 | — | S5CUNEUMB05674 | 1.2.2 | 33856 | 198 | 177 | 0 | 0% |
| nvme5 | c6:00.0 | S47VNY0K600270 | 1.2.1 | 18823 | 5357 | 5342 | 0 | 1% |
| nvme6 | — | S5CUNEUMB05697 | 1.2.2 | 15570 | 88 | 73 | 0 | 0% |
| nvme7 | 07:00.0 (slot 0-5) | S47VNY0K600221 | 1.2.0 | 19525 | 3093 | 3083 | 2084 | 2% |
⚠ nvme7 was ABSENT from every boot 2026-04-23 → 2026-09-05 (kernel enumerated 7
NVMes per boot; PCIe downstream port 02:04.0 had nothing on bus 07). It reappeared at
the 09-05 14:26 cold boot, the pool resilvered 638 GB, and 2 CKSUM errors landed on it
at import. While it was missing tank was DEGRADED, and Debian's zfsutils-linux cron
(/usr/lib/zfs-linux/{scrub,trim}) only touches pools whose health is ONLINE, so tank
got no scrub and no trim from 04-12 to 09-06. ZED's ZED_EMAIL_ADDR=root has no
MTA behind it, so the 4½-month degradation alerted nobody. media_errors=2084 on
nvme7 is a lifetime counter.
Settled by the 2026-09-09 scrub (00:29–02:02 PT, scrub repaired 0B in 01:32:44 with 0 errors, then zpool clear tank → CKSUM 2 → 0): media_errors read 2084
before and 2084 after a full 6.84 TiB verify, so the counter is prior-life
history, not an active fault, and the 2 CKSUM were the stale-block artefact of the
09-05 late resilver. nvme7 stays in service; watch the counter at every visit and
replace on growth (zpool replace tank nvme7n1 <new>; any PM1725b 1.6 TB or
larger). Slot 0-5 itself deserves a reseat / cable check at the next hands-on
visit — a bay that dropped a drive for 4½ months is the likelier fault than the
drive. Playbook: playbooks/fv-ml1-pool-health.yaml (idempotent; rerunning is a
no-op). ⚠ Nothing alerts on this — see the open follow-up in
persistent-memory.d/2026-09-09-fv-ml1-pool-actions-done.md.
Key paths
| Path | Purpose |
|---|---|
/opt/docker/compose/<stack>/ |
Compose files |
/opt/docker/conf/<stack>/ |
Config bind mounts |
/tank/aimodels/huggingface/ |
HF cache (267 GB, pre-downloaded models) |
/tank/aimodels/llm/ |
Legacy GGUF models (790 GB, referenced by llama-swap as /models/) |
/var/lib/docker/ |
Docker data (on zroot) |
Running stacks
Live inventory as of 2026-07-22. Each model is its own compose stack now
(container vllm-<x> / llama-<x>); the vllm stack proper is just the
embed/rerank/reward trio. GPUs are pinned per container via
deploy.resources.reservations.devices[].device_ids.
GPU 0 — heavy RP / reasoning seats (~88/98 GB, hot serving path):
| Container | Port | Served model | Quant | Ctx |
|---|---|---|---|---|
vllm-gen (project gen-seat) |
8015 | qwen3.8-27b-uncensored — the "gen" hero seat (Qwen3.8-27B Heretic-abliterated, in-house NVFP4 W4A16 + grafted MTP) |
NVFP4 W4A16 (compressed-tensors) | 262k |
vllm-charrp-reasoning-nvfp4 |
8018 | char-rp-reasoning (R36 reasoning RP) |
NVFP4 (modelopt) | 256k |
GPU 1 — light / eval / retrieval + char-RP GGUF (~91/98 GB, on-demand):
| Container | Port | Served model | Quant | Ctx |
|---|---|---|---|---|
vllm-granite |
8004 | granite-4.1-8b — fleet summarizer/classifier |
FP8 (compressed-tensors) | 131k |
llama-charrp |
8016 | Magidonia-24B-v4.3 Q6_K — char-RP (llama.cpp) |
GGUF Q6_K | — |
vllm-selene |
RETIRED 2026-08-23 — lost a head-to-head against gen on its own judge task (see stacks/selene/README.md); seat downed to reclaim 17.2 GiB on GPU 1. selene-1-mini-8b now 404s by design; use chat-judge. |
— | — | |
vllm-reward |
8003 | Skywork-Reward-V2-Llama-3.1-8B-AWQ — reward classifier |
AWQ | 16k |
vllm-embed |
8001 | Qwen3-Embedding-0.6B |
— | 8k |
vllm-rerank |
8002 | Qwen3-Reranker-0.6B |
— | 8k |
Infra / non-GPU:
| Container | Port | Notes |
|---|---|---|
dockge |
5001 | Docker stack management UI |
dozzle-agent |
7007 | Log agent → Dozzle hub on ana-docker |
beszel-agent |
45876 | Metrics agent → Beszel hub on ana-docker |
Both cards run near-full (~7–10 GB headroom each) — adding a seat means placing it on the card with room or evicting a dormant one first.
Dormant (compose present on disk, containers stopped) — rollback / audition
seats, safe to leave: mistral-medium-3.5, mistral-small-4(-heretic),
ms32-24b-angel, qwen3.5-122b, qwopus3.5-122b, qwen35-vl, qwen36-vl,
qwen36-27b-aeon, qwen-image-bench, vibevoice, comfyui, kokoro,
vllm-qwen3.
GPU 3 — utility card:
| Container | Port | Serves | Notes |
|---|---|---|---|
parakeet |
8300 | Parakeet-TDT 0.6B v3 int8 (25 languages) | ASR via sherpa-onnx, LiteLLM ext-stt / whisper-1. Relocated from irv-ml1 2026-09-15. stacks/parakeet/. |
⚠ The other three cards run 85-98% full, so GPU 3 is where a new small seat goes until something bigger claims it.
Retired:
llama-swap(former GGUF multiplexer on :9292) — replaced by dedicated per-model seats (e.g.llama-charrp); no longer running.infinity— replaced by thevllmstack (originallyvllm-qwen3, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping atransformersbuild that knew Qwen3.LibreChat (+ rag_api, vectordb, mongodb, meilisearch),searxng— removed from this host (searxng now on ana-docker fleet-wide).
Refresh state
scripts/refresh-server-info.sh fv-ml1
Latest snapshot: system-details.txt (regenerate as needed).
GPU allocation policy
Every seat is explicitly pinned via device_ids (no unpinned containers), and
both cards run ~90% full:
- GPU 0: the two heavy NVFP4 seats —
vllm-gen(gen) andvllm-charrp-reasoning-nvfp4. The live serving path (near-100% util under load), ~42 + 45 GB. - GPU 1: everything else — reward, embed, rerank, and the Magidonia char-RP GGUF seat. Bursty/on-demand, idle between calls, ~91 GB resident.
Pin with deploy.resources.reservations.devices[].device_ids: ["<id>"] in
compose. Each service caps its share with --gpu-memory-utilization; with both
cards near-full, placing a new seat means freeing room (evict a dormant one) or
trimming a neighbour's utilization first.