vh
caa04801f3
Placed on GPU 3 first because it was the empty card. That was the wrong read:
the seat is ~800 MiB, under 1% of a 96 GB card, so the question was never "where
does it fit" but "whose headroom is cheapest to spend".
vLLM sizes its KV cache as a fraction of TOTAL VRAM, not free VRAM. A resident
tenant on an otherwise-clean card therefore does not cost its own megabytes — it
costs the profiling margin of whatever full-size seat lands there later, and
flash-next needs 93 GiB of 96. A 96 GB card at 2 MiB can still take that; the
same card at 922 MiB is one where the next big seat needs its utilization
hand-trimmed, which this repo's flash-next history shows is both thin and silent
when it goes wrong.
Committed utilization per card is the number that governs, not free bytes:
GPU 0 0.40 + 0.48 = 0.88 ~13 GB spare <- moved here
GPU 1 0.52+0.24+0.10+0.055+0.03+0.03 = 0.975 ~4.3 GB
GPU 2 0.96 ~1.8 GB
GPU 3 - kept empty as reserve
GPU 3 is back to 2 MiB / 97,247 MiB free and is now documented as a deliberate
reserve rather than a spare.
Post-move n=5 on the same clip: 0.68 / 0.54 / 0.54 / 0.52 / 0.53 s, median 0.54 s
against 0.50 s on GPU 3. The spreads overlap at this sample size and no difference
is claimed; the GPU 3 figure was taken on an idle card and is now noted as a best
case, since the seat shares GPU 0 with the hot serving path. Silence control and
the gateway round-trip both re-verified after the move.
Also records both Parakeet endpoints (FV v3 on Blackwell, IRV v2 on a 3090) and
the four confounds that make them not an A/B pair, sent to tts-dev for the bench.
fv-ml1
Primary AI inference host for PFI.
Network
- LAN IP: 10.251.50.54 (in-band, OS-side)
- BMC (OOB): 10.251.250.50 — Supermicro IPMI web UI at https://10.251.250.50 (homepage card: PFI-ANA-ML2 BMC)
- SSH: standard port 22 on 10.251.50.54
Hardware
- Chassis: Supermicro mid-range inferencing server (bare metal, NOT Dell / not the same box as sf-r630 / sfsrv-ana)
- CPU: AMD EPYC 9254 24-core (96 threads)
- RAM: 566 GB
- GPUs: 2x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB VRAM each, cc 12.0 / sm_120, GPU 0 and GPU 1) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores and doubles VRAM.
- Storage: ZFS
zroot(434 GB root) +tankpool (raidz2, 8× NVMe, 8.6 TB at/tank) — drive inventory below - OS: Debian 13 (trixie), kernel 6.12.x
- Docker: 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2
NVMe drive inventory (tank, raidz2-0) — read 2026-09-09 via nvme-cli
All eight are Dell Express Flash PM1725b 1.6 TB SFF (Samsung OEM), PCIe 3.0 x4 behind
a Broadcom PEX switch. Two provenance batches: the S5CU… six (fw 1.2.2) and the
S47V… pair (fw 1.2.0 / 1.2.1) with thousands of prior-life power cycles.
| dev | PCI | serial | fw | pwr-on h | pwr cycles | unsafe shut. | media err | used |
|---|---|---|---|---|---|---|---|---|
| nvme0 | 46:00.0 | S5CUNEUMB05672 | 1.2.2 | 33856 | 196 | 175 | 0 | 0% |
| nvme1 | — | S5CUNEUMB05671 | 1.2.2 | 33856 | 199 | 178 | 0 | 0% |
| nvme2 | — | S5CUNEUMB05694 | 1.2.2 | 15688 | 90 | 75 | 0 | 0% |
| nvme3 | — | S5CUNEUMB05667 | 1.2.2 | 33857 | 197 | 176 | 0 | 0% |
| nvme4 | — | S5CUNEUMB05674 | 1.2.2 | 33856 | 198 | 177 | 0 | 0% |
| nvme5 | c6:00.0 | S47VNY0K600270 | 1.2.1 | 18823 | 5357 | 5342 | 0 | 1% |
| nvme6 | — | S5CUNEUMB05697 | 1.2.2 | 15570 | 88 | 73 | 0 | 0% |
| nvme7 | 07:00.0 (slot 0-5) | S47VNY0K600221 | 1.2.0 | 19525 | 3093 | 3083 | 2084 | 2% |
⚠ nvme7 was ABSENT from every boot 2026-04-23 → 2026-09-05 (kernel enumerated 7
NVMes per boot; PCIe downstream port 02:04.0 had nothing on bus 07). It reappeared at
the 09-05 14:26 cold boot, the pool resilvered 638 GB, and 2 CKSUM errors landed on it
at import. While it was missing tank was DEGRADED, and Debian's zfsutils-linux cron
(/usr/lib/zfs-linux/{scrub,trim}) only touches pools whose health is ONLINE, so tank
got no scrub and no trim from 04-12 to 09-06. ZED's ZED_EMAIL_ADDR=root has no
MTA behind it, so the 4½-month degradation alerted nobody. media_errors=2084 on
nvme7 is a lifetime counter.
Settled by the 2026-09-09 scrub (00:29–02:02 PT, scrub repaired 0B in 01:32:44 with 0 errors, then zpool clear tank → CKSUM 2 → 0): media_errors read 2084
before and 2084 after a full 6.84 TiB verify, so the counter is prior-life
history, not an active fault, and the 2 CKSUM were the stale-block artefact of the
09-05 late resilver. nvme7 stays in service; watch the counter at every visit and
replace on growth (zpool replace tank nvme7n1 <new>; any PM1725b 1.6 TB or
larger). Slot 0-5 itself deserves a reseat / cable check at the next hands-on
visit — a bay that dropped a drive for 4½ months is the likelier fault than the
drive. Playbook: playbooks/fv-ml1-pool-health.yaml (idempotent; rerunning is a
no-op). ⚠ Nothing alerts on this — see the open follow-up in
persistent-memory.d/2026-09-09-fv-ml1-pool-actions-done.md.
Key paths
| Path | Purpose |
|---|---|
/opt/docker/compose/<stack>/ |
Compose files |
/opt/docker/conf/<stack>/ |
Config bind mounts |
/tank/aimodels/huggingface/ |
HF cache (267 GB, pre-downloaded models) |
/tank/aimodels/llm/ |
Legacy GGUF models (790 GB, referenced by llama-swap as /models/) |
/var/lib/docker/ |
Docker data (on zroot) |
Running stacks
Live inventory as of 2026-07-22. Each model is its own compose stack now
(container vllm-<x> / llama-<x>); the vllm stack proper is just the
embed/rerank/reward trio. GPUs are pinned per container via
deploy.resources.reservations.devices[].device_ids.
GPU 0 — heavy RP / reasoning seats (~88/98 GB, hot serving path):
| Container | Port | Served model | Quant | Ctx |
|---|---|---|---|---|
vllm-gen (project gen-seat) |
8015 | qwen3.8-27b-uncensored — the "gen" hero seat (Qwen3.8-27B Heretic-abliterated, in-house NVFP4 W4A16 + grafted MTP) |
NVFP4 W4A16 (compressed-tensors) | 262k |
vllm-charrp-reasoning-nvfp4 |
8018 | char-rp-reasoning (R36 reasoning RP) |
NVFP4 (modelopt) | 256k |
GPU 1 — light / eval / retrieval + char-RP GGUF (~91/98 GB, on-demand):
| Container | Port | Served model | Quant | Ctx |
|---|---|---|---|---|
vllm-granite |
8004 | granite-4.1-8b — fleet summarizer/classifier |
FP8 (compressed-tensors) | 131k |
llama-charrp |
8016 | Magidonia-24B-v4.3 Q6_K — char-RP (llama.cpp) |
GGUF Q6_K | — |
vllm-selene |
RETIRED 2026-08-23 — lost a head-to-head against gen on its own judge task (see stacks/selene/README.md); seat downed to reclaim 17.2 GiB on GPU 1. selene-1-mini-8b now 404s by design; use chat-judge. |
— | — | |
vllm-reward |
8003 | Skywork-Reward-V2-Llama-3.1-8B-AWQ — reward classifier |
AWQ | 16k |
vllm-embed |
8001 | Qwen3-Embedding-0.6B |
— | 8k |
vllm-rerank |
8002 | Qwen3-Reranker-0.6B |
— | 8k |
Infra / non-GPU:
| Container | Port | Notes |
|---|---|---|
dockge |
5001 | Docker stack management UI |
dozzle-agent |
7007 | Log agent → Dozzle hub on ana-docker |
beszel-agent |
45876 | Metrics agent → Beszel hub on ana-docker |
Both cards run near-full (~7–10 GB headroom each) — adding a seat means placing it on the card with room or evicting a dormant one first.
Dormant (compose present on disk, containers stopped) — rollback / audition
seats, safe to leave: mistral-medium-3.5, mistral-small-4(-heretic),
ms32-24b-angel, qwen3.5-122b, qwopus3.5-122b, qwen35-vl, qwen36-vl,
qwen36-27b-aeon, qwen-image-bench, vibevoice, comfyui, kokoro,
vllm-qwen3.
Also on GPU 0 (non-vLLM):
| Container | Port | Serves | Notes |
|---|---|---|---|
parakeet |
8300 | Parakeet-TDT 0.6B v3 int8 (25 languages) | ASR via sherpa-onnx, LiteLLM ext-stt / whisper-1. Relocated from irv-ml1 2026-09-15. ~800 MiB. stacks/parakeet/. |
⚠ GPU 3 is deliberately kept EMPTY (2 MiB). It is the only card that can still
take a full-size seat — flash-next needs 93 GiB of 96 — and vLLM sizes its KV
cache against total VRAM rather than free VRAM, so even a sub-1 GB tenant there
eats into a future big seat's profiling margin. Small seats go on GPU 0, which has
the most uncommitted headroom (its seats commit util 0.88; GPU 1 is at 0.975 and
GPU 2 at 0.96).
Retired:
llama-swap(former GGUF multiplexer on :9292) — replaced by dedicated per-model seats (e.g.llama-charrp); no longer running.infinity— replaced by thevllmstack (originallyvllm-qwen3, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping atransformersbuild that knew Qwen3.LibreChat (+ rag_api, vectordb, mongodb, meilisearch),searxng— removed from this host (searxng now on ana-docker fleet-wide).
Refresh state
scripts/refresh-server-info.sh fv-ml1
Latest snapshot: system-details.txt (regenerate as needed).
GPU allocation policy
Every seat is explicitly pinned via device_ids (no unpinned containers), and
both cards run ~90% full:
- GPU 0: the two heavy NVFP4 seats —
vllm-gen(gen) andvllm-charrp-reasoning-nvfp4. The live serving path (near-100% util under load), ~42 + 45 GB. - GPU 1: everything else — reward, embed, rerank, and the Magidonia char-RP GGUF seat. Bursty/on-demand, idle between calls, ~91 GB resident.
Pin with deploy.resources.reservations.devices[].device_ids: ["<id>"] in
compose. Each service caps its share with --gpu-memory-utilization; with both
cards near-full, placing a new seat means freeing room (evict a dormant one) or
trimming a neighbour's utilization first.