vh
91bda3c480
The box is physically at Fountain Valley, renamed, renumbered onto 10.251/16, and serving inference again. This lands the repo half of that. Host: hostname ana-ml2 -> fv-ml1, pinned to 10.251.50.54 by a dnsmasq reservation so the address the runbook, DNS and LiteLLM all assume is the address it actually has. Its headscale node is renamed too. The sweep ran from scripts/fv-ml1-rename-sweep.sh, whose allowlist is the reason this diff touches current-state files and not the record. Dated persistent-memory entries, archival-memory and incident notes still say ana-ml2 in 31 and 62 places respectively, because that is what the box was when those things happened. Rewriting them would make the history lie. LiteLLM was the load-bearing piece and needed more than the api_base sed the runbook describes. Twenty api_base entries repointed, but a grep-and-verify pass also caught a LIVE pass_through_endpoints target for the scalar-judge reward route still on the old address -- an api_base-only substitution would have left it dead. Four prose references describing current state were repointed as well; one historical note recording where a hand-test was run is deliberately left pointing at 10.250.50.54. Two facts in the server tables were wrong and are corrected here. The site is Fountain Valley, not Anaheim. And the box has FOUR RTX PRO 6000 Blackwell Max-Q, not two -- verified by nvidia-smi -L and independently by PCI enumeration of four GB202GL devices. That is 391 GB of VRAM rather than 196, which changes what fits on it. DNS: fv-ml1, fv-ml1-bmc and fv-gw added under the fv site via the piggyback approach, scriberr re-homed, and the ana-ml2 records removed. Applied to all three resolvers. The BMC record carries a warning that its 802.1q VLAN tag must stay disabled -- it shipped tagging VLAN 250 into an untagged port, which made it invisible to every network-side diagnostic and is the reason it appeared dead through several cable changes. Verified end to end: summarizer and sec both answer through the Anaheim gateway across the mesh to FV seats on different ports.
fv-ml1
Primary AI inference host for PFI.
Network
- LAN IP: 10.251.50.54 (in-band, OS-side)
- BMC (OOB): 10.251.250.50 — Supermicro IPMI web UI at https://10.251.250.50 (homepage card: PFI-ANA-ML2 BMC)
- SSH: standard port 22 on 10.251.50.54
Hardware
- Chassis: Supermicro mid-range inferencing server (bare metal, NOT Dell / not the same box as sf-r630 / sfsrv-ana)
- CPU: AMD EPYC 9254 24-core (96 threads)
- RAM: 566 GB
- GPUs: 2x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB VRAM each, cc 12.0 / sm_120, GPU 0 and GPU 1) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores and doubles VRAM.
- Storage: ZFS
zroot(434 GB root) +tankpool (raidz2, 8× NVMe, 8.6 TB at/tank) — drive inventory below - OS: Debian 13 (trixie), kernel 6.12.x
- Docker: 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2
NVMe drive inventory (tank, raidz2-0) — read 2026-09-09 via nvme-cli
All eight are Dell Express Flash PM1725b 1.6 TB SFF (Samsung OEM), PCIe 3.0 x4 behind
a Broadcom PEX switch. Two provenance batches: the S5CU… six (fw 1.2.2) and the
S47V… pair (fw 1.2.0 / 1.2.1) with thousands of prior-life power cycles.
| dev | PCI | serial | fw | pwr-on h | pwr cycles | unsafe shut. | media err | used |
|---|---|---|---|---|---|---|---|---|
| nvme0 | 46:00.0 | S5CUNEUMB05672 | 1.2.2 | 33856 | 196 | 175 | 0 | 0% |
| nvme1 | — | S5CUNEUMB05671 | 1.2.2 | 33856 | 199 | 178 | 0 | 0% |
| nvme2 | — | S5CUNEUMB05694 | 1.2.2 | 15688 | 90 | 75 | 0 | 0% |
| nvme3 | — | S5CUNEUMB05667 | 1.2.2 | 33857 | 197 | 176 | 0 | 0% |
| nvme4 | — | S5CUNEUMB05674 | 1.2.2 | 33856 | 198 | 177 | 0 | 0% |
| nvme5 | c6:00.0 | S47VNY0K600270 | 1.2.1 | 18823 | 5357 | 5342 | 0 | 1% |
| nvme6 | — | S5CUNEUMB05697 | 1.2.2 | 15570 | 88 | 73 | 0 | 0% |
| nvme7 | 07:00.0 (slot 0-5) | S47VNY0K600221 | 1.2.0 | 19525 | 3093 | 3083 | 2084 | 2% |
⚠ nvme7 was ABSENT from every boot 2026-04-23 → 2026-09-05 (kernel enumerated 7
NVMes per boot; PCIe downstream port 02:04.0 had nothing on bus 07). It reappeared at
the 09-05 14:26 cold boot, the pool resilvered 638 GB, and 2 CKSUM errors landed on it
at import. While it was missing tank was DEGRADED, and Debian's zfsutils-linux cron
(/usr/lib/zfs-linux/{scrub,trim}) only touches pools whose health is ONLINE, so tank
got no scrub and no trim from 04-12 to 09-06. ZED's ZED_EMAIL_ADDR=root has no
MTA behind it, so the 4½-month degradation alerted nobody. media_errors=2084 on
nvme7 is a lifetime counter.
Settled by the 2026-09-09 scrub (00:29–02:02 PT, scrub repaired 0B in 01:32:44 with 0 errors, then zpool clear tank → CKSUM 2 → 0): media_errors read 2084
before and 2084 after a full 6.84 TiB verify, so the counter is prior-life
history, not an active fault, and the 2 CKSUM were the stale-block artefact of the
09-05 late resilver. nvme7 stays in service; watch the counter at every visit and
replace on growth (zpool replace tank nvme7n1 <new>; any PM1725b 1.6 TB or
larger). Slot 0-5 itself deserves a reseat / cable check at the next hands-on
visit — a bay that dropped a drive for 4½ months is the likelier fault than the
drive. Playbook: playbooks/fv-ml1-pool-health.yaml (idempotent; rerunning is a
no-op). ⚠ Nothing alerts on this — see the open follow-up in
persistent-memory.d/2026-09-09-fv-ml1-pool-actions-done.md.
Key paths
| Path | Purpose |
|---|---|
/opt/docker/compose/<stack>/ |
Compose files |
/opt/docker/conf/<stack>/ |
Config bind mounts |
/tank/aimodels/huggingface/ |
HF cache (267 GB, pre-downloaded models) |
/tank/aimodels/llm/ |
Legacy GGUF models (790 GB, referenced by llama-swap as /models/) |
/var/lib/docker/ |
Docker data (on zroot) |
Running stacks
Live inventory as of 2026-07-22. Each model is its own compose stack now
(container vllm-<x> / llama-<x>); the vllm stack proper is just the
embed/rerank/reward trio. GPUs are pinned per container via
deploy.resources.reservations.devices[].device_ids.
GPU 0 — heavy RP / reasoning seats (~88/98 GB, hot serving path):
| Container | Port | Served model | Quant | Ctx |
|---|---|---|---|---|
vllm-gen (project gen-seat) |
8015 | qwen3.8-27b-uncensored — the "gen" hero seat (Qwen3.8-27B Heretic-abliterated, in-house NVFP4 W4A16 + grafted MTP) |
NVFP4 W4A16 (compressed-tensors) | 262k |
vllm-charrp-reasoning-nvfp4 |
8018 | char-rp-reasoning (R36 reasoning RP) |
NVFP4 (modelopt) | 256k |
GPU 1 — light / eval / retrieval + char-RP GGUF (~91/98 GB, on-demand):
| Container | Port | Served model | Quant | Ctx |
|---|---|---|---|---|
vllm-granite |
8004 | granite-4.1-8b — fleet summarizer/classifier |
FP8 (compressed-tensors) | 131k |
llama-charrp |
8016 | Magidonia-24B-v4.3 Q6_K — char-RP (llama.cpp) |
GGUF Q6_K | — |
vllm-selene |
RETIRED 2026-08-23 — lost a head-to-head against gen on its own judge task (see stacks/selene/README.md); seat downed to reclaim 17.2 GiB on GPU 1. selene-1-mini-8b now 404s by design; use chat-judge. |
— | — | |
vllm-reward |
8003 | Skywork-Reward-V2-Llama-3.1-8B-AWQ — reward classifier |
AWQ | 16k |
vllm-embed |
8001 | Qwen3-Embedding-0.6B |
— | 8k |
vllm-rerank |
8002 | Qwen3-Reranker-0.6B |
— | 8k |
Infra / non-GPU:
| Container | Port | Notes |
|---|---|---|
dockge |
5001 | Docker stack management UI |
dozzle-agent |
7007 | Log agent → Dozzle hub on ana-docker |
beszel-agent |
45876 | Metrics agent → Beszel hub on ana-docker |
Both cards run near-full (~7–10 GB headroom each) — adding a seat means placing it on the card with room or evicting a dormant one first.
Dormant (compose present on disk, containers stopped) — rollback / audition
seats, safe to leave: mistral-medium-3.5, mistral-small-4(-heretic),
ms32-24b-angel, qwen3.5-122b, qwopus3.5-122b, qwen35-vl, qwen36-vl,
qwen36-27b-aeon, qwen-image-bench, vibevoice, comfyui, kokoro,
parakeet, vllm-qwen3.
Retired:
llama-swap(former GGUF multiplexer on :9292) — replaced by dedicated per-model seats (e.g.llama-charrp); no longer running.infinity— replaced by thevllmstack (originallyvllm-qwen3, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping atransformersbuild that knew Qwen3.LibreChat (+ rag_api, vectordb, mongodb, meilisearch),searxng— removed from this host (searxng now on ana-docker fleet-wide).
Refresh state
scripts/refresh-server-info.sh fv-ml1
Latest snapshot: system-details.txt (regenerate as needed).
GPU allocation policy
Every seat is explicitly pinned via device_ids (no unpinned containers), and
both cards run ~90% full:
- GPU 0: the two heavy NVFP4 seats —
vllm-gen(gen) andvllm-charrp-reasoning-nvfp4. The live serving path (near-100% util under load), ~42 + 45 GB. - GPU 1: everything else — reward, embed, rerank, and the Magidonia char-RP GGUF seat. Bursty/on-demand, idle between calls, ~91 GB resident.
Pin with deploy.resources.reservations.devices[].device_ids: ["<id>"] in
compose. Each service caps its share with --gpu-memory-utilization; with both
cards near-full, placing a new seat means freeing room (evict a dormant one) or
trimming a neighbour's utilization first.