vh
8c8559b8ec
WATCHDOG (done, proven). fv-mesh-watchdog probes two independent anchors every
minute and, after 5 consecutive failures, puts Tailscale back to known-good:
accept-routes off, re-up against headscale with a stored key. It touches
nothing else — a watchdog with a wide remit is a second way to lose the box.
Two anchors that cannot share a failure mode: a plain-internet one and a
mesh-only one. If BOTH fail the site uplink is down, Tailscale cannot fix that,
and it deliberately does nothing — thrashing tailscaled during an ISP outage
turns a wait into an incident. Disable file at /etc/fv-watchdog.disable for
planned work.
Proven by positive control, not assumed: counter incremented 1..4 without
acting, fired the restore at 5 (tailscale up ran, tailscaled restarted), and
reset to 0 once the real anchor returned. fv-ml1 stayed reachable throughout.
This exists because a on fv-ml1 black-holed it
from its own LAN earlier the same day: it accepted 10.251.0.0/16 from the
gateway — its OWN subnet — and routed the local network through the tunnel.
FV CROSS-SITE ROUTING (partial). Two changes landed, the path is still broken:
1. acceptSubnetRoutes 0 -> 1 on the FV gateway's tailscale plugin, via
settings/set + service/reconfigure (the documented apply, not a reboot).
The GATEWAY now has 10.0/16, 10.100/16 and 10.250/16 in its routing table
and reaches NH3 and ESH itself. It could not before.
2. Remote-site MASQUERADE rules on nh3-scale. The existing jump matched only
-s 100.64.0.0/10, so traffic from another site's LAN never entered
MESH-EXIT and kept its original source; an NH3 host then replied via its
own LAN router instead of back through nh3-scale, making the path
asymmetric. The rule is confirmed firing (counter increments on FV
traffic) but does not complete the path.
Still failing: fv-ml1 -> NH3/ESH LAN addresses. Mesh addresses work perfectly
from fv-ml1 (100.64.0.1, 100.64.0.4), Anaheim works over the metro link, and
the FV firewall log shows the outbound passing on tailscale0 with
src=10.251.50.54 and no reply ever returning. The remaining gap is forwarded
FV-LAN traffic specifically, not the gateway's own.
Full regression sweep clean: nh3-dev, ana-docker and esh-docker-vm all reach
all four sites plus the internet.
fv-ml1
Primary AI inference host for PFI.
Network
- LAN IP: 10.251.50.54 (in-band, OS-side)
- BMC (OOB): 10.251.250.50 — Supermicro IPMI web UI at https://10.251.250.50 (homepage card: PFI-ANA-ML2 BMC)
- SSH: standard port 22 on 10.251.50.54
Hardware
- Chassis: Supermicro mid-range inferencing server (bare metal, NOT Dell / not the same box as sf-r630 / sfsrv-ana)
- CPU: AMD EPYC 9254 24-core (96 threads)
- RAM: 566 GB
- GPUs: 2x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB VRAM each, cc 12.0 / sm_120, GPU 0 and GPU 1) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores and doubles VRAM.
- Storage: ZFS
zroot(434 GB root) +tankpool (raidz2, 8× NVMe, 8.6 TB at/tank) — drive inventory below - OS: Debian 13 (trixie), kernel 6.12.x
- Docker: 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2
NVMe drive inventory (tank, raidz2-0) — read 2026-09-09 via nvme-cli
All eight are Dell Express Flash PM1725b 1.6 TB SFF (Samsung OEM), PCIe 3.0 x4 behind
a Broadcom PEX switch. Two provenance batches: the S5CU… six (fw 1.2.2) and the
S47V… pair (fw 1.2.0 / 1.2.1) with thousands of prior-life power cycles.
| dev | PCI | serial | fw | pwr-on h | pwr cycles | unsafe shut. | media err | used |
|---|---|---|---|---|---|---|---|---|
| nvme0 | 46:00.0 | S5CUNEUMB05672 | 1.2.2 | 33856 | 196 | 175 | 0 | 0% |
| nvme1 | — | S5CUNEUMB05671 | 1.2.2 | 33856 | 199 | 178 | 0 | 0% |
| nvme2 | — | S5CUNEUMB05694 | 1.2.2 | 15688 | 90 | 75 | 0 | 0% |
| nvme3 | — | S5CUNEUMB05667 | 1.2.2 | 33857 | 197 | 176 | 0 | 0% |
| nvme4 | — | S5CUNEUMB05674 | 1.2.2 | 33856 | 198 | 177 | 0 | 0% |
| nvme5 | c6:00.0 | S47VNY0K600270 | 1.2.1 | 18823 | 5357 | 5342 | 0 | 1% |
| nvme6 | — | S5CUNEUMB05697 | 1.2.2 | 15570 | 88 | 73 | 0 | 0% |
| nvme7 | 07:00.0 (slot 0-5) | S47VNY0K600221 | 1.2.0 | 19525 | 3093 | 3083 | 2084 | 2% |
⚠ nvme7 was ABSENT from every boot 2026-04-23 → 2026-09-05 (kernel enumerated 7
NVMes per boot; PCIe downstream port 02:04.0 had nothing on bus 07). It reappeared at
the 09-05 14:26 cold boot, the pool resilvered 638 GB, and 2 CKSUM errors landed on it
at import. While it was missing tank was DEGRADED, and Debian's zfsutils-linux cron
(/usr/lib/zfs-linux/{scrub,trim}) only touches pools whose health is ONLINE, so tank
got no scrub and no trim from 04-12 to 09-06. ZED's ZED_EMAIL_ADDR=root has no
MTA behind it, so the 4½-month degradation alerted nobody. media_errors=2084 on
nvme7 is a lifetime counter.
Settled by the 2026-09-09 scrub (00:29–02:02 PT, scrub repaired 0B in 01:32:44 with 0 errors, then zpool clear tank → CKSUM 2 → 0): media_errors read 2084
before and 2084 after a full 6.84 TiB verify, so the counter is prior-life
history, not an active fault, and the 2 CKSUM were the stale-block artefact of the
09-05 late resilver. nvme7 stays in service; watch the counter at every visit and
replace on growth (zpool replace tank nvme7n1 <new>; any PM1725b 1.6 TB or
larger). Slot 0-5 itself deserves a reseat / cable check at the next hands-on
visit — a bay that dropped a drive for 4½ months is the likelier fault than the
drive. Playbook: playbooks/fv-ml1-pool-health.yaml (idempotent; rerunning is a
no-op). ⚠ Nothing alerts on this — see the open follow-up in
persistent-memory.d/2026-09-09-fv-ml1-pool-actions-done.md.
Key paths
| Path | Purpose |
|---|---|
/opt/docker/compose/<stack>/ |
Compose files |
/opt/docker/conf/<stack>/ |
Config bind mounts |
/tank/aimodels/huggingface/ |
HF cache (267 GB, pre-downloaded models) |
/tank/aimodels/llm/ |
Legacy GGUF models (790 GB, referenced by llama-swap as /models/) |
/var/lib/docker/ |
Docker data (on zroot) |
Running stacks
Live inventory as of 2026-07-22. Each model is its own compose stack now
(container vllm-<x> / llama-<x>); the vllm stack proper is just the
embed/rerank/reward trio. GPUs are pinned per container via
deploy.resources.reservations.devices[].device_ids.
GPU 0 — heavy RP / reasoning seats (~88/98 GB, hot serving path):
| Container | Port | Served model | Quant | Ctx |
|---|---|---|---|---|
vllm-gen (project gen-seat) |
8015 | qwen3.8-27b-uncensored — the "gen" hero seat (Qwen3.8-27B Heretic-abliterated, in-house NVFP4 W4A16 + grafted MTP) |
NVFP4 W4A16 (compressed-tensors) | 262k |
vllm-charrp-reasoning-nvfp4 |
8018 | char-rp-reasoning (R36 reasoning RP) |
NVFP4 (modelopt) | 256k |
GPU 1 — light / eval / retrieval + char-RP GGUF (~91/98 GB, on-demand):
| Container | Port | Served model | Quant | Ctx |
|---|---|---|---|---|
vllm-granite |
8004 | granite-4.1-8b — fleet summarizer/classifier |
FP8 (compressed-tensors) | 131k |
llama-charrp |
8016 | Magidonia-24B-v4.3 Q6_K — char-RP (llama.cpp) |
GGUF Q6_K | — |
vllm-selene |
RETIRED 2026-08-23 — lost a head-to-head against gen on its own judge task (see stacks/selene/README.md); seat downed to reclaim 17.2 GiB on GPU 1. selene-1-mini-8b now 404s by design; use chat-judge. |
— | — | |
vllm-reward |
8003 | Skywork-Reward-V2-Llama-3.1-8B-AWQ — reward classifier |
AWQ | 16k |
vllm-embed |
8001 | Qwen3-Embedding-0.6B |
— | 8k |
vllm-rerank |
8002 | Qwen3-Reranker-0.6B |
— | 8k |
Infra / non-GPU:
| Container | Port | Notes |
|---|---|---|
dockge |
5001 | Docker stack management UI |
dozzle-agent |
7007 | Log agent → Dozzle hub on ana-docker |
beszel-agent |
45876 | Metrics agent → Beszel hub on ana-docker |
Both cards run near-full (~7–10 GB headroom each) — adding a seat means placing it on the card with room or evicting a dormant one first.
Dormant (compose present on disk, containers stopped) — rollback / audition
seats, safe to leave: mistral-medium-3.5, mistral-small-4(-heretic),
ms32-24b-angel, qwen3.5-122b, qwopus3.5-122b, qwen35-vl, qwen36-vl,
qwen36-27b-aeon, qwen-image-bench, vibevoice, comfyui, kokoro,
parakeet, vllm-qwen3.
Retired:
llama-swap(former GGUF multiplexer on :9292) — replaced by dedicated per-model seats (e.g.llama-charrp); no longer running.infinity— replaced by thevllmstack (originallyvllm-qwen3, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping atransformersbuild that knew Qwen3.LibreChat (+ rag_api, vectordb, mongodb, meilisearch),searxng— removed from this host (searxng now on ana-docker fleet-wide).
Refresh state
scripts/refresh-server-info.sh fv-ml1
Latest snapshot: system-details.txt (regenerate as needed).
GPU allocation policy
Every seat is explicitly pinned via device_ids (no unpinned containers), and
both cards run ~90% full:
- GPU 0: the two heavy NVFP4 seats —
vllm-gen(gen) andvllm-charrp-reasoning-nvfp4. The live serving path (near-100% util under load), ~42 + 45 GB. - GPU 1: everything else — reward, embed, rerank, and the Magidonia char-RP GGUF seat. Bursty/on-demand, idle between calls, ~91 GB resident.
Pin with deploy.resources.reservations.devices[].device_ids: ["<id>"] in
compose. Each service caps its share with --gpu-memory-utilization; with both
cards near-full, placing a new seat means freeing room (evict a dormant one) or
trimming a neighbour's utilization first.