vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6e203dcb999b084b383b93ba42a46425eebf6534
esh-pfi-infrastructure/servers/fv-ml1/README.md
T
vh ad484c3c99 chore(augaman): remove the fv-ml1 instance (Prime) 
Prime removed the second instance after the v0.1.3 bench. esh-ml1 handles a face
in ~48 ms, sits in the house next to the cameras, and holds the verified
backup. fv-ml1's gallery was empty (0 identities). The container, gallery volume,
image, compose dir (with its .env), backup dir and build sources are removed from
fv-ml1. GPU_ID / CARD_SUFFIX stay in the compose for any future second host.
2026-09-27 01:31:25 -07:00

256 lines
13 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# fv-ml1
Primary AI inference host for PFI.
## Network
- **LAN IP:** 10.251.50.54 (in-band, OS-side)
- **BMC (OOB):** 10.251.250.50 — Supermicro IPMI web UI
at <https://10.251.250.50> (homepage card: *PFI-ANA-ML2 BMC*)
- **SSH:** standard port 22 on 10.251.50.54
## Hardware
- **Chassis:** Supermicro mid-range inferencing server (bare metal,
NOT Dell / not the same box as sf-r630 / sfsrv-ana)
- **CPU:** AMD EPYC 9254 24-core (96 threads)
- **RAM:** 566 GB
- **GPUs:** **4x** NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (95.6 GB VRAM each = 382 GB total, cc 12.0 / sm_120, GPU 0-3) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores. ⚠ This line read "2x" until 2026-09-19; `nvidia-smi` reports four. Read the host, not the doc.
- **PSUs:** four present (PS1-PS4, all `ok`). ⚠ See § Power — on a single circuit that redundancy does not protect against the failure most likely to happen.
- **Storage:** ZFS `zroot` (434 GB root) + `tank` pool (raidz2, 8× NVMe, 8.6 TB at `/tank`) — drive inventory below
- **OS:** Debian 13 (trixie), kernel 6.12.x
- **Docker:** 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2
## NVMe drive inventory (`tank`, raidz2-0) — read 2026-09-09 via `nvme-cli`
All eight are Dell Express Flash PM1725b 1.6 TB SFF (Samsung OEM), PCIe 3.0 x4 behind
a Broadcom PEX switch. Two provenance batches: the `S5CU…` six (fw 1.2.2) and the
`S47V…` pair (fw 1.2.0 / 1.2.1) with thousands of prior-life power cycles.
| dev | PCI | serial | fw | pwr-on h | pwr cycles | unsafe shut. | media err | used |
|---|---|---|---|---|---|---|---|---|
| nvme0 | 46:00.0 | S5CUNEUMB05672 | 1.2.2 | 33856 | 196 | 175 | 0 | 0% |
| nvme1 | — | S5CUNEUMB05671 | 1.2.2 | 33856 | 199 | 178 | 0 | 0% |
| nvme2 | — | S5CUNEUMB05694 | 1.2.2 | 15688 | 90 | 75 | 0 | 0% |
| nvme3 | — | S5CUNEUMB05667 | 1.2.2 | 33857 | 197 | 176 | 0 | 0% |
| nvme4 | — | S5CUNEUMB05674 | 1.2.2 | 33856 | 198 | 177 | 0 | 0% |
| nvme5 | c6:00.0 | S47VNY0K600270 | 1.2.1 | 18823 | 5357 | 5342 | 0 | 1% |
| nvme6 | — | S5CUNEUMB05697 | 1.2.2 | 15570 | 88 | 73 | 0 | 0% |
| **nvme7** | 07:00.0 (slot 0-5) | S47VNY0K600221 | 1.2.0 | 19525 | 3093 | 3083 | **2084** | 2% |
⚠ **nvme7 was ABSENT from every boot 2026-04-23 → 2026-09-05** (kernel enumerated 7
NVMes per boot; PCIe downstream port `02:04.0` had nothing on bus 07). It reappeared at
the 09-05 14:26 cold boot, the pool resilvered 638 GB, and 2 CKSUM errors landed on it
at import. While it was missing `tank` was DEGRADED, and Debian's `zfsutils-linux` cron
(`/usr/lib/zfs-linux/{scrub,trim}`) only touches pools whose health is `ONLINE`, so tank
got **no scrub and no trim from 04-12 to 09-06**. ZED's `ZED_EMAIL_ADDR=root` has no
MTA behind it, so the 4½-month degradation alerted nobody. `media_errors=2084` on
nvme7 is a lifetime counter.
**Settled by the 2026-09-09 scrub** (00:29–02:02 PT, `scrub repaired 0B in 01:32:44
with 0 errors`, then `zpool clear tank` → CKSUM 2 → 0): `media_errors` read **2084
before and 2084 after** a full 6.84 TiB verify, so the counter is prior-life
history, not an active fault, and the 2 CKSUM were the stale-block artefact of the
09-05 late resilver. **nvme7 stays in service; watch the counter at every visit and
replace on growth** (`zpool replace tank nvme7n1 <new>`; any PM1725b 1.6 TB or
larger). Slot 0-5 itself deserves a reseat / cable check at the next hands-on
visit — a bay that dropped a drive for 4½ months is the likelier fault than the
drive. Playbook: `playbooks/fv-ml1-pool-health.yaml` (idempotent; rerunning is a
no-op). ⚠ **Nothing alerts on this** — see the open follow-up in
`persistent-memory.d/2026-09-09-fv-ml1-pool-actions-done.md`.
## Power — a single 20 A circuit, shared with the FV edge router
**Confirmed by the operator 2026-09-19: fv-ml1 and the R420 running OPNsense are
the ONLY loads on a dedicated 20 A circuit.**
### The budget
| | |
|---|---|
| Circuit | 20 A @ 120 V = 2400 VA absolute |
| **Continuous limit (NEC 80%)** | **1920 W** |
A GPU inference host running for hours is a continuous load by definition, so
1920 W is the real ceiling, not 2400.
### Measured (BMC, 2026-09-19, GPUs at idle)
ipmitool dcmi power reading
instantaneous 390 W min 386 W max 461 W avg 412 W
sampling period 2423 s
nvidia-smi per GPU, all four identical:
power.min_limit 250 W
power.limit 275 W <- currently ENFORCED
power.default_limit 300 W <- the card's stock Max-Q TGP
power.max_limit 325 W <- firmware ceiling, NOT an operating point
GPU draw at time of reading: 3.6 / 3.7 / 62.7 / 7.0 W ≈ 77 W total
⚠ **Compare the cap against 300 W, not 325 W.** The meaningful number is the
stock TGP the cards would otherwise run at; 325 W is an overclock ceiling nobody
should pick. So the 275 W cap is a **100 W** saving across four cards
(4 × 25 W) — not the 200 W you get by measuring against the firmware max. This
file said 200 W until the operator corrected it on 2026-09-19.
So the **non-GPU baseline is ~313 W** (EPYC 9254 24C/96T, 5+ drives, fans, board).
### Derived worst case
| Load | capped 275 W | stock 300 W |
|---|---|---|
| 4 GPUs | 1100 | 1200 |
| CPU + board + drives under load | ~400 | ~400 |
| **fv-ml1 subtotal** | **~1500** | **~1600** |
| R420 / OPNsense (estimate) | ~125 | ~125 |
| **Total** | **~1625** | **~1725** |
| **% of the 1920 W continuous budget** | **~85%** | **~90%** |
| Headroom | ~295 W | ~195 W |
**So the cap buys about 5 points of margin — 85% instead of 90%.**
### ⚠ What this forbids
- **Keep the 275 W caps.** Stock 300 W is not itself illegal — it lands near 90%
of continuous — but 90% leaves nothing for the R420 being heavier than
estimated, for PSU efficiency if the BMC reports DC, or for a warm day. The
cap costs ~8% of GPU power headroom and buys back ~100 W of circuit margin;
on a shared breaker feeding the site's router, that is a good trade. Same
posture as `feedback_idle_vram_is_reserved_not_waste`: the margin is the point,
not waste waiting to be reclaimed.
- **Never go to 325 W.** That is a firmware ceiling, not an operating point, and
it puts the circuit around 95% of continuous.
- **Do not add a fifth GPU, or another box, on this circuit.**
- Anything new here needs a load calculation first, against 1920 W, not 2400.
### ⚠ The coupling risk, which is worse than the trip
OPNsense on the R420 **is the Fountain Valley edge**. It shares the breaker with
the thing most likely to trip it. So a GPU overload does not just reboot the
inference host — it takes the site's router with it, and with the router gone
there is no remote path in to diagnose or power-cycle anything. The failure is
correlated and it locks you out of its own recovery.
Four PSUs on fv-ml1 do not help: PSU redundancy protects against a PSU dying,
not against the circuit going away, and all four are downstream of one breaker.
Breaker trips are not hypothetical on this fleet — the ANA colo has 2026
incident history for exactly this (`docs/pfi/headscale-mesh-plan.md`: "breaker,
PSU1, WAN admin closed").
### ⚠ What is NOT measured
Stated so nobody reads the table above as more solid than it is:
1. **fv-ml1 has never been measured under real 4-GPU load.** The 461 W max above
is a 40-minute idle-ish sample. The ~1500 W figure is derived from the caps,
not observed.
2. **Unknown whether the BMC reports AC input or DC output.** If DC, add ~8-10%
for PSU efficiency — about 150 W at full load, which would take the circuit
from 86% to ~94%.
3. **The R420's draw is an estimate**, not a reading.
The cheap way to close 1 and 2 together: run all four GPUs at cap (a saturating
load), read `ipmitool dcmi power reading` at the top, and compare against a clamp
meter on the circuit. Until then, treat 1650 W as a floor.
## Key paths
| Path | Purpose |
|------|---------|
| `/opt/docker/compose/<stack>/` | Compose files |
| `/opt/docker/conf/<stack>/` | Config bind mounts |
| `/tank/aimodels/huggingface/` | HF cache (267 GB, pre-downloaded models) |
| `/tank/aimodels/llm/` | Legacy GGUF models (790 GB, referenced by llama-swap as `/models/`) |
| `/var/lib/docker/` | Docker data (on zroot) |
## Running stacks
Live inventory as of 2026-07-22. Each model is its own compose stack now
(container `vllm-<x>` / `llama-<x>`); the `vllm` stack proper is just the
embed/rerank/reward trio. GPUs are pinned per container via
`deploy.resources.reservations.devices[].device_ids`.
**GPU 0 — heavy RP / reasoning seats (~88/98 GB, hot serving path):**
| Container | Port | Served model | Quant | Ctx |
|-----------|------|--------------|-------|-----|
| `vllm-gen` (project `gen-seat`) | 8015 | `qwen3.8-27b-uncensored` — the "gen" hero seat (Qwen3.8-27B Heretic-abliterated, in-house NVFP4 W4A16 + grafted MTP) | NVFP4 W4A16 (compressed-tensors) | 262k |
| `vllm-charrp-reasoning-nvfp4` | 8018 | `char-rp-reasoning` (R36 reasoning RP) | NVFP4 (modelopt) | 256k |
**GPU 1 — light / eval / retrieval + char-RP GGUF (~91/98 GB, on-demand):**
> ⚠ **This table is stale (checked 2026-09-27).** Live GPU 1 residents were `scriberr`,
> `vllm-coder`, `vllm-erp-seat` and `vllm-meromero-rp`. Read the host
> (`docker inspect … DeviceRequests`), not this table. (A fixtures-only `augaman` instance ran
> here for about an hour on 2026-09-27 for a speed bench, and was then removed on Prime's call.)
| Container | Port | Served model | Quant | Ctx |
|-----------|------|--------------|-------|-----|
| `vllm-granite` | 8004 | `granite-4.1-8b` — fleet summarizer/classifier | FP8 (compressed-tensors) | 131k |
| `llama-charrp` | 8016 | `Magidonia-24B-v4.3` Q6_K — char-RP (llama.cpp) | GGUF Q6_K | — |
| ~~`vllm-selene`~~ | ~~8011~~ | **RETIRED 2026-08-23** — lost a head-to-head against `gen` on its own judge task (see `stacks/selene/README.md`); seat downed to reclaim 17.2 GiB on GPU 1. `selene-1-mini-8b` now 404s by design; use `chat-judge`. | — | — |
| `vllm-reward` | 8003 | `Skywork-Reward-V2-Llama-3.1-8B-AWQ` — reward classifier | AWQ | 16k |
| `vllm-embed` | 8001 | `Qwen3-Embedding-0.6B` | — | 8k |
| `vllm-rerank` | 8002 | `Qwen3-Reranker-0.6B` | — | 8k |
**Infra / non-GPU:**
| Container | Port | Notes |
|-----------|------|-------|
| `dockge` | 5001 | Docker stack management UI |
| `dozzle-agent` | 7007 | Log agent → Dozzle hub on ana-docker |
| `beszel-agent` | 45876 | Metrics agent → Beszel hub on ana-docker |
Both cards run near-full (~7–10 GB headroom each) — adding a seat means placing
it on the card with room or evicting a dormant one first.
**Dormant (compose present on disk, containers stopped)** — rollback / audition
seats, safe to leave: `mistral-medium-3.5`, `mistral-small-4(-heretic)`,
`ms32-24b-angel`, `qwen3.5-122b`, `qwopus3.5-122b`, `qwen35-vl`, `qwen36-vl`,
`qwen36-27b-aeon`, `qwen-image-bench`, `vibevoice`, `comfyui`, `kokoro`,
`vllm-qwen3`.
**Also on GPU 0 (non-vLLM):**
| Container | Port | Serves | Notes |
|-----------|------|--------|-------|
| `parakeet` | 8300 | Parakeet-TDT 0.6B v3 int8 (25 languages) | ASR via sherpa-onnx, LiteLLM `ext-stt` / `whisper-1`. Relocated from irv-ml1 2026-09-15. ~800 MiB. `stacks/parakeet/`. |
⚠ **GPU 3 is deliberately kept EMPTY (2 MiB).** It is the only card that can still
take a full-size seat — `flash-next` needs 93 GiB of 96 — and vLLM sizes its KV
cache against *total* VRAM rather than free VRAM, so even a sub-1 GB tenant there
eats into a future big seat's profiling margin. Small seats go on GPU 0, which has
the most uncommitted headroom (its seats commit util 0.88; GPU 1 is at 0.975 and
GPU 2 at 0.96).
**Retired:**
- `llama-swap` (former GGUF multiplexer on :9292) — replaced by dedicated
per-model seats (e.g. `llama-charrp`); no longer running.
- `infinity` — replaced by the `vllm` stack (originally `vllm-qwen3`, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping a `transformers` build that knew Qwen3.
- `LibreChat (+ rag_api, vectordb, mongodb, meilisearch)`, `searxng` — removed from this host (searxng now on ana-docker fleet-wide).
## Refresh state
```bash
scripts/refresh-server-info.sh fv-ml1
```
Latest snapshot: `system-details.txt` (regenerate as needed).
## GPU allocation policy
Every seat is explicitly pinned via `device_ids` (no unpinned containers), and
both cards run ~90% full:
- **GPU 0:** the two heavy NVFP4 seats — `vllm-gen` (gen) and
`vllm-charrp-reasoning-nvfp4`. The live serving path (near-100% util under
load), ~42 + 45 GB.
- **GPU 1:** everything else — reward,
embed, rerank, and the Magidonia char-RP GGUF seat. Bursty/on-demand, idle
between calls, ~91 GB resident.
Pin with `deploy.resources.reservations.devices[].device_ids: ["<id>"]` in
compose. Each service caps its share with `--gpu-memory-utilization`; with both
cards near-full, placing a new seat means freeing room (evict a dormant one) or
trimming a neighbour's utilization first.
Reference in New Issue View Git Blame Copy Permalink