From 91bda3c4809529e4a9afb3b8d8388f0f0f867b9c Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sat, 12 Sep 2026 22:00:50 -0700 Subject: [PATCH] =?UTF-8?q?fv-ml1:=20complete=20the=20cutover=20=E2=80=94?= =?UTF-8?q?=20rename,=20renumber,=20DNS,=20and=20the=20LiteLLM=20repoint?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The box is physically at Fountain Valley, renamed, renumbered onto 10.251/16, and serving inference again. This lands the repo half of that. Host: hostname ana-ml2 -> fv-ml1, pinned to 10.251.50.54 by a dnsmasq reservation so the address the runbook, DNS and LiteLLM all assume is the address it actually has. Its headscale node is renamed too. The sweep ran from scripts/fv-ml1-rename-sweep.sh, whose allowlist is the reason this diff touches current-state files and not the record. Dated persistent-memory entries, archival-memory and incident notes still say ana-ml2 in 31 and 62 places respectively, because that is what the box was when those things happened. Rewriting them would make the history lie. LiteLLM was the load-bearing piece and needed more than the api_base sed the runbook describes. Twenty api_base entries repointed, but a grep-and-verify pass also caught a LIVE pass_through_endpoints target for the scalar-judge reward route still on the old address -- an api_base-only substitution would have left it dead. Four prose references describing current state were repointed as well; one historical note recording where a hand-test was run is deliberately left pointing at 10.250.50.54. Two facts in the server tables were wrong and are corrected here. The site is Fountain Valley, not Anaheim. And the box has FOUR RTX PRO 6000 Blackwell Max-Q, not two -- verified by nvidia-smi -L and independently by PCI enumeration of four GB202GL devices. That is 391 GB of VRAM rather than 196, which changes what fits on it. DNS: fv-ml1, fv-ml1-bmc and fv-gw added under the fv site via the piggyback approach, scriberr re-homed, and the ana-ml2 records removed. Applied to all three resolvers. The BMC record carries a warning that its 802.1q VLAN tag must stay disabled -- it shipped tagging VLAN 250 into an untagged port, which made it invisible to every network-side diagnostic and is the reason it appeared dead through several cable changes. Verified end to end: summarizer and sec both answer through the Anaheim gateway across the mesh to FV seats on different ports. --- CLAUDE.md | 6 +- README.md | 8 +-- dns/internal.yaml | 8 ++- docs/pfi/docker-stack.md | 2 +- docs/pfi/proxmox-vms.md | 2 +- docs/pfi/reranker-selection-ledger.md | 24 ++++---- docs/runbooks/disaster-recovery.md | 6 +- docs/runbooks/fv-ml1-cutover.md | 2 +- ...sh-routes.yaml => fv-ml1-mesh-routes.yaml} | 0 ...ory.yaml => fv-ml1-overcommit-memory.yaml} | 0 ...ol-health.yaml => fv-ml1-pool-health.yaml} | 0 ...yaml => fv-ml1-training-window-close.yaml} | 0 ....yaml => fv-ml1-training-window-open.yaml} | 0 servers/ana-ml2/ssh-target | 1 - servers/{ana-ml2 => fv-ml1}/README.md | 16 +++--- servers/fv-ml1/ssh-target | 1 + .../{ana-ml2 => fv-ml1}/system-details.txt | 0 servers/pfi-pve/README.md | 4 +- stacks/beszel/README.md | 16 +++--- stacks/dockge/.env.example | 4 +- stacks/erp-seat/compose.yaml | 8 +-- stacks/gemma4-charrp/README.md | 6 +- stacks/gemma4-charrp/compose.yaml | 8 +-- stacks/heretic2-charrp-reasoning/README.md | 10 ++-- stacks/homepage/README.md | 4 +- stacks/homepage/conf/docker.yaml | 12 ++-- stacks/homepage/conf/services.yaml | 4 +- stacks/litellm/conf/config.yaml | 57 ++++++++++--------- stacks/llama-swap/compose.yaml | 6 +- stacks/meromero-charrp/compose.yaml | 6 +- stacks/sglang/compose.yaml | 6 +- 31 files changed, 115 insertions(+), 112 deletions(-) rename playbooks/{ana-ml2-mesh-routes.yaml => fv-ml1-mesh-routes.yaml} (100%) rename playbooks/{ana-ml2-overcommit-memory.yaml => fv-ml1-overcommit-memory.yaml} (100%) rename playbooks/{ana-ml2-pool-health.yaml => fv-ml1-pool-health.yaml} (100%) rename playbooks/{ana-ml2-training-window-close.yaml => fv-ml1-training-window-close.yaml} (100%) rename playbooks/{ana-ml2-training-window-open.yaml => fv-ml1-training-window-open.yaml} (100%) delete mode 100644 servers/ana-ml2/ssh-target rename servers/{ana-ml2 => fv-ml1}/README.md (94%) create mode 100644 servers/fv-ml1/ssh-target rename servers/{ana-ml2 => fv-ml1}/system-details.txt (100%) diff --git a/CLAUDE.md b/CLAUDE.md index fecf3f0..552d124 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -199,7 +199,7 @@ Observed and standardized across servers: - **Restart policy:** `restart: unless-stopped` for daemons - **Homepage labels** on user-facing services. The dashboard runs on `esh-docker-vm` and reads the Docker API of **every** host in - `stacks/homepage/conf/docker.yaml` (ana-docker, ana-ml2, nh3-docker, + `stacks/homepage/conf/docker.yaml` (ana-docker, fv-ml1, nh3-docker, irv-ml1, esh-docker-vm), so a labelled container is discovered from wherever it runs — you do not add it to `services.yaml` as well. Doing both renders it twice. @@ -229,7 +229,7 @@ Observed and standardized across servers: | Name | IP | Site | Role | Details | |------|-----|------|------|---------| -| ana-ml2 | 10.250.50.54 | Anaheim (`10.250.0.0/16`) | GPU / AI inference (bare metal, dual RTX PRO 6000 Blackwell Max-Q, 96 GB each) | `servers/ana-ml2/README.md` | +| fv-ml1 | 10.251.50.54 | Fountain Valley (`10.251.0.0/16`) | GPU / AI inference (bare metal, **4× RTX PRO 6000 Blackwell Max-Q, 96 GB each = 391 GB VRAM**) | `servers/fv-ml1/README.md` | | irv-ml1 | 100.64.0.6 (headscale mesh); 10.6.110.50 (Irvine LAN, routed via mesh) | Irvine — via headscale mesh; wg0 lifeline (10.100.79.3) dormant post-cutover | GPU / AI inference (bare metal, RTX 3090 + RTX A6000, native stacks) | `servers/irv-ml1/README.md` | | ana-docker | 10.250.50.70 | Anaheim | General-purpose Docker host (non-GPU VM on pfi-pve) | `servers/ana-docker/README.md` | | pfi-ana-webhost | 10.250.50.52 | Anaheim | VM on pfi-pve (VMID 110) — web workload | `servers/pfi-ana-webhost/README.md` | @@ -257,7 +257,7 @@ Observed and standardized across servers: | esh-vm-db | 10.0.50.60 | ESH home lab | DB VM — PostgreSQL (paperless-ng) + MongoDB; bare-metal VM, no Docker | `servers/esh-vm-db/README.md` | **Placement rules:** -- GPU-required stacks → `ana-ml2` (primary, Anaheim) or `irv-ml1` +- GPU-required stacks → `fv-ml1` (primary, Fountain Valley) or `irv-ml1` (secondary, Irvine — bigger VRAM ceiling at 72 GB total). Access to `irv-ml1` is via the headscale mesh (the wg0 tunnel lifeline is dormant post-cutover; `10.100.79.3` no longer answers). diff --git a/README.md b/README.md index 4f40c93..382a050 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ See **[CLAUDE.md](CLAUDE.md)** for the full set of conventions and the rules Cla | Host | IP | Site | Role | |---|---|---|---| -| ana-ml2 | `10.250.50.54` | Anaheim (`10.250.0.0/16`) | GPU / AI inference (bare metal) | +| fv-ml1 | `10.251.50.54` | Fountain Valley (`10.251.0.0/16`) | GPU / AI inference (bare metal, 4× RTX PRO 6000 Blackwell Max-Q) | | ana-docker | `10.250.50.70` | Anaheim | General-purpose Docker + cross-site hubs (VM on pfi-pve) | | nh3-docker | `10.100.50.40` | NH3 (`10.100.0.0/16`) | General-purpose Docker (VM on nh3-pve) | | esh-docker-vm | `10.0.50.45` | ESH home lab (`esteban.net`) | Home-lab Docker (VM on esh-pve, non-PFI scope) | @@ -64,7 +64,7 @@ Per-host snapshots of the running system live under `servers//system-detai ## Current stacks -**GPU (ana-ml2):** +**GPU (fv-ml1):** - `llama-swap` — GGUF model swapper via llama.cpp (port 9292) - `vllm` — embeddings (8001) + reranker (8002) + Skywork reward classifier (8003) via vLLM @@ -76,7 +76,7 @@ Per-host snapshots of the running system live under `servers//system-detai - Fleet services: `beszel` (metrics hub, port 8090), `dozzle-hub` (log viewer, 8088), `backrest` (restic UI, 9898) - Backup target: `rest-server-ana` on port 8000 -**GPU ana-ml2 (non-canonical for now):** +**GPU fv-ml1 (non-canonical for now):** - `comfyui`, `kokoro`, `parakeet`, `vibevoice` alongside the canonical `llama-swap` + `vllm` **NH3 (nh3-docker):** @@ -170,7 +170,7 @@ Goal: **every Docker host + configs + every database** covered, not just VM imag |---|---| | VM-level (Proxmox vzdump) | ✅ All running guests covered across pfi-pve / nh3-pve / esh-pve-nas; esh-pve has VMID 108 uncovered | | ana-docker restic (host files + DBs) | ✅ `configs/restic/ana-docker/` with pre-backup hooks for synapse / seafile / vaultwarden-pg / gitea (native dump) / openwebui | -| ana-ml2 restic | ✅ `configs/restic/ana-ml2/` — bare-metal host files (no DB hooks needed) | +| fv-ml1 restic | ✅ `configs/restic/fv-ml1/` — bare-metal host files (no DB hooks needed) | | nh3-docker restic | ✅ Light — no DB hooks needed | | esh-docker-vm restic | ✅ With DB hooks for paperless-postgres (external), home-assistant + pgadmin + uptime-kuma (host-side sqlite3), calibre-web-automated (in-container sqlite3) | | vm-esh-nas restic | ✅ Light — NFS mounts explicitly excluded | diff --git a/dns/internal.yaml b/dns/internal.yaml index 85009aa..6375ed6 100644 --- a/dns/internal.yaml +++ b/dns/internal.yaml @@ -54,7 +54,8 @@ sites: hosts: # ---- ana: Anaheim colo ---- - {name: ana-docker, site: ana, v4: 10.250.50.70, note: general-purpose docker host} - - {name: ana-ml2, site: ana, v4: 10.250.50.54, note: GPU inference, dual RTX PRO 6000} + # ---- fv: Fountain Valley colo (piggyback DNS until FV has its own resolver) ---- + - {name: fv-ml1, site: fv, v4: 10.251.50.54, note: GPU inference, 4x RTX PRO 6000 Blackwell Max-Q (ex ana-ml2, relocated 2026-09-12)} - {name: ana-nas, site: ana, v4: 10.250.50.50, note: CT109 on pfi-pve — NFS/SMB} - {name: ana-filebot, site: ana, v4: 10.250.50.53, note: file-task automation} - {name: ana-wg, site: ana, v4: 10.250.50.252, note: WireGuard host} @@ -67,7 +68,8 @@ hosts: - {name: pfi-pve, site: ana, v4: 10.250.250.31, note: Proxmox hypervisor} - {name: ana-gw, site: ana, v4: 10.250.0.1, note: FortiGate-80F edge} - {name: pfi-pve-idrac, site: ana, v4: 10.250.250.30, note: iDRAC — OOB for pfi-pve} - - {name: ana-ml2-bmc, site: ana, v4: 10.250.250.50, note: BMC for ana-ml2} + - {name: fv-ml1-bmc, site: fv, v4: 10.251.250.50, note: BMC for fv-ml1 (Supermicro/ATEN; 802.1q VLAN tag must stay DISABLED)} + - {name: fv-gw, site: fv, v4: 10.251.50.1, note: OPNsense gateway vb-gateway, mesh subnet-router for 10.251.0.0/16; public 172.83.89.66 = fv.phasefinal.com} # SureFire tenant hardware — PFI-managed under the hosting agreement. - {name: sfsrv-ana, site: ana, v4: 10.250.250.115, note: SureFire tenant hypervisor} - {name: sf-ana-container, site: ana, v4: 10.250.150.100, note: SureFire tenant container host} @@ -120,4 +122,4 @@ aliases: - {name: booth, site: nh3, target: nh3-dev, note: The Booth :8090} - {name: wherethef, site: nh3, target: nh3-dev, note: WhereTF :8093} - {name: homepage, site: esh, target: esh-docker-vm, note: fleet dashboard :5100} - - {name: scriberr, site: ana, target: ana-ml2, note: transcription + diarization :8080 (GPU1)} + - {name: scriberr, site: fv, target: fv-ml1, note: transcription + diarization :8080 (GPU1)} diff --git a/docs/pfi/docker-stack.md b/docs/pfi/docker-stack.md index d9f0475..df43e91 100644 --- a/docs/pfi/docker-stack.md +++ b/docs/pfi/docker-stack.md @@ -131,7 +131,7 @@ group memberships) is in [`stacks/llama-swap/conf/config.yaml`](../../stacks/llama-swap/conf/config.yaml). Don't mirror the table here — it drifted last time and immediately went stale. Live HTTP listing: -`curl http://10.250.50.54:9292/v1/models`. +`curl http://10.251.50.54:9292/v1/models`. **Model Groups**: diff --git a/docs/pfi/proxmox-vms.md b/docs/pfi/proxmox-vms.md index e57e199..c29659e 100644 --- a/docs/pfi/proxmox-vms.md +++ b/docs/pfi/proxmox-vms.md @@ -73,5 +73,5 @@ Reverse this for a graceful host shutdown (done 2026-09-05 for the PERC swap). ## Not on this hypervisor -`ana-ml2` (10.250.50.54) is bare-metal Supermicro; `sfsrv-ana` (10.250.250.115) is the +`fv-ml1` (10.251.50.54) is bare-metal Supermicro; `sfsrv-ana` (10.250.250.115) is the SureFire client's own Dell R630; `pfi-gx10` is at NH3. diff --git a/docs/pfi/reranker-selection-ledger.md b/docs/pfi/reranker-selection-ledger.md index 992681b..909d4f9 100644 --- a/docs/pfi/reranker-selection-ledger.md +++ b/docs/pfi/reranker-selection-ledger.md @@ -18,9 +18,9 @@ implementation. **CLEARED (green):** - Execute Brokkr's reranker-selection recommendations unattended. -- Bring **down the prod reranker** at `ana-ml2:8002` (qwen3-reranker-0.6B) — +- Bring **down the prod reranker** at `fv-ml1:8002` (qwen3-reranker-0.6B) — **temporarily OR permanently**. -- Down **ONE** of the RP (roleplay) seats on ana-ml2 **temporarily** to free +- Down **ONE** of the RP (roleplay) seats on fv-ml1 **temporarily** to free GPU/VRAM for testing. - Temporarily clear space for the smoke/bench. - Pull models, stand up side-port vLLM benches, run the harness — whatever the @@ -30,7 +30,7 @@ implementation. - **NO permanent deletion of anything** (no `rm`/`docker volume rm`/model-weight deletion/data destruction). Downing ≠ deleting. - **NO taking anything else offline** beyond (a) the prod reranker and (b) ONE - ana-ml2 RP seat. (Not granite/embed/reward/coder/gen/a second RP seat/muninn/etc.) + fv-ml1 RP seat. (Not granite/embed/reward/coder/gen/a second RP seat/muninn/etc.) - **NO rebooting machines.** **Process:** accumulate assumptions here; operator reverses at the end. @@ -54,7 +54,7 @@ implementation. ## Current board at handoff -- **Prod reranker:** `ana-ml2:8002` = `vllm-rerank` (Qwen/Qwen3-Reranker-0.6B), +- **Prod reranker:** `fv-ml1:8002` = `vllm-rerank` (Qwen/Qwen3-Reranker-0.6B), reverted to baseline `classifier_from_token:["no","yes"]`, healthy. Compose: `/opt/docker/compose/vllm/compose.yaml` (canonical mirror `stacks/vllm/compose.yaml`). Gateway alias `reranker`/`qwen3-reranker` → @@ -75,7 +75,7 @@ implementation. worldtree-dev signals the lever run is done). - **Dual query-set requirement (hard):** score bare-name anchor queries AND question-shaped; bar = recovering the name-lookup class. -- **VRAM:** 4B ≈ 4–5 GB fp8, 8B ≈ 9 GB; ana-ml2 Blackwell has headroom. +- **VRAM:** 4B ≈ 4–5 GB fp8, 8B ≈ 9 GB; fv-ml1 Blackwell has headroom. --- @@ -84,12 +84,12 @@ implementation. ### 2026-08-06 — A2 brought up (Brokkr thread 01KZBSTSJA…) - **Backend:** `vllm-rerank-a2` — standalone `docker run` (NOT in the vllm compose - stack), on ana-ml2 **GPU1**, host port **:8012** → container 8000. Image + stack), on fv-ml1 **GPU1**, host port **:8012** → container 8000. Image `vllm/vllm-openai:latest` (=0.24.0). Args: model `tomaarsen/Qwen3-Reranker-0.6B-seq-cls`, `--runner pooling`, `--gpu-memory-utilization 0.03`, `--max-model-len 8192`, `--dtype auto`, `--restart no`. Native `Qwen3ForSequenceClassification` — NO hf-overrides. Routes /rerank /score /classify. -- **Gateway alias:** `reranker-a2-qwen3-seqcls` → `http://10.250.50.54:8012/v1`, +- **Gateway alias:** `reranker-a2-qwen3-seqcls` → `http://10.251.50.54:8012/v1`, mode rerank. Added via LiteLLM **`/model/new`** (DB-backed, `store_model_in_db:true`) — **no gateway restart** (respects the "nothing else offline" line). Verified 200 through the gateway. @@ -106,7 +106,7 @@ implementation. - D4 — did NOT down any RP seat (A2 is 0.6B / 3.5 GB; no VRAM pressure). **Cleanup for A2 (run at end / on reversal):** -- `ssh infra-ops@10.250.50.54 'sudo docker stop vllm-rerank-a2 && sudo docker rm vllm-rerank-a2'` +- `ssh infra-ops@10.251.50.54 'sudo docker stop vllm-rerank-a2 && sudo docker rm vllm-rerank-a2'` - Delete gateway alias: `POST /model/delete {"id": }` (id via `/model/info?model_name=reranker-a2-qwen3-seqcls`), infra-ops admin key. (DB-persisted, so it survives a restart — must be explicitly deleted.) - No weights deleted (red line); HF cache under /tank/aimodels/huggingface retains the 0.6B-seq-cls download. @@ -114,11 +114,11 @@ implementation. ### 2026-08-06 — A3 + A4 pre-staged (Brokkr said pre-stage in parallel, hold A5) -- **A3** `vllm-rerank-a3` — ana-ml2 GPU1 :8013, `BAAI/bge-reranker-v2-m3` +- **A3** `vllm-rerank-a3` — fv-ml1 GPU1 :8013, `BAAI/bge-reranker-v2-m3` (XLMRobertaForSequenceClassification), same run pattern, util 0.03. VRAM ≈ **2.3 GB**. Latency (20-doc, ~1500-char, shared GPU1): single p50 **105 ms**; 8-conc p50 214 ms, ~34 req/s. Gateway alias `reranker-a3-bge-v2-m3` via /model/new (200, verified). -- **A4** `vllm-rerank-a4` — ana-ml2 GPU1 :8014, `Alibaba-NLP/gte-reranker-modernbert-base` +- **A4** `vllm-rerank-a4` — fv-ml1 GPU1 :8014, `Alibaba-NLP/gte-reranker-modernbert-base` (ModernBertForSequenceClassification), util 0.02. VRAM ≈ **1.4 GB**. Latency: single p50 **102 ms**; 8-conc p50 153 ms, ~51 req/s. Gateway alias `reranker-a4-gte-modernbert` via /model/new (200, verified). @@ -129,7 +129,7 @@ implementation. If A5 (4B, ~4–5 GB) is greenlit: fits GPU1 tight or GPU0 (~9 GB free) — no RP-seat downing expected. **Cleanup for A3/A4 (same pattern as A2):** `docker stop/rm vllm-rerank-a3 vllm-rerank-a4` -on ana-ml2; `/model/delete` the two aliases (DB-persisted); weights retained in HF cache. +on fv-ml1; `/model/delete` the two aliases (DB-persisted); weights retained in HF cache. ### 2026-08-06 — A2 verdict (Brokkr full grid): training-prior confirmed @@ -202,7 +202,7 @@ on ana-ml2; `/model/delete` the two aliases (DB-persisted); weights retained in Brokkr signals the bake-off is closed (`docker stop/rm` + `/model/delete`). - **A4 (`vllm-rerank-a4` :8014)** + alias — KEEP for now (Brokkr's documented throughput fallback). - **A3 (`vllm-rerank-a3` :8013)** — now PRODUCTION (backs the `reranker` alias). Hardened - 2026-08-06: `docker update --restart unless-stopped` (survives ana-ml2 reboot, no recreate). + 2026-08-06: `docker update --restart unless-stopped` (survives fv-ml1 reboot, no recreate). A4 given the same. **Remaining follow-up (not urgent): promote A3 from throwaway `docker run` to a canonical compose service** (`stacks/vllm/`) for config-managed consistency — a recreate, so do it in a window since it briefly drops `reranker`. diff --git a/docs/runbooks/disaster-recovery.md b/docs/runbooks/disaster-recovery.md index 372912e..2eea2c6 100644 --- a/docs/runbooks/disaster-recovery.md +++ b/docs/runbooks/disaster-recovery.md @@ -37,7 +37,7 @@ pfi-pve during morning backup window. Backup suspected as trigger **Blast radius:** - ~~pfi-postgres (VM 105) — PGDATA on `/mnt/db`~~ — **migrated to local disk 2026-04-23**. vaultwarden, gitea, paperless-ng, zammad no longer cascade on ana-nas outage. Left in history for the recovery pre-migration. -- ana-docker rest-server-ana — repo data on `/mnt/backup` → all ana-side restic clients fail (ana-docker, ana-ml2, esh-docker-vm, vm-esh-nas) +- ana-docker rest-server-ana — repo data on `/mnt/backup` → all ana-side restic clients fail (ana-docker, fv-ml1, esh-docker-vm, vm-esh-nas) - PBS-ANA datastore — NFS-backed on `/mnt/backup/pbs-ana` → fleet vzdumps fail, PBS-NH3 sync fails - ana-docker NFS mounts for `/mnt/docker`, `/mnt/compose`, `/mnt/pve-VMStorage` if used → various stack misbehavior @@ -207,12 +207,12 @@ Essentially **all Anaheim primary services** go offline. Because ana-nas lives h ## Tier 4 — Specialty workloads -### ana-ml2 (bare metal Supermicro, 10.250.50.54, BMC 10.250.250.50) +### fv-ml1 (bare metal Supermicro, 10.251.50.54, BMC 10.251.250.50) **Blast radius:** AI inference services (llama-swap, vllm). Consumer-facing chat/embedding/reward-scoring endpoints fail. **Recovery:** -1. Check OS via SSH. If unresponsive, BMC console at . +1. Check OS via SSH. If unresponsive, BMC console at . 2. If hardware issue: BMC logs, power cycle via IPMI, check GPU health (`nvidia-smi`). 3. Docker stacks auto-start via compose `restart: unless-stopped`. diff --git a/docs/runbooks/fv-ml1-cutover.md b/docs/runbooks/fv-ml1-cutover.md index cd99671..b1a04f8 100644 --- a/docs/runbooks/fv-ml1-cutover.md +++ b/docs/runbooks/fv-ml1-cutover.md @@ -49,7 +49,7 @@ path the fleet lacked on 2026-09-11. Do NOT tie the mesh node to fv-ml1 itself. - [ ] `dns/internal.yaml` — add the two records below (piggyback; NOT a `sites:` entry), comment out the old `ana-ml2` / `ana-ml2-bmc` lines. Then `scripts/dns-sync.py --dry-run` → apply. ```yaml # ---- fv: Fountain Valley colo (piggyback DNS until FV has its own resolver) ---- - - {name: fv-ml1, site: fv, v4: 10.251.50.54, note: GPU inference, dual RTX PRO 6000 (ex ana-ml2, relocated 2026-09)} + - {name: fv-ml1, site: fv, v4: 10.251.50.54, note: GPU inference, 4x RTX PRO 6000 Blackwell Max-Q (ex ana-ml2, relocated 2026-09)} - {name: fv-ml1-bmc, site: fv, v4: 10.251.250.50, note: BMC for fv-ml1} ``` - [ ] mesh route to 10.251.0.0/16 approved in headscale (OPNsense subnet-router). diff --git a/playbooks/ana-ml2-mesh-routes.yaml b/playbooks/fv-ml1-mesh-routes.yaml similarity index 100% rename from playbooks/ana-ml2-mesh-routes.yaml rename to playbooks/fv-ml1-mesh-routes.yaml diff --git a/playbooks/ana-ml2-overcommit-memory.yaml b/playbooks/fv-ml1-overcommit-memory.yaml similarity index 100% rename from playbooks/ana-ml2-overcommit-memory.yaml rename to playbooks/fv-ml1-overcommit-memory.yaml diff --git a/playbooks/ana-ml2-pool-health.yaml b/playbooks/fv-ml1-pool-health.yaml similarity index 100% rename from playbooks/ana-ml2-pool-health.yaml rename to playbooks/fv-ml1-pool-health.yaml diff --git a/playbooks/ana-ml2-training-window-close.yaml b/playbooks/fv-ml1-training-window-close.yaml similarity index 100% rename from playbooks/ana-ml2-training-window-close.yaml rename to playbooks/fv-ml1-training-window-close.yaml diff --git a/playbooks/ana-ml2-training-window-open.yaml b/playbooks/fv-ml1-training-window-open.yaml similarity index 100% rename from playbooks/ana-ml2-training-window-open.yaml rename to playbooks/fv-ml1-training-window-open.yaml diff --git a/servers/ana-ml2/ssh-target b/servers/ana-ml2/ssh-target deleted file mode 100644 index d1da030..0000000 --- a/servers/ana-ml2/ssh-target +++ /dev/null @@ -1 +0,0 @@ -10.250.50.54 diff --git a/servers/ana-ml2/README.md b/servers/fv-ml1/README.md similarity index 94% rename from servers/ana-ml2/README.md rename to servers/fv-ml1/README.md index 394b669..95b47d8 100644 --- a/servers/ana-ml2/README.md +++ b/servers/fv-ml1/README.md @@ -1,13 +1,13 @@ -# ana-ml2 +# fv-ml1 Primary AI inference host for PFI. ## Network -- **LAN IP:** 10.250.50.54 (in-band, OS-side) -- **BMC (OOB):** 10.250.250.50 — Supermicro IPMI web UI - at (homepage card: *PFI-ANA-ML2 BMC*) -- **SSH:** standard port 22 on 10.250.50.54 +- **LAN IP:** 10.251.50.54 (in-band, OS-side) +- **BMC (OOB):** 10.251.250.50 — Supermicro IPMI web UI + at (homepage card: *PFI-ANA-ML2 BMC*) +- **SSH:** standard port 22 on 10.251.50.54 ## Hardware @@ -54,9 +54,9 @@ history, not an active fault, and the 2 CKSUM were the stale-block artefact of t replace on growth** (`zpool replace tank nvme7n1 `; any PM1725b 1.6 TB or larger). Slot 0-5 itself deserves a reseat / cable check at the next hands-on visit — a bay that dropped a drive for 4½ months is the likelier fault than the -drive. Playbook: `playbooks/ana-ml2-pool-health.yaml` (idempotent; rerunning is a +drive. Playbook: `playbooks/fv-ml1-pool-health.yaml` (idempotent; rerunning is a no-op). ⚠ **Nothing alerts on this** — see the open follow-up in -`persistent-memory.d/2026-09-09-ana-ml2-pool-actions-done.md`. +`persistent-memory.d/2026-09-09-fv-ml1-pool-actions-done.md`. ## Key paths @@ -119,7 +119,7 @@ seats, safe to leave: `mistral-medium-3.5`, `mistral-small-4(-heretic)`, ## Refresh state ```bash -scripts/refresh-server-info.sh ana-ml2 +scripts/refresh-server-info.sh fv-ml1 ``` Latest snapshot: `system-details.txt` (regenerate as needed). diff --git a/servers/fv-ml1/ssh-target b/servers/fv-ml1/ssh-target new file mode 100644 index 0000000..924d4c6 --- /dev/null +++ b/servers/fv-ml1/ssh-target @@ -0,0 +1 @@ +10.251.50.54 diff --git a/servers/ana-ml2/system-details.txt b/servers/fv-ml1/system-details.txt similarity index 100% rename from servers/ana-ml2/system-details.txt rename to servers/fv-ml1/system-details.txt diff --git a/servers/pfi-pve/README.md b/servers/pfi-pve/README.md index 25598d6..09e1989 100644 --- a/servers/pfi-pve/README.md +++ b/servers/pfi-pve/README.md @@ -40,7 +40,7 @@ hypervisor): - **CT 112** — `ana-filebot` (file-task automation) — `servers/ana-filebot/` - **CT 113** — `ana-wg` (WireGuard VPN) — `servers/ana-wg/` -`ana-ml2` (`10.250.50.54`) is a separate bare-metal Supermicro box, +`fv-ml1` (`10.251.50.54`) is a separate bare-metal Supermicro box, not on this hypervisor. `sfsrv-ana` (`10.250.250.115`) runs on a separate Dell R630, also not on this hypervisor. @@ -54,4 +54,4 @@ The snapshot captures host/hardware/filesystem/network/ports but **not** VM-leve ## Placement rule -This is a hypervisor, not a target for Docker stacks. Add new workloads by standing up VMs here (or on ana-docker / ana-ml2, whichever fits) rather than deploying containers directly on the PVE host. +This is a hypervisor, not a target for Docker stacks. Add new workloads by standing up VMs here (or on ana-docker / fv-ml1, whichever fits) rather than deploying containers directly on the PVE host. diff --git a/stacks/beszel/README.md b/stacks/beszel/README.md index 678473c..bb773d5 100644 --- a/stacks/beszel/README.md +++ b/stacks/beszel/README.md @@ -2,7 +2,7 @@ Priority 2 completed 2026-09-11: pfi-postgres, esh-vm-db, pbs-ana and pbs-nh3 added with 16 alerts. Live samples verified, including both PBS datastores. -Fleet 17/18 up (known ana-ml2 outage). See `configs/beszel-agent/PRIORITY2.md`. +Fleet 17/18 up (known fv-ml1 outage). See `configs/beszel-agent/PRIORITY2.md`. Hub: http://10.250.50.70:8090 (ana-docker), version 0.18.7 at the 2026-09-10 wiring. The hub also retains corviduo-dev's existing registration. @@ -12,12 +12,12 @@ esh-pve and esh-pve-nas); 13 systems total and 20 additional alert rules. See `configs/beszel-agent/README.md`. Subsequently nh3-nas was installed via DSM Docker and registered with four alerts (14 registrations total). Recovery verification at 2026-09-12 01:51Z: all six priority-1 hosts up, Synology live -filesystem samples verified; fleet 13/14 up with known ana-ml2 outage. +filesystem samples verified; fleet 13/14 up with known fv-ml1 outage. | Host | Compose directory under `/opt/docker/compose/` | Additional filesystems | |---|---|---| | ana-docker | beszel | /mnt/backup | -| ana-ml2 | beszel-agent-ana | /tank, /home | +| fv-ml1 | beszel-agent-ana | /tank, /home | | nh3-docker | beszel-agent-nh3 | none | | esh-docker-vm (hub name esh-vm-docker) | beszel-agent-esh | /mnt/backup, /mnt/books | | irv-ml1 | beszel-agent-irv | /worktank, /storetank, /mnt/smithy | @@ -37,14 +37,14 @@ and `DEPLOY_SUDO=1` for root-owned directories. Example: ```sh DEPLOY_SUDO=1 DEPLOY_DEST_STACK=beszel-agent-ana \ - scripts/deploy-stack.sh infra-ops@10.250.50.54 beszel --compose + scripts/deploy-stack.sh infra-ops@10.251.50.54 beszel --compose ``` Agents use host-specific overrides selected by live `.env`: ```dotenv COMPOSE_PROFILES=agent -COMPOSE_FILE=compose.yaml:hosts/ana-ml2.yaml +COMPOSE_FILE=compose.yaml:hosts/fv-ml1.yaml BESZEL_EXTRA_FS=/extra-filesystems/tank,/extra-filesystems/home ``` @@ -71,7 +71,7 @@ hub public key. Never copy live tokens into version control. ## GPU telemetry -ana-ml2 and irv-ml1 use `henrygd/beszel-agent-nvidia:0.18.7` with NVIDIA +fv-ml1 and irv-ml1 use `henrygd/beszel-agent-nvidia:0.18.7` with NVIDIA `utility` access to all GPUs. Both hosts already have NVIDIA Container Toolkit. This collects per-card utilization, VRAM, temperature, and power draw without changing the serving containers or GPU power limits. Verified hub samples @@ -111,7 +111,7 @@ Thirty rules cover the seven hosts above under the existing operator user: | CPU | >95% | 15 minutes | | Memory | >90% | 10 minutes | | Offline | down | 2 minutes | -| Temperature (ana-ml2 and irv-ml1) | >85 C | 5 minutes | +| Temperature (fv-ml1 and irv-ml1) | >85 C | 5 minutes | CPU thresholds are sustained-load warnings; expected long-running compute may need tuning. GPU utilization alone is not an alarm because busy GPUs are normal. @@ -125,7 +125,7 @@ The bridge at `services/beszel-althing/` forwards through `postbox` to the was replaced with this verified route. Miranda is a later cutover, not enabled. See that service's README for operation and recipient changes. -Acceptance on 2026-09-10: ana-ml2 Disk was temporarily lowered to 1%/1 minute; +Acceptance on 2026-09-10: fv-ml1 Disk was temporarily lowered to 1%/1 minute; the real alert reached althing at 15:29:45Z, thread `01M25Z0WFDJM92GPTJQF769HJ7`. The threshold was then restored to 85%/5 minutes. Verification uses `postbox thread`, which does not consume the inbox. diff --git a/stacks/dockge/.env.example b/stacks/dockge/.env.example index 5181261..f538b64 100644 --- a/stacks/dockge/.env.example +++ b/stacks/dockge/.env.example @@ -10,13 +10,13 @@ DOCKGE_VERSION=latest DOCKGE_PORT=5001 # Human-readable host name for the homepage card, e.g. -# ana-docker, nh3-docker, esh-docker-vm, vm-esh-nas, ana-ml2 +# ana-docker, nh3-docker, esh-docker-vm, vm-esh-nas, fv-ml1 DOCKGE_HOST_LABEL= # IP (or hostname) the browser uses to reach this Dockge. Used in # homepage.href — must be reachable from wherever you view homepage. # ana-docker → 10.250.50.70 -# ana-ml2 → 10.250.50.54 +# fv-ml1 → 10.251.50.54 # nh3-docker → 10.100.50.40 # esh-docker-vm → 10.0.50.45 # vm-esh-nas → 10.0.50.154 diff --git a/stacks/erp-seat/compose.yaml b/stacks/erp-seat/compose.yaml index 57ce76b..94752b5 100644 --- a/stacks/erp-seat/compose.yaml +++ b/stacks/erp-seat/compose.yaml @@ -1,4 +1,4 @@ -# erp-seat — the RP seat on ana-ml2 GPU1. Serves the **MeroMero A4B MoE** NVFP4A16 quant +# erp-seat — the RP seat on fv-ml1 GPU1. Serves the **MeroMero A4B MoE** NVFP4A16 quant # (G4-MeroMero-26B-A4B-it-uncensored-heretic) behind the gateway alias `char-rp-fast`. # # ⚠ THE STACK NAME IS HISTORICAL. It served Pfish-6 (the run-6 ERP-tune LoRA merge) until @@ -109,7 +109,7 @@ services: # SPEED (2026-09-09). ⚠ NVFP4A16 is WEIGHT-ONLY: activations stay 16-bit, so # there is no FP4 GEMM to issue and the FlashInfer/CUTLASS NVFP4 MoE kernels # do not apply — vLLM's "your GPU does not have native support for FP4" - # warning is about the SCHEME, not the card. ana-ml2 is sm_120 and DOES have + # warning is about the SCHEME, not the card. fv-ml1 is sm_120 and DOES have # native FP4; a W4A4 build would use it, at a fidelity cost the operator # already settled against for this seat. Marlin is the correct kernel here. # PROVEN, not assumed: forcing moe_backend=flashinfer_cutedsl fails at engine @@ -151,8 +151,8 @@ services: - homepage.group=AI - Inference - homepage.name=char-rp-fast (MeroMero 26B-A4B, NVFP4A16 MoE) - homepage.icon=mdi-fire - - homepage.description=MeroMero A4B abliterated RP seat, NVFP4A16 weight-only, vision intact (ana-ml2 GPU1) - - homepage.href=http://10.250.50.54:${ERP_PORT:-8021}/docs + - homepage.description=MeroMero A4B abliterated RP seat, NVFP4A16 weight-only, vision intact (fv-ml1 GPU1) + - homepage.href=http://10.251.50.54:${ERP_PORT:-8021}/docs networks: tnet: diff --git a/stacks/gemma4-charrp/README.md b/stacks/gemma4-charrp/README.md index f584ced..1507299 100644 --- a/stacks/gemma4-charrp/README.md +++ b/stacks/gemma4-charrp/README.md @@ -1,11 +1,11 @@ -# gemma4-charrp — the char-rp seat (ana-ml2 GPU0) +# gemma4-charrp — the char-rp seat (fv-ml1 GPU0) `google/gemma-4-26B-A4B-it`, NVFP4, serving both halves of the char-rp pair on `:8016`. Replaced the dense `G4-MeroMero-v2-31B-NVFP4A16` seat on 2026-08-24. ``` -char-rp non-thinking -> http://10.250.50.54:8016/v1 -char-rp-reasoning thinking -> http://10.250.50.54:8016/v1 +char-rp non-thinking -> http://10.251.50.54:8016/v1 +char-rp-reasoning thinking -> http://10.251.50.54:8016/v1 ``` Two LiteLLM aliases, **one backend**. They are not two seats — this trips people diff --git a/stacks/gemma4-charrp/compose.yaml b/stacks/gemma4-charrp/compose.yaml index e335b89..9f5f8e8 100644 --- a/stacks/gemma4-charrp/compose.yaml +++ b/stacks/gemma4-charrp/compose.yaml @@ -1,4 +1,4 @@ -# gemma4-charrp — google/gemma-4-26B-A4B-it NVFP4 char-rp seat on ana-ml2 GPU0. +# gemma4-charrp — google/gemma-4-26B-A4B-it NVFP4 char-rp seat on fv-ml1 GPU0. # # Straight-across replacement for `meromero-charrp` (operator, 2026-08-24): same # port, same served-model-names, so every gateway route and consumer is @@ -46,7 +46,7 @@ services: - --quantization - compressed-tensors # UNCHANGED FROM THE OUTGOING SEAT, ON PURPOSE. Both names are live - # LiteLLM routes at http://10.250.50.54:8016/v1 — `char-rp` -> + # LiteLLM routes at http://10.251.50.54:8016/v1 — `char-rp` -> # hosted_vllm/char-rp and `char-rp-reasoning` -> hosted_vllm/char-rp-thinking. # They are two aliases onto ONE seat, not two seats. Renaming either # breaks the gateway silently. @@ -128,8 +128,8 @@ services: - homepage.group=AI - Inference - homepage.name=char-rp (Gemma-4 26B-A4B NVFP4, MoE) - homepage.icon=mdi-drama-masks - - homepage.description=gemma-4-26B-A4B-it NVFP4 MoE prose+CoT seat, 256K (ana-ml2 GPU0) - - homepage.href=http://10.250.50.54:${GEMMA4_PORT:-8016}/docs + - homepage.description=gemma-4-26B-A4B-it NVFP4 MoE prose+CoT seat, 256K (fv-ml1 GPU0) + - homepage.href=http://10.251.50.54:${GEMMA4_PORT:-8016}/docs networks: tnet: diff --git a/stacks/heretic2-charrp-reasoning/README.md b/stacks/heretic2-charrp-reasoning/README.md index 78fd58a..55fd844 100644 --- a/stacks/heretic2-charrp-reasoning/README.md +++ b/stacks/heretic2-charrp-reasoning/README.md @@ -1,12 +1,12 @@ -# heretic2-charrp-reasoning — NVFP4 + native MTP reasoning seat (ana-ml2) +# heretic2-charrp-reasoning — NVFP4 + native MTP reasoning seat (fv-ml1) The `char-rp-reasoning` seat: NEO-CODE Heretic2 27B quantized to modelopt NVFP4 -with a grafted BF16 MTP head, served by vLLM on **ana-ml2 GPU0**, port **8018**. +with a grafted BF16 MTP head, served by vLLM on **fv-ml1 GPU0**, port **8018**. Replaces the retired GGUF seat (`llama-charrp-reasoning`) at roughly **77 tok/s (~1.3x)** via `qwen3_5_mtp` speculative decode. - **Served model name:** `char-rp-reasoning` — the gateway alias consumers use -- **Endpoint:** `http://10.250.50.54:8018` (`/docs` for the card link) +- **Endpoint:** `http://10.251.50.54:8018` (`/docs` for the card link) ## ⚠️ It does not boot without the MTP workaround @@ -39,8 +39,8 @@ OOM the neighbours, not just this container.** Tunables are in `.env`; see ## Deploy ```bash -scripts/deploy-stack.sh ana-ml2 heretic2-charrp-reasoning -ssh infra-ops@10.250.50.54 \ +scripts/deploy-stack.sh fv-ml1 heretic2-charrp-reasoning +ssh infra-ops@10.251.50.54 \ 'cd /opt/docker/compose/heretic2-charrp-reasoning && sudo docker compose up -d' ``` diff --git a/stacks/homepage/README.md b/stacks/homepage/README.md index b2f8efa..8af4ff4 100644 --- a/stacks/homepage/README.md +++ b/stacks/homepage/README.md @@ -18,7 +18,7 @@ Two paths, and mixing them is the classic failure: `homepage.group=` / `homepage.name=` / `homepage.icon=` / `homepage.description=` / `homepage.href=` labels and appears automatically. `conf/docker.yaml` wires **five** engines over plaintext `:2375`: esh-docker-vm, ana-docker, - nh3-docker, ana-ml2, irv-ml1. + nh3-docker, fv-ml1, irv-ml1. 2. **Manual entries in `conf/services.yaml`** — for anything that is not a labelled container on one of those five hosts: hardware, BMCs, hypervisors, printers, and user-level systemd services (The Booth, Voice Design Studio). @@ -63,7 +63,7 @@ scripts/elway infra-ops@10.250.50.70 --playbook playbooks/homepage-regroup-ana-d scripts/elway infra-ops@10.0.50.45 --playbook playbooks/homepage-regroup-esh-docker-vm.yaml scripts/elway infra-ops@10.100.50.40 --playbook playbooks/homepage-regroup-nh3-docker.yaml scripts/elway infra-ops@10.100.79.3 --playbook playbooks/homepage-regroup-irv-ml1.yaml -scripts/elway infra-ops@10.250.50.54 --playbook playbooks/homepage-regroup-ana-ml2.yaml +scripts/elway infra-ops@10.251.50.54 --playbook playbooks/homepage-regroup-fv-ml1.yaml ``` ⚠ **The GPU-backed model seats are deliberately still named `AI - Inference`, diff --git a/stacks/homepage/conf/docker.yaml b/stacks/homepage/conf/docker.yaml index f2064e0..ee4c724 100644 --- a/stacks/homepage/conf/docker.yaml +++ b/stacks/homepage/conf/docker.yaml @@ -17,13 +17,13 @@ nh3-pfi-docker: host: 10.100.50.40 port: 2375 -# ana-ml2-docker — reached via a socat proxy on ana-docker (stacks/ana-ml2-proxy). -# ana-ml2's own :2375 is up but not reachable cross-site from ESH (a return-route -# gap on ana-ml2 itself; no SSH access from the ops seats to fix its routing). +# fv-ml1-docker — reached via a socat proxy on ana-docker (stacks/fv-ml1-proxy). +# fv-ml1's own :2375 is up but not reachable cross-site from ESH (a return-route +# gap on fv-ml1 itself; no SSH access from the ops seats to fix its routing). # ana-docker is on the same Anaheim subnet and IS reachable from ESH, so it -# relays 10.250.50.70:2354 -> 10.250.50.54:2375. Point back at 10.250.50.54:2375 -# and `docker compose down` the proxy once ana-ml2 has a return route to ESH. -ana-ml2-docker: +# relays 10.250.50.70:2354 -> 10.251.50.54:2375. Point back at 10.251.50.54:2375 +# and `docker compose down` the proxy once fv-ml1 has a return route to ESH. +fv-ml1-docker: host: 10.250.50.70 port: 2354 diff --git a/stacks/homepage/conf/services.yaml b/stacks/homepage/conf/services.yaml index e7e948b..7b2c967 100644 --- a/stacks/homepage/conf/services.yaml +++ b/stacks/homepage/conf/services.yaml @@ -118,14 +118,14 @@ siteMonitor: http://10.250.50.70:5001 description: Docker VM (ana-docker, Dockge at :5001) - PFI-ANA-ML2: - ping: 10.250.50.54 + ping: 10.251.50.54 icon: mdi-brain description: GPU host (bare-metal) - PFI-ANA-ML2 BMC: href: https://10.250.250.50 icon: mdi-brain siteMonitor: https://10.250.250.50 - description: BMC (ana-ml2) + description: BMC (fv-ml1) - SFsrv-ANA: href: https://10.250.250.115:8006 icon: si-proxmox diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index a1437d5..ca26bc5 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -1,5 +1,5 @@ # LiteLLM gateway config — fronts the vLLM services on ana-ml2 -# (10.250.50.54) and logs every request + response so they're +# (10.251.50.54) and logs every request + response so they're # inspectable in the Logs UI at http://10.250.50.70:4000/ui. # # Deploys to /opt/docker/conf/litellm/config.yaml (mounted read-only @@ -22,7 +22,7 @@ model_list: # ⚠ ALIAS COLLISION — SEVERAL NAMES, ONE SET OF WEIGHTS # # As of 2026-08-23 these SEVEN aliases all resolve to the same backend -# (qwen3.8-27b-uncensored @ 10.250.50.54:8015): +# (qwen3.8-27b-uncensored @ 10.251.50.54:8015): # # chat-judge classifier gen image-judge # qwen-image-bench summarizer summarizer-large @@ -59,7 +59,7 @@ model_list: # - model_name: granite-4.1-8b # litellm_params: # model: hosted_vllm/granite-4.1-8b -# api_base: http://10.250.50.54:8004/v1 +# api_base: http://10.251.50.54:8004/v1 # api_key: os.environ/VLLM_API_KEY # temperature: 0 # model_info: @@ -71,7 +71,7 @@ model_list: - model_name: summarizer litellm_params: model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.250.50.54:8015/v1 + api_base: http://10.251.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY temperature: 0 extra_body: @@ -85,7 +85,7 @@ model_list: - model_name: classifier litellm_params: model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.250.50.54:8015/v1 + api_base: http://10.251.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY temperature: 0 extra_body: @@ -100,7 +100,7 @@ model_list: - model_name: summarizer-large litellm_params: model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.250.50.54:8015/v1 + api_base: http://10.251.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY presence_penalty: 1.5 temperature: 0.7 @@ -123,7 +123,7 @@ model_list: - model_name: qwen-image-bench litellm_params: model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.250.50.54:8015/v1 + api_base: http://10.251.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY temperature: 0 top_p: 1.0 @@ -137,7 +137,7 @@ model_list: - model_name: image-judge litellm_params: model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.250.50.54:8015/v1 + api_base: http://10.251.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY temperature: 0 top_p: 1.0 @@ -182,7 +182,7 @@ model_list: - model_name: gen litellm_params: model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.250.50.54:8015/v1 + api_base: http://10.251.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY presence_penalty: 1.5 temperature: 0.7 @@ -200,7 +200,7 @@ model_list: # Distinct served-name so a thinking-off `gen` request can't mutate this deployment's # enable_thinking (shared-config-mutation footgun). Same backend :8015, different model id. model: hosted_vllm/qwen3.8-27b-uncensored-thinking - api_base: http://10.250.50.54:8015/v1 + api_base: http://10.251.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY # CANONICAL Qwen3.8 THINKING sampling set (Qwen + unsloth "Best Practices" # §1, identical in both): temperature=1.0, top_p=0.95, top_k=20, @@ -248,7 +248,7 @@ model_list: - model_name: char-rp litellm_params: model: hosted_vllm/char-rp - api_base: http://10.250.50.54:8016/v1 + api_base: http://10.251.50.54:8016/v1 api_key: os.environ/VLLM_API_KEY temperature: 1.1 top_p: 0.95 @@ -274,7 +274,7 @@ model_list: - model_name: char-rp-reasoning litellm_params: model: hosted_vllm/char-rp-thinking - api_base: http://10.250.50.54:8016/v1 + api_base: http://10.251.50.54:8016/v1 api_key: os.environ/VLLM_API_KEY temperature: 1.1 top_p: 0.95 @@ -310,8 +310,8 @@ model_list: # character card both sit at ~0%. Fable-Fusion is Heretic-abliterated. # # ROLLBACK (restores Dark-Scarlett): - # ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe && sudo docker compose down' - # ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && sudo docker compose up -d' + # ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe && sudo docker compose down' + # ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && sudo docker compose up -d' # then revert this block to api_base :8018 / model hosted_vllm/char-rp-reasoning # and restart litellm (~52s). # @@ -337,7 +337,7 @@ model_list: - model_name: sec litellm_params: model: hosted_vllm/mog-sec-27b - api_base: http://10.250.50.54:8019/v1 + api_base: http://10.251.50.54:8019/v1 api_key: os.environ/VLLM_API_KEY temperature: 0.7 top_p: 0.8 @@ -356,7 +356,7 @@ model_list: - model_name: sec-reasoning litellm_params: model: hosted_vllm/mog-sec-27b-thinking - api_base: http://10.250.50.54:8019/v1 + api_base: http://10.251.50.54:8019/v1 api_key: os.environ/VLLM_API_KEY temperature: 1.0 top_p: 0.95 @@ -393,7 +393,7 @@ model_list: - model_name: qwen3-embedding litellm_params: model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B - api_base: http://10.250.50.54:8001/v1 + api_base: http://10.251.50.54:8001/v1 api_key: os.environ/VLLM_API_KEY model_info: mode: embedding @@ -423,7 +423,7 @@ model_list: - model_name: chat-judge litellm_params: model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.250.50.54:8015/v1 + api_base: http://10.251.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY temperature: 0 top_p: 1.0 @@ -438,7 +438,7 @@ model_list: - model_name: reranker litellm_params: model: hosted_vllm/BAAI/bge-reranker-v2-m3 - api_base: http://10.250.50.54:8013/v1 + api_base: http://10.251.50.54:8013/v1 api_key: os.environ/VLLM_API_KEY model_info: mode: rerank @@ -456,7 +456,7 @@ model_list: - model_name: coder-fast litellm_params: model: hosted_vllm/qwen2.5-coder-1.5b - api_base: http://10.250.50.54:8020/v1 + api_base: http://10.251.50.54:8020/v1 api_key: os.environ/VLLM_API_KEY model_info: mode: completion @@ -681,7 +681,7 @@ model_list: - model_name: erp-tune-v2 litellm_params: model: hosted_vllm/erp-tune-v2 - api_base: http://10.250.50.54:8098/v1 + api_base: http://10.251.50.54:8098/v1 api_key: os.environ/VLLM_API_KEY # gemma4-26b-a4b-it-base -> the UNMODIFIED upstream instruct release, bf16 @@ -743,7 +743,7 @@ model_list: - model_name: gemma4-26b-a4b-it-base litellm_params: model: hosted_vllm/gemma4-26b-a4b-it-base - api_base: http://10.250.50.54:8099/v1 + api_base: http://10.251.50.54:8099/v1 api_key: os.environ/VLLM_API_KEY # merged-run03 -> the in-house ERP/RP SFT, RUN 3, MERGED bf16 (:8099, vLLM, ana-ml2 @@ -848,7 +848,7 @@ model_list: # - model_name: merged-run03 # litellm_params: # model: hosted_vllm/merged-run03 - # api_base: http://10.250.50.54:8099/v1 + # api_base: http://10.251.50.54:8099/v1 # api_key: os.environ/VLLM_API_KEY # trial -> ERP/RP SFT RUN 7, NVFP4A16 (weight-only) quant of the merged LoRA, served on @@ -879,9 +879,10 @@ model_list: # # THIS GATEWAY IS THE SHARED-KEY SURFACE: `all-agents-local` reaches every model here, # in every session and project. Removing this alias does not remove the operator's - # access -- the direct endpoint http://10.250.50.54:8021/v1 is unaffected. + # access -- the direct endpoint http://10.251.50.54:8021/v1 is unaffected. # - # Same-site: seat and gateway are both at Anaheim (local hop, no mesh crossing). + # CROSS-SITE since 2026-09-12: gateway on ana-docker (Anaheim), seat on fv-ml1 +# (Fountain Valley) -- metro hop over the headscale mesh, ~6ms. # char-rp-fast — the MeroMero A4B MoE, on ana-ml2 GPU1 (:8021). Operator, 2026-09-10: # "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias @@ -919,7 +920,7 @@ model_list: - model_name: char-rp-fast litellm_params: model: hosted_vllm/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 - api_base: http://10.250.50.54:8021/v1 + api_base: http://10.251.50.54:8021/v1 api_key: os.environ/VLLM_API_KEY extra_body: min_p: 0.05 @@ -950,14 +951,14 @@ general_settings: maximum_spend_logs_retention_period: "7d" maximum_spend_logs_retention_interval: "1d" # scalar-judge → Skywork-Reward-V2 (scalar reward model; vLLM pooling on - # ana-ml2:8003). LiteLLM has no reward/pooling MODE, so this is a passthrough, + # fv-ml1:8003). LiteLLM has no reward/pooling MODE, so this is a passthrough, # not a model_list alias. Gateway-key-gated. Consumers POST the reward body to # /scalar-judge/ (e.g. /pooling or /classify), forwarded to :8003. # SWAP-SENSITIVE: a different reward model shifts the score scale, so consumers # must recalibrate thresholds after a backing swap. pass_through_endpoints: - path: "/scalar-judge" - target: "http://10.250.50.54:8003" + target: "http://10.251.50.54:8003" forward_headers: true include_subpath: true diff --git a/stacks/llama-swap/compose.yaml b/stacks/llama-swap/compose.yaml index 2cec60c..584f36d 100644 --- a/stacks/llama-swap/compose.yaml +++ b/stacks/llama-swap/compose.yaml @@ -2,7 +2,7 @@ # # Proxies OpenAI-compatible API requests to llama.cpp server instances # and swaps which model is loaded into VRAM per request. Runs on -# ana-ml2 using both GPUs dynamically (no explicit device pinning — +# fv-ml1 using both GPUs dynamically (no explicit device pinning — # llama-swap picks per-model-definition). # # Model definitions live in /opt/docker/conf/llama-swap/config.yaml on @@ -47,8 +47,8 @@ services: - homepage.group=AI Systems - homepage.name=llama-swap - homepage.icon=mdi-swap-horizontal - - homepage.description=GGUF model swapper (llama.cpp; ana-ml2) - - homepage.href=http://10.250.50.54:${LLAMA_SWAP_PORT} + - homepage.description=GGUF model swapper (llama.cpp; fv-ml1) + - homepage.href=http://10.251.50.54:${LLAMA_SWAP_PORT} networks: tnet: diff --git a/stacks/meromero-charrp/compose.yaml b/stacks/meromero-charrp/compose.yaml index 04308ac..46cc02b 100644 --- a/stacks/meromero-charrp/compose.yaml +++ b/stacks/meromero-charrp/compose.yaml @@ -1,5 +1,5 @@ # meromero-charrp — G4-MeroMero-v2-31B NVFP4A16 char-rp PROSE seat (non-thinking, MULTIMODAL) -# on ana-ml2 GPU0. Replaces the retired Magidonia GGUF seat (char-rp-gguf / llama-charrp). +# on fv-ml1 GPU0. Replaces the retired Magidonia GGUF seat (char-rp-gguf / llama-charrp). # # Gemma-4 dense but with SLIDING-WINDOW attention -> KV-efficient, serves the full native 256K # at ~2x concurrency. Vision enabled: preprocessor_config.json was materialized from @@ -93,8 +93,8 @@ services: - homepage.group=AI - Dormant - homepage.name=char-rp (MeroMero-v2 NVFP4) — rollback - homepage.icon=mdi-drama-masks - - homepage.description=Superseded 2026-08-24 by the Gemma-4 26B-A4B MoE seat; dense 31B NVFP4A16, kept for rollback (ana-ml2 GPU0) - - homepage.href=http://10.250.50.54:${MEROMERO_PORT:-8016}/docs + - homepage.description=Superseded 2026-08-24 by the Gemma-4 26B-A4B MoE seat; dense 31B NVFP4A16, kept for rollback (fv-ml1 GPU0) + - homepage.href=http://10.251.50.54:${MEROMERO_PORT:-8016}/docs networks: tnet: diff --git a/stacks/sglang/compose.yaml b/stacks/sglang/compose.yaml index cb44d97..51ae8da 100644 --- a/stacks/sglang/compose.yaml +++ b/stacks/sglang/compose.yaml @@ -1,4 +1,4 @@ -# SGLang — alternative LLM serving engine, stood up on ana-ml2 to bench +# SGLang — alternative LLM serving engine, stood up on fv-ml1 to bench # head-to-head against vLLM on the same model + hardware. # # Bench-oriented (not yet a permanent service): point it at the SAME checkpoint @@ -68,8 +68,8 @@ services: - homepage.group=AI Systems - homepage.name=SGLang (bench) - homepage.icon=mdi-speedometer - - homepage.description=SGLang serving — vLLM bench comparison (ana-ml2) - - homepage.href=http://10.250.50.54:${SGLANG_PORT} + - homepage.description=SGLang serving — vLLM bench comparison (fv-ml1) + - homepage.href=http://10.251.50.54:${SGLANG_PORT} networks: tnet: