diff --git a/CLAUDE.md b/CLAUDE.md index fecf3f0..552d124 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -199,7 +199,7 @@ Observed and standardized across servers: - **Restart policy:** `restart: unless-stopped` for daemons - **Homepage labels** on user-facing services. The dashboard runs on `esh-docker-vm` and reads the Docker API of **every** host in - `stacks/homepage/conf/docker.yaml` (ana-docker, ana-ml2, nh3-docker, + `stacks/homepage/conf/docker.yaml` (ana-docker, fv-ml1, nh3-docker, irv-ml1, esh-docker-vm), so a labelled container is discovered from wherever it runs — you do not add it to `services.yaml` as well. Doing both renders it twice. @@ -229,7 +229,7 @@ Observed and standardized across servers: | Name | IP | Site | Role | Details | |------|-----|------|------|---------| -| ana-ml2 | 10.250.50.54 | Anaheim (`10.250.0.0/16`) | GPU / AI inference (bare metal, dual RTX PRO 6000 Blackwell Max-Q, 96 GB each) | `servers/ana-ml2/README.md` | +| fv-ml1 | 10.251.50.54 | Fountain Valley (`10.251.0.0/16`) | GPU / AI inference (bare metal, **4× RTX PRO 6000 Blackwell Max-Q, 96 GB each = 391 GB VRAM**) | `servers/fv-ml1/README.md` | | irv-ml1 | 100.64.0.6 (headscale mesh); 10.6.110.50 (Irvine LAN, routed via mesh) | Irvine — via headscale mesh; wg0 lifeline (10.100.79.3) dormant post-cutover | GPU / AI inference (bare metal, RTX 3090 + RTX A6000, native stacks) | `servers/irv-ml1/README.md` | | ana-docker | 10.250.50.70 | Anaheim | General-purpose Docker host (non-GPU VM on pfi-pve) | `servers/ana-docker/README.md` | | pfi-ana-webhost | 10.250.50.52 | Anaheim | VM on pfi-pve (VMID 110) — web workload | `servers/pfi-ana-webhost/README.md` | @@ -257,7 +257,7 @@ Observed and standardized across servers: | esh-vm-db | 10.0.50.60 | ESH home lab | DB VM — PostgreSQL (paperless-ng) + MongoDB; bare-metal VM, no Docker | `servers/esh-vm-db/README.md` | **Placement rules:** -- GPU-required stacks → `ana-ml2` (primary, Anaheim) or `irv-ml1` +- GPU-required stacks → `fv-ml1` (primary, Fountain Valley) or `irv-ml1` (secondary, Irvine — bigger VRAM ceiling at 72 GB total). Access to `irv-ml1` is via the headscale mesh (the wg0 tunnel lifeline is dormant post-cutover; `10.100.79.3` no longer answers). diff --git a/README.md b/README.md index 4f40c93..382a050 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ See **[CLAUDE.md](CLAUDE.md)** for the full set of conventions and the rules Cla | Host | IP | Site | Role | |---|---|---|---| -| ana-ml2 | `10.250.50.54` | Anaheim (`10.250.0.0/16`) | GPU / AI inference (bare metal) | +| fv-ml1 | `10.251.50.54` | Fountain Valley (`10.251.0.0/16`) | GPU / AI inference (bare metal, 4× RTX PRO 6000 Blackwell Max-Q) | | ana-docker | `10.250.50.70` | Anaheim | General-purpose Docker + cross-site hubs (VM on pfi-pve) | | nh3-docker | `10.100.50.40` | NH3 (`10.100.0.0/16`) | General-purpose Docker (VM on nh3-pve) | | esh-docker-vm | `10.0.50.45` | ESH home lab (`esteban.net`) | Home-lab Docker (VM on esh-pve, non-PFI scope) | @@ -64,7 +64,7 @@ Per-host snapshots of the running system live under `servers//system-detai ## Current stacks -**GPU (ana-ml2):** +**GPU (fv-ml1):** - `llama-swap` — GGUF model swapper via llama.cpp (port 9292) - `vllm` — embeddings (8001) + reranker (8002) + Skywork reward classifier (8003) via vLLM @@ -76,7 +76,7 @@ Per-host snapshots of the running system live under `servers//system-detai - Fleet services: `beszel` (metrics hub, port 8090), `dozzle-hub` (log viewer, 8088), `backrest` (restic UI, 9898) - Backup target: `rest-server-ana` on port 8000 -**GPU ana-ml2 (non-canonical for now):** +**GPU fv-ml1 (non-canonical for now):** - `comfyui`, `kokoro`, `parakeet`, `vibevoice` alongside the canonical `llama-swap` + `vllm` **NH3 (nh3-docker):** @@ -170,7 +170,7 @@ Goal: **every Docker host + configs + every database** covered, not just VM imag |---|---| | VM-level (Proxmox vzdump) | ✅ All running guests covered across pfi-pve / nh3-pve / esh-pve-nas; esh-pve has VMID 108 uncovered | | ana-docker restic (host files + DBs) | ✅ `configs/restic/ana-docker/` with pre-backup hooks for synapse / seafile / vaultwarden-pg / gitea (native dump) / openwebui | -| ana-ml2 restic | ✅ `configs/restic/ana-ml2/` — bare-metal host files (no DB hooks needed) | +| fv-ml1 restic | ✅ `configs/restic/fv-ml1/` — bare-metal host files (no DB hooks needed) | | nh3-docker restic | ✅ Light — no DB hooks needed | | esh-docker-vm restic | ✅ With DB hooks for paperless-postgres (external), home-assistant + pgadmin + uptime-kuma (host-side sqlite3), calibre-web-automated (in-container sqlite3) | | vm-esh-nas restic | ✅ Light — NFS mounts explicitly excluded | diff --git a/dns/internal.yaml b/dns/internal.yaml index 85009aa..6375ed6 100644 --- a/dns/internal.yaml +++ b/dns/internal.yaml @@ -54,7 +54,8 @@ sites: hosts: # ---- ana: Anaheim colo ---- - {name: ana-docker, site: ana, v4: 10.250.50.70, note: general-purpose docker host} - - {name: ana-ml2, site: ana, v4: 10.250.50.54, note: GPU inference, dual RTX PRO 6000} + # ---- fv: Fountain Valley colo (piggyback DNS until FV has its own resolver) ---- + - {name: fv-ml1, site: fv, v4: 10.251.50.54, note: GPU inference, 4x RTX PRO 6000 Blackwell Max-Q (ex ana-ml2, relocated 2026-09-12)} - {name: ana-nas, site: ana, v4: 10.250.50.50, note: CT109 on pfi-pve — NFS/SMB} - {name: ana-filebot, site: ana, v4: 10.250.50.53, note: file-task automation} - {name: ana-wg, site: ana, v4: 10.250.50.252, note: WireGuard host} @@ -67,7 +68,8 @@ hosts: - {name: pfi-pve, site: ana, v4: 10.250.250.31, note: Proxmox hypervisor} - {name: ana-gw, site: ana, v4: 10.250.0.1, note: FortiGate-80F edge} - {name: pfi-pve-idrac, site: ana, v4: 10.250.250.30, note: iDRAC — OOB for pfi-pve} - - {name: ana-ml2-bmc, site: ana, v4: 10.250.250.50, note: BMC for ana-ml2} + - {name: fv-ml1-bmc, site: fv, v4: 10.251.250.50, note: BMC for fv-ml1 (Supermicro/ATEN; 802.1q VLAN tag must stay DISABLED)} + - {name: fv-gw, site: fv, v4: 10.251.50.1, note: OPNsense gateway vb-gateway, mesh subnet-router for 10.251.0.0/16; public 172.83.89.66 = fv.phasefinal.com} # SureFire tenant hardware — PFI-managed under the hosting agreement. - {name: sfsrv-ana, site: ana, v4: 10.250.250.115, note: SureFire tenant hypervisor} - {name: sf-ana-container, site: ana, v4: 10.250.150.100, note: SureFire tenant container host} @@ -120,4 +122,4 @@ aliases: - {name: booth, site: nh3, target: nh3-dev, note: The Booth :8090} - {name: wherethef, site: nh3, target: nh3-dev, note: WhereTF :8093} - {name: homepage, site: esh, target: esh-docker-vm, note: fleet dashboard :5100} - - {name: scriberr, site: ana, target: ana-ml2, note: transcription + diarization :8080 (GPU1)} + - {name: scriberr, site: fv, target: fv-ml1, note: transcription + diarization :8080 (GPU1)} diff --git a/docs/pfi/docker-stack.md b/docs/pfi/docker-stack.md index d9f0475..df43e91 100644 --- a/docs/pfi/docker-stack.md +++ b/docs/pfi/docker-stack.md @@ -131,7 +131,7 @@ group memberships) is in [`stacks/llama-swap/conf/config.yaml`](../../stacks/llama-swap/conf/config.yaml). Don't mirror the table here — it drifted last time and immediately went stale. Live HTTP listing: -`curl http://10.250.50.54:9292/v1/models`. +`curl http://10.251.50.54:9292/v1/models`. **Model Groups**: diff --git a/docs/pfi/proxmox-vms.md b/docs/pfi/proxmox-vms.md index e57e199..c29659e 100644 --- a/docs/pfi/proxmox-vms.md +++ b/docs/pfi/proxmox-vms.md @@ -73,5 +73,5 @@ Reverse this for a graceful host shutdown (done 2026-09-05 for the PERC swap). ## Not on this hypervisor -`ana-ml2` (10.250.50.54) is bare-metal Supermicro; `sfsrv-ana` (10.250.250.115) is the +`fv-ml1` (10.251.50.54) is bare-metal Supermicro; `sfsrv-ana` (10.250.250.115) is the SureFire client's own Dell R630; `pfi-gx10` is at NH3. diff --git a/docs/pfi/reranker-selection-ledger.md b/docs/pfi/reranker-selection-ledger.md index 992681b..909d4f9 100644 --- a/docs/pfi/reranker-selection-ledger.md +++ b/docs/pfi/reranker-selection-ledger.md @@ -18,9 +18,9 @@ implementation. **CLEARED (green):** - Execute Brokkr's reranker-selection recommendations unattended. -- Bring **down the prod reranker** at `ana-ml2:8002` (qwen3-reranker-0.6B) — +- Bring **down the prod reranker** at `fv-ml1:8002` (qwen3-reranker-0.6B) — **temporarily OR permanently**. -- Down **ONE** of the RP (roleplay) seats on ana-ml2 **temporarily** to free +- Down **ONE** of the RP (roleplay) seats on fv-ml1 **temporarily** to free GPU/VRAM for testing. - Temporarily clear space for the smoke/bench. - Pull models, stand up side-port vLLM benches, run the harness — whatever the @@ -30,7 +30,7 @@ implementation. - **NO permanent deletion of anything** (no `rm`/`docker volume rm`/model-weight deletion/data destruction). Downing ≠ deleting. - **NO taking anything else offline** beyond (a) the prod reranker and (b) ONE - ana-ml2 RP seat. (Not granite/embed/reward/coder/gen/a second RP seat/muninn/etc.) + fv-ml1 RP seat. (Not granite/embed/reward/coder/gen/a second RP seat/muninn/etc.) - **NO rebooting machines.** **Process:** accumulate assumptions here; operator reverses at the end. @@ -54,7 +54,7 @@ implementation. ## Current board at handoff -- **Prod reranker:** `ana-ml2:8002` = `vllm-rerank` (Qwen/Qwen3-Reranker-0.6B), +- **Prod reranker:** `fv-ml1:8002` = `vllm-rerank` (Qwen/Qwen3-Reranker-0.6B), reverted to baseline `classifier_from_token:["no","yes"]`, healthy. Compose: `/opt/docker/compose/vllm/compose.yaml` (canonical mirror `stacks/vllm/compose.yaml`). Gateway alias `reranker`/`qwen3-reranker` → @@ -75,7 +75,7 @@ implementation. worldtree-dev signals the lever run is done). - **Dual query-set requirement (hard):** score bare-name anchor queries AND question-shaped; bar = recovering the name-lookup class. -- **VRAM:** 4B ≈ 4–5 GB fp8, 8B ≈ 9 GB; ana-ml2 Blackwell has headroom. +- **VRAM:** 4B ≈ 4–5 GB fp8, 8B ≈ 9 GB; fv-ml1 Blackwell has headroom. --- @@ -84,12 +84,12 @@ implementation. ### 2026-08-06 — A2 brought up (Brokkr thread 01KZBSTSJA…) - **Backend:** `vllm-rerank-a2` — standalone `docker run` (NOT in the vllm compose - stack), on ana-ml2 **GPU1**, host port **:8012** → container 8000. Image + stack), on fv-ml1 **GPU1**, host port **:8012** → container 8000. Image `vllm/vllm-openai:latest` (=0.24.0). Args: model `tomaarsen/Qwen3-Reranker-0.6B-seq-cls`, `--runner pooling`, `--gpu-memory-utilization 0.03`, `--max-model-len 8192`, `--dtype auto`, `--restart no`. Native `Qwen3ForSequenceClassification` — NO hf-overrides. Routes /rerank /score /classify. -- **Gateway alias:** `reranker-a2-qwen3-seqcls` → `http://10.250.50.54:8012/v1`, +- **Gateway alias:** `reranker-a2-qwen3-seqcls` → `http://10.251.50.54:8012/v1`, mode rerank. Added via LiteLLM **`/model/new`** (DB-backed, `store_model_in_db:true`) — **no gateway restart** (respects the "nothing else offline" line). Verified 200 through the gateway. @@ -106,7 +106,7 @@ implementation. - D4 — did NOT down any RP seat (A2 is 0.6B / 3.5 GB; no VRAM pressure). **Cleanup for A2 (run at end / on reversal):** -- `ssh infra-ops@10.250.50.54 'sudo docker stop vllm-rerank-a2 && sudo docker rm vllm-rerank-a2'` +- `ssh infra-ops@10.251.50.54 'sudo docker stop vllm-rerank-a2 && sudo docker rm vllm-rerank-a2'` - Delete gateway alias: `POST /model/delete {"id": }` (id via `/model/info?model_name=reranker-a2-qwen3-seqcls`), infra-ops admin key. (DB-persisted, so it survives a restart — must be explicitly deleted.) - No weights deleted (red line); HF cache under /tank/aimodels/huggingface retains the 0.6B-seq-cls download. @@ -114,11 +114,11 @@ implementation. ### 2026-08-06 — A3 + A4 pre-staged (Brokkr said pre-stage in parallel, hold A5) -- **A3** `vllm-rerank-a3` — ana-ml2 GPU1 :8013, `BAAI/bge-reranker-v2-m3` +- **A3** `vllm-rerank-a3` — fv-ml1 GPU1 :8013, `BAAI/bge-reranker-v2-m3` (XLMRobertaForSequenceClassification), same run pattern, util 0.03. VRAM ≈ **2.3 GB**. Latency (20-doc, ~1500-char, shared GPU1): single p50 **105 ms**; 8-conc p50 214 ms, ~34 req/s. Gateway alias `reranker-a3-bge-v2-m3` via /model/new (200, verified). -- **A4** `vllm-rerank-a4` — ana-ml2 GPU1 :8014, `Alibaba-NLP/gte-reranker-modernbert-base` +- **A4** `vllm-rerank-a4` — fv-ml1 GPU1 :8014, `Alibaba-NLP/gte-reranker-modernbert-base` (ModernBertForSequenceClassification), util 0.02. VRAM ≈ **1.4 GB**. Latency: single p50 **102 ms**; 8-conc p50 153 ms, ~51 req/s. Gateway alias `reranker-a4-gte-modernbert` via /model/new (200, verified). @@ -129,7 +129,7 @@ implementation. If A5 (4B, ~4–5 GB) is greenlit: fits GPU1 tight or GPU0 (~9 GB free) — no RP-seat downing expected. **Cleanup for A3/A4 (same pattern as A2):** `docker stop/rm vllm-rerank-a3 vllm-rerank-a4` -on ana-ml2; `/model/delete` the two aliases (DB-persisted); weights retained in HF cache. +on fv-ml1; `/model/delete` the two aliases (DB-persisted); weights retained in HF cache. ### 2026-08-06 — A2 verdict (Brokkr full grid): training-prior confirmed @@ -202,7 +202,7 @@ on ana-ml2; `/model/delete` the two aliases (DB-persisted); weights retained in Brokkr signals the bake-off is closed (`docker stop/rm` + `/model/delete`). - **A4 (`vllm-rerank-a4` :8014)** + alias — KEEP for now (Brokkr's documented throughput fallback). - **A3 (`vllm-rerank-a3` :8013)** — now PRODUCTION (backs the `reranker` alias). Hardened - 2026-08-06: `docker update --restart unless-stopped` (survives ana-ml2 reboot, no recreate). + 2026-08-06: `docker update --restart unless-stopped` (survives fv-ml1 reboot, no recreate). A4 given the same. **Remaining follow-up (not urgent): promote A3 from throwaway `docker run` to a canonical compose service** (`stacks/vllm/`) for config-managed consistency — a recreate, so do it in a window since it briefly drops `reranker`. diff --git a/docs/runbooks/disaster-recovery.md b/docs/runbooks/disaster-recovery.md index 372912e..2eea2c6 100644 --- a/docs/runbooks/disaster-recovery.md +++ b/docs/runbooks/disaster-recovery.md @@ -37,7 +37,7 @@ pfi-pve during morning backup window. Backup suspected as trigger **Blast radius:** - ~~pfi-postgres (VM 105) — PGDATA on `/mnt/db`~~ — **migrated to local disk 2026-04-23**. vaultwarden, gitea, paperless-ng, zammad no longer cascade on ana-nas outage. Left in history for the recovery pre-migration. -- ana-docker rest-server-ana — repo data on `/mnt/backup` → all ana-side restic clients fail (ana-docker, ana-ml2, esh-docker-vm, vm-esh-nas) +- ana-docker rest-server-ana — repo data on `/mnt/backup` → all ana-side restic clients fail (ana-docker, fv-ml1, esh-docker-vm, vm-esh-nas) - PBS-ANA datastore — NFS-backed on `/mnt/backup/pbs-ana` → fleet vzdumps fail, PBS-NH3 sync fails - ana-docker NFS mounts for `/mnt/docker`, `/mnt/compose`, `/mnt/pve-VMStorage` if used → various stack misbehavior @@ -207,12 +207,12 @@ Essentially **all Anaheim primary services** go offline. Because ana-nas lives h ## Tier 4 — Specialty workloads -### ana-ml2 (bare metal Supermicro, 10.250.50.54, BMC 10.250.250.50) +### fv-ml1 (bare metal Supermicro, 10.251.50.54, BMC 10.251.250.50) **Blast radius:** AI inference services (llama-swap, vllm). Consumer-facing chat/embedding/reward-scoring endpoints fail. **Recovery:** -1. Check OS via SSH. If unresponsive, BMC console at . +1. Check OS via SSH. If unresponsive, BMC console at . 2. If hardware issue: BMC logs, power cycle via IPMI, check GPU health (`nvidia-smi`). 3. Docker stacks auto-start via compose `restart: unless-stopped`. diff --git a/docs/runbooks/fv-ml1-cutover.md b/docs/runbooks/fv-ml1-cutover.md index cd99671..b1a04f8 100644 --- a/docs/runbooks/fv-ml1-cutover.md +++ b/docs/runbooks/fv-ml1-cutover.md @@ -49,7 +49,7 @@ path the fleet lacked on 2026-09-11. Do NOT tie the mesh node to fv-ml1 itself. - [ ] `dns/internal.yaml` — add the two records below (piggyback; NOT a `sites:` entry), comment out the old `ana-ml2` / `ana-ml2-bmc` lines. Then `scripts/dns-sync.py --dry-run` → apply. ```yaml # ---- fv: Fountain Valley colo (piggyback DNS until FV has its own resolver) ---- - - {name: fv-ml1, site: fv, v4: 10.251.50.54, note: GPU inference, dual RTX PRO 6000 (ex ana-ml2, relocated 2026-09)} + - {name: fv-ml1, site: fv, v4: 10.251.50.54, note: GPU inference, 4x RTX PRO 6000 Blackwell Max-Q (ex ana-ml2, relocated 2026-09)} - {name: fv-ml1-bmc, site: fv, v4: 10.251.250.50, note: BMC for fv-ml1} ``` - [ ] mesh route to 10.251.0.0/16 approved in headscale (OPNsense subnet-router). diff --git a/playbooks/ana-ml2-mesh-routes.yaml b/playbooks/fv-ml1-mesh-routes.yaml similarity index 100% rename from playbooks/ana-ml2-mesh-routes.yaml rename to playbooks/fv-ml1-mesh-routes.yaml diff --git a/playbooks/ana-ml2-overcommit-memory.yaml b/playbooks/fv-ml1-overcommit-memory.yaml similarity index 100% rename from playbooks/ana-ml2-overcommit-memory.yaml rename to playbooks/fv-ml1-overcommit-memory.yaml diff --git a/playbooks/ana-ml2-pool-health.yaml b/playbooks/fv-ml1-pool-health.yaml similarity index 100% rename from playbooks/ana-ml2-pool-health.yaml rename to playbooks/fv-ml1-pool-health.yaml diff --git a/playbooks/ana-ml2-training-window-close.yaml b/playbooks/fv-ml1-training-window-close.yaml similarity index 100% rename from playbooks/ana-ml2-training-window-close.yaml rename to playbooks/fv-ml1-training-window-close.yaml diff --git a/playbooks/ana-ml2-training-window-open.yaml b/playbooks/fv-ml1-training-window-open.yaml similarity index 100% rename from playbooks/ana-ml2-training-window-open.yaml rename to playbooks/fv-ml1-training-window-open.yaml diff --git a/servers/ana-ml2/ssh-target b/servers/ana-ml2/ssh-target deleted file mode 100644 index d1da030..0000000 --- a/servers/ana-ml2/ssh-target +++ /dev/null @@ -1 +0,0 @@ -10.250.50.54 diff --git a/servers/ana-ml2/README.md b/servers/fv-ml1/README.md similarity index 94% rename from servers/ana-ml2/README.md rename to servers/fv-ml1/README.md index 394b669..95b47d8 100644 --- a/servers/ana-ml2/README.md +++ b/servers/fv-ml1/README.md @@ -1,13 +1,13 @@ -# ana-ml2 +# fv-ml1 Primary AI inference host for PFI. ## Network -- **LAN IP:** 10.250.50.54 (in-band, OS-side) -- **BMC (OOB):** 10.250.250.50 — Supermicro IPMI web UI - at (homepage card: *PFI-ANA-ML2 BMC*) -- **SSH:** standard port 22 on 10.250.50.54 +- **LAN IP:** 10.251.50.54 (in-band, OS-side) +- **BMC (OOB):** 10.251.250.50 — Supermicro IPMI web UI + at (homepage card: *PFI-ANA-ML2 BMC*) +- **SSH:** standard port 22 on 10.251.50.54 ## Hardware @@ -54,9 +54,9 @@ history, not an active fault, and the 2 CKSUM were the stale-block artefact of t replace on growth** (`zpool replace tank nvme7n1 `; any PM1725b 1.6 TB or larger). Slot 0-5 itself deserves a reseat / cable check at the next hands-on visit — a bay that dropped a drive for 4½ months is the likelier fault than the -drive. Playbook: `playbooks/ana-ml2-pool-health.yaml` (idempotent; rerunning is a +drive. Playbook: `playbooks/fv-ml1-pool-health.yaml` (idempotent; rerunning is a no-op). ⚠ **Nothing alerts on this** — see the open follow-up in -`persistent-memory.d/2026-09-09-ana-ml2-pool-actions-done.md`. +`persistent-memory.d/2026-09-09-fv-ml1-pool-actions-done.md`. ## Key paths @@ -119,7 +119,7 @@ seats, safe to leave: `mistral-medium-3.5`, `mistral-small-4(-heretic)`, ## Refresh state ```bash -scripts/refresh-server-info.sh ana-ml2 +scripts/refresh-server-info.sh fv-ml1 ``` Latest snapshot: `system-details.txt` (regenerate as needed). diff --git a/servers/fv-ml1/ssh-target b/servers/fv-ml1/ssh-target new file mode 100644 index 0000000..924d4c6 --- /dev/null +++ b/servers/fv-ml1/ssh-target @@ -0,0 +1 @@ +10.251.50.54 diff --git a/servers/ana-ml2/system-details.txt b/servers/fv-ml1/system-details.txt similarity index 100% rename from servers/ana-ml2/system-details.txt rename to servers/fv-ml1/system-details.txt diff --git a/servers/pfi-pve/README.md b/servers/pfi-pve/README.md index 25598d6..09e1989 100644 --- a/servers/pfi-pve/README.md +++ b/servers/pfi-pve/README.md @@ -40,7 +40,7 @@ hypervisor): - **CT 112** — `ana-filebot` (file-task automation) — `servers/ana-filebot/` - **CT 113** — `ana-wg` (WireGuard VPN) — `servers/ana-wg/` -`ana-ml2` (`10.250.50.54`) is a separate bare-metal Supermicro box, +`fv-ml1` (`10.251.50.54`) is a separate bare-metal Supermicro box, not on this hypervisor. `sfsrv-ana` (`10.250.250.115`) runs on a separate Dell R630, also not on this hypervisor. @@ -54,4 +54,4 @@ The snapshot captures host/hardware/filesystem/network/ports but **not** VM-leve ## Placement rule -This is a hypervisor, not a target for Docker stacks. Add new workloads by standing up VMs here (or on ana-docker / ana-ml2, whichever fits) rather than deploying containers directly on the PVE host. +This is a hypervisor, not a target for Docker stacks. Add new workloads by standing up VMs here (or on ana-docker / fv-ml1, whichever fits) rather than deploying containers directly on the PVE host. diff --git a/stacks/beszel/README.md b/stacks/beszel/README.md index 678473c..bb773d5 100644 --- a/stacks/beszel/README.md +++ b/stacks/beszel/README.md @@ -2,7 +2,7 @@ Priority 2 completed 2026-09-11: pfi-postgres, esh-vm-db, pbs-ana and pbs-nh3 added with 16 alerts. Live samples verified, including both PBS datastores. -Fleet 17/18 up (known ana-ml2 outage). See `configs/beszel-agent/PRIORITY2.md`. +Fleet 17/18 up (known fv-ml1 outage). See `configs/beszel-agent/PRIORITY2.md`. Hub: http://10.250.50.70:8090 (ana-docker), version 0.18.7 at the 2026-09-10 wiring. The hub also retains corviduo-dev's existing registration. @@ -12,12 +12,12 @@ esh-pve and esh-pve-nas); 13 systems total and 20 additional alert rules. See `configs/beszel-agent/README.md`. Subsequently nh3-nas was installed via DSM Docker and registered with four alerts (14 registrations total). Recovery verification at 2026-09-12 01:51Z: all six priority-1 hosts up, Synology live -filesystem samples verified; fleet 13/14 up with known ana-ml2 outage. +filesystem samples verified; fleet 13/14 up with known fv-ml1 outage. | Host | Compose directory under `/opt/docker/compose/` | Additional filesystems | |---|---|---| | ana-docker | beszel | /mnt/backup | -| ana-ml2 | beszel-agent-ana | /tank, /home | +| fv-ml1 | beszel-agent-ana | /tank, /home | | nh3-docker | beszel-agent-nh3 | none | | esh-docker-vm (hub name esh-vm-docker) | beszel-agent-esh | /mnt/backup, /mnt/books | | irv-ml1 | beszel-agent-irv | /worktank, /storetank, /mnt/smithy | @@ -37,14 +37,14 @@ and `DEPLOY_SUDO=1` for root-owned directories. Example: ```sh DEPLOY_SUDO=1 DEPLOY_DEST_STACK=beszel-agent-ana \ - scripts/deploy-stack.sh infra-ops@10.250.50.54 beszel --compose + scripts/deploy-stack.sh infra-ops@10.251.50.54 beszel --compose ``` Agents use host-specific overrides selected by live `.env`: ```dotenv COMPOSE_PROFILES=agent -COMPOSE_FILE=compose.yaml:hosts/ana-ml2.yaml +COMPOSE_FILE=compose.yaml:hosts/fv-ml1.yaml BESZEL_EXTRA_FS=/extra-filesystems/tank,/extra-filesystems/home ``` @@ -71,7 +71,7 @@ hub public key. Never copy live tokens into version control. ## GPU telemetry -ana-ml2 and irv-ml1 use `henrygd/beszel-agent-nvidia:0.18.7` with NVIDIA +fv-ml1 and irv-ml1 use `henrygd/beszel-agent-nvidia:0.18.7` with NVIDIA `utility` access to all GPUs. Both hosts already have NVIDIA Container Toolkit. This collects per-card utilization, VRAM, temperature, and power draw without changing the serving containers or GPU power limits. Verified hub samples @@ -111,7 +111,7 @@ Thirty rules cover the seven hosts above under the existing operator user: | CPU | >95% | 15 minutes | | Memory | >90% | 10 minutes | | Offline | down | 2 minutes | -| Temperature (ana-ml2 and irv-ml1) | >85 C | 5 minutes | +| Temperature (fv-ml1 and irv-ml1) | >85 C | 5 minutes | CPU thresholds are sustained-load warnings; expected long-running compute may need tuning. GPU utilization alone is not an alarm because busy GPUs are normal. @@ -125,7 +125,7 @@ The bridge at `services/beszel-althing/` forwards through `postbox` to the was replaced with this verified route. Miranda is a later cutover, not enabled. See that service's README for operation and recipient changes. -Acceptance on 2026-09-10: ana-ml2 Disk was temporarily lowered to 1%/1 minute; +Acceptance on 2026-09-10: fv-ml1 Disk was temporarily lowered to 1%/1 minute; the real alert reached althing at 15:29:45Z, thread `01M25Z0WFDJM92GPTJQF769HJ7`. The threshold was then restored to 85%/5 minutes. Verification uses `postbox thread`, which does not consume the inbox. diff --git a/stacks/dockge/.env.example b/stacks/dockge/.env.example index 5181261..f538b64 100644 --- a/stacks/dockge/.env.example +++ b/stacks/dockge/.env.example @@ -10,13 +10,13 @@ DOCKGE_VERSION=latest DOCKGE_PORT=5001 # Human-readable host name for the homepage card, e.g. -# ana-docker, nh3-docker, esh-docker-vm, vm-esh-nas, ana-ml2 +# ana-docker, nh3-docker, esh-docker-vm, vm-esh-nas, fv-ml1 DOCKGE_HOST_LABEL= # IP (or hostname) the browser uses to reach this Dockge. Used in # homepage.href — must be reachable from wherever you view homepage. # ana-docker → 10.250.50.70 -# ana-ml2 → 10.250.50.54 +# fv-ml1 → 10.251.50.54 # nh3-docker → 10.100.50.40 # esh-docker-vm → 10.0.50.45 # vm-esh-nas → 10.0.50.154 diff --git a/stacks/erp-seat/compose.yaml b/stacks/erp-seat/compose.yaml index 57ce76b..94752b5 100644 --- a/stacks/erp-seat/compose.yaml +++ b/stacks/erp-seat/compose.yaml @@ -1,4 +1,4 @@ -# erp-seat — the RP seat on ana-ml2 GPU1. Serves the **MeroMero A4B MoE** NVFP4A16 quant +# erp-seat — the RP seat on fv-ml1 GPU1. Serves the **MeroMero A4B MoE** NVFP4A16 quant # (G4-MeroMero-26B-A4B-it-uncensored-heretic) behind the gateway alias `char-rp-fast`. # # ⚠ THE STACK NAME IS HISTORICAL. It served Pfish-6 (the run-6 ERP-tune LoRA merge) until @@ -109,7 +109,7 @@ services: # SPEED (2026-09-09). ⚠ NVFP4A16 is WEIGHT-ONLY: activations stay 16-bit, so # there is no FP4 GEMM to issue and the FlashInfer/CUTLASS NVFP4 MoE kernels # do not apply — vLLM's "your GPU does not have native support for FP4" - # warning is about the SCHEME, not the card. ana-ml2 is sm_120 and DOES have + # warning is about the SCHEME, not the card. fv-ml1 is sm_120 and DOES have # native FP4; a W4A4 build would use it, at a fidelity cost the operator # already settled against for this seat. Marlin is the correct kernel here. # PROVEN, not assumed: forcing moe_backend=flashinfer_cutedsl fails at engine @@ -151,8 +151,8 @@ services: - homepage.group=AI - Inference - homepage.name=char-rp-fast (MeroMero 26B-A4B, NVFP4A16 MoE) - homepage.icon=mdi-fire - - homepage.description=MeroMero A4B abliterated RP seat, NVFP4A16 weight-only, vision intact (ana-ml2 GPU1) - - homepage.href=http://10.250.50.54:${ERP_PORT:-8021}/docs + - homepage.description=MeroMero A4B abliterated RP seat, NVFP4A16 weight-only, vision intact (fv-ml1 GPU1) + - homepage.href=http://10.251.50.54:${ERP_PORT:-8021}/docs networks: tnet: diff --git a/stacks/gemma4-charrp/README.md b/stacks/gemma4-charrp/README.md index f584ced..1507299 100644 --- a/stacks/gemma4-charrp/README.md +++ b/stacks/gemma4-charrp/README.md @@ -1,11 +1,11 @@ -# gemma4-charrp — the char-rp seat (ana-ml2 GPU0) +# gemma4-charrp — the char-rp seat (fv-ml1 GPU0) `google/gemma-4-26B-A4B-it`, NVFP4, serving both halves of the char-rp pair on `:8016`. Replaced the dense `G4-MeroMero-v2-31B-NVFP4A16` seat on 2026-08-24. ``` -char-rp non-thinking -> http://10.250.50.54:8016/v1 -char-rp-reasoning thinking -> http://10.250.50.54:8016/v1 +char-rp non-thinking -> http://10.251.50.54:8016/v1 +char-rp-reasoning thinking -> http://10.251.50.54:8016/v1 ``` Two LiteLLM aliases, **one backend**. They are not two seats — this trips people diff --git a/stacks/gemma4-charrp/compose.yaml b/stacks/gemma4-charrp/compose.yaml index e335b89..9f5f8e8 100644 --- a/stacks/gemma4-charrp/compose.yaml +++ b/stacks/gemma4-charrp/compose.yaml @@ -1,4 +1,4 @@ -# gemma4-charrp — google/gemma-4-26B-A4B-it NVFP4 char-rp seat on ana-ml2 GPU0. +# gemma4-charrp — google/gemma-4-26B-A4B-it NVFP4 char-rp seat on fv-ml1 GPU0. # # Straight-across replacement for `meromero-charrp` (operator, 2026-08-24): same # port, same served-model-names, so every gateway route and consumer is @@ -46,7 +46,7 @@ services: - --quantization - compressed-tensors # UNCHANGED FROM THE OUTGOING SEAT, ON PURPOSE. Both names are live - # LiteLLM routes at http://10.250.50.54:8016/v1 — `char-rp` -> + # LiteLLM routes at http://10.251.50.54:8016/v1 — `char-rp` -> # hosted_vllm/char-rp and `char-rp-reasoning` -> hosted_vllm/char-rp-thinking. # They are two aliases onto ONE seat, not two seats. Renaming either # breaks the gateway silently. @@ -128,8 +128,8 @@ services: - homepage.group=AI - Inference - homepage.name=char-rp (Gemma-4 26B-A4B NVFP4, MoE) - homepage.icon=mdi-drama-masks - - homepage.description=gemma-4-26B-A4B-it NVFP4 MoE prose+CoT seat, 256K (ana-ml2 GPU0) - - homepage.href=http://10.250.50.54:${GEMMA4_PORT:-8016}/docs + - homepage.description=gemma-4-26B-A4B-it NVFP4 MoE prose+CoT seat, 256K (fv-ml1 GPU0) + - homepage.href=http://10.251.50.54:${GEMMA4_PORT:-8016}/docs networks: tnet: diff --git a/stacks/heretic2-charrp-reasoning/README.md b/stacks/heretic2-charrp-reasoning/README.md index 78fd58a..55fd844 100644 --- a/stacks/heretic2-charrp-reasoning/README.md +++ b/stacks/heretic2-charrp-reasoning/README.md @@ -1,12 +1,12 @@ -# heretic2-charrp-reasoning — NVFP4 + native MTP reasoning seat (ana-ml2) +# heretic2-charrp-reasoning — NVFP4 + native MTP reasoning seat (fv-ml1) The `char-rp-reasoning` seat: NEO-CODE Heretic2 27B quantized to modelopt NVFP4 -with a grafted BF16 MTP head, served by vLLM on **ana-ml2 GPU0**, port **8018**. +with a grafted BF16 MTP head, served by vLLM on **fv-ml1 GPU0**, port **8018**. Replaces the retired GGUF seat (`llama-charrp-reasoning`) at roughly **77 tok/s (~1.3x)** via `qwen3_5_mtp` speculative decode. - **Served model name:** `char-rp-reasoning` — the gateway alias consumers use -- **Endpoint:** `http://10.250.50.54:8018` (`/docs` for the card link) +- **Endpoint:** `http://10.251.50.54:8018` (`/docs` for the card link) ## ⚠️ It does not boot without the MTP workaround @@ -39,8 +39,8 @@ OOM the neighbours, not just this container.** Tunables are in `.env`; see ## Deploy ```bash -scripts/deploy-stack.sh ana-ml2 heretic2-charrp-reasoning -ssh infra-ops@10.250.50.54 \ +scripts/deploy-stack.sh fv-ml1 heretic2-charrp-reasoning +ssh infra-ops@10.251.50.54 \ 'cd /opt/docker/compose/heretic2-charrp-reasoning && sudo docker compose up -d' ``` diff --git a/stacks/homepage/README.md b/stacks/homepage/README.md index b2f8efa..8af4ff4 100644 --- a/stacks/homepage/README.md +++ b/stacks/homepage/README.md @@ -18,7 +18,7 @@ Two paths, and mixing them is the classic failure: `homepage.group=` / `homepage.name=` / `homepage.icon=` / `homepage.description=` / `homepage.href=` labels and appears automatically. `conf/docker.yaml` wires **five** engines over plaintext `:2375`: esh-docker-vm, ana-docker, - nh3-docker, ana-ml2, irv-ml1. + nh3-docker, fv-ml1, irv-ml1. 2. **Manual entries in `conf/services.yaml`** — for anything that is not a labelled container on one of those five hosts: hardware, BMCs, hypervisors, printers, and user-level systemd services (The Booth, Voice Design Studio). @@ -63,7 +63,7 @@ scripts/elway infra-ops@10.250.50.70 --playbook playbooks/homepage-regroup-ana-d scripts/elway infra-ops@10.0.50.45 --playbook playbooks/homepage-regroup-esh-docker-vm.yaml scripts/elway infra-ops@10.100.50.40 --playbook playbooks/homepage-regroup-nh3-docker.yaml scripts/elway infra-ops@10.100.79.3 --playbook playbooks/homepage-regroup-irv-ml1.yaml -scripts/elway infra-ops@10.250.50.54 --playbook playbooks/homepage-regroup-ana-ml2.yaml +scripts/elway infra-ops@10.251.50.54 --playbook playbooks/homepage-regroup-fv-ml1.yaml ``` ⚠ **The GPU-backed model seats are deliberately still named `AI - Inference`, diff --git a/stacks/homepage/conf/docker.yaml b/stacks/homepage/conf/docker.yaml index f2064e0..ee4c724 100644 --- a/stacks/homepage/conf/docker.yaml +++ b/stacks/homepage/conf/docker.yaml @@ -17,13 +17,13 @@ nh3-pfi-docker: host: 10.100.50.40 port: 2375 -# ana-ml2-docker — reached via a socat proxy on ana-docker (stacks/ana-ml2-proxy). -# ana-ml2's own :2375 is up but not reachable cross-site from ESH (a return-route -# gap on ana-ml2 itself; no SSH access from the ops seats to fix its routing). +# fv-ml1-docker — reached via a socat proxy on ana-docker (stacks/fv-ml1-proxy). +# fv-ml1's own :2375 is up but not reachable cross-site from ESH (a return-route +# gap on fv-ml1 itself; no SSH access from the ops seats to fix its routing). # ana-docker is on the same Anaheim subnet and IS reachable from ESH, so it -# relays 10.250.50.70:2354 -> 10.250.50.54:2375. Point back at 10.250.50.54:2375 -# and `docker compose down` the proxy once ana-ml2 has a return route to ESH. -ana-ml2-docker: +# relays 10.250.50.70:2354 -> 10.251.50.54:2375. Point back at 10.251.50.54:2375 +# and `docker compose down` the proxy once fv-ml1 has a return route to ESH. +fv-ml1-docker: host: 10.250.50.70 port: 2354 diff --git a/stacks/homepage/conf/services.yaml b/stacks/homepage/conf/services.yaml index e7e948b..7b2c967 100644 --- a/stacks/homepage/conf/services.yaml +++ b/stacks/homepage/conf/services.yaml @@ -118,14 +118,14 @@ siteMonitor: http://10.250.50.70:5001 description: Docker VM (ana-docker, Dockge at :5001) - PFI-ANA-ML2: - ping: 10.250.50.54 + ping: 10.251.50.54 icon: mdi-brain description: GPU host (bare-metal) - PFI-ANA-ML2 BMC: href: https://10.250.250.50 icon: mdi-brain siteMonitor: https://10.250.250.50 - description: BMC (ana-ml2) + description: BMC (fv-ml1) - SFsrv-ANA: href: https://10.250.250.115:8006 icon: si-proxmox diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index a1437d5..ca26bc5 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -1,5 +1,5 @@ # LiteLLM gateway config — fronts the vLLM services on ana-ml2 -# (10.250.50.54) and logs every request + response so they're +# (10.251.50.54) and logs every request + response so they're # inspectable in the Logs UI at http://10.250.50.70:4000/ui. # # Deploys to /opt/docker/conf/litellm/config.yaml (mounted read-only @@ -22,7 +22,7 @@ model_list: # ⚠ ALIAS COLLISION — SEVERAL NAMES, ONE SET OF WEIGHTS # # As of 2026-08-23 these SEVEN aliases all resolve to the same backend -# (qwen3.8-27b-uncensored @ 10.250.50.54:8015): +# (qwen3.8-27b-uncensored @ 10.251.50.54:8015): # # chat-judge classifier gen image-judge # qwen-image-bench summarizer summarizer-large @@ -59,7 +59,7 @@ model_list: # - model_name: granite-4.1-8b # litellm_params: # model: hosted_vllm/granite-4.1-8b -# api_base: http://10.250.50.54:8004/v1 +# api_base: http://10.251.50.54:8004/v1 # api_key: os.environ/VLLM_API_KEY # temperature: 0 # model_info: @@ -71,7 +71,7 @@ model_list: - model_name: summarizer litellm_params: model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.250.50.54:8015/v1 + api_base: http://10.251.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY temperature: 0 extra_body: @@ -85,7 +85,7 @@ model_list: - model_name: classifier litellm_params: model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.250.50.54:8015/v1 + api_base: http://10.251.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY temperature: 0 extra_body: @@ -100,7 +100,7 @@ model_list: - model_name: summarizer-large litellm_params: model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.250.50.54:8015/v1 + api_base: http://10.251.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY presence_penalty: 1.5 temperature: 0.7 @@ -123,7 +123,7 @@ model_list: - model_name: qwen-image-bench litellm_params: model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.250.50.54:8015/v1 + api_base: http://10.251.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY temperature: 0 top_p: 1.0 @@ -137,7 +137,7 @@ model_list: - model_name: image-judge litellm_params: model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.250.50.54:8015/v1 + api_base: http://10.251.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY temperature: 0 top_p: 1.0 @@ -182,7 +182,7 @@ model_list: - model_name: gen litellm_params: model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.250.50.54:8015/v1 + api_base: http://10.251.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY presence_penalty: 1.5 temperature: 0.7 @@ -200,7 +200,7 @@ model_list: # Distinct served-name so a thinking-off `gen` request can't mutate this deployment's # enable_thinking (shared-config-mutation footgun). Same backend :8015, different model id. model: hosted_vllm/qwen3.8-27b-uncensored-thinking - api_base: http://10.250.50.54:8015/v1 + api_base: http://10.251.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY # CANONICAL Qwen3.8 THINKING sampling set (Qwen + unsloth "Best Practices" # §1, identical in both): temperature=1.0, top_p=0.95, top_k=20, @@ -248,7 +248,7 @@ model_list: - model_name: char-rp litellm_params: model: hosted_vllm/char-rp - api_base: http://10.250.50.54:8016/v1 + api_base: http://10.251.50.54:8016/v1 api_key: os.environ/VLLM_API_KEY temperature: 1.1 top_p: 0.95 @@ -274,7 +274,7 @@ model_list: - model_name: char-rp-reasoning litellm_params: model: hosted_vllm/char-rp-thinking - api_base: http://10.250.50.54:8016/v1 + api_base: http://10.251.50.54:8016/v1 api_key: os.environ/VLLM_API_KEY temperature: 1.1 top_p: 0.95 @@ -310,8 +310,8 @@ model_list: # character card both sit at ~0%. Fable-Fusion is Heretic-abliterated. # # ROLLBACK (restores Dark-Scarlett): - # ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe && sudo docker compose down' - # ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && sudo docker compose up -d' + # ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe && sudo docker compose down' + # ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && sudo docker compose up -d' # then revert this block to api_base :8018 / model hosted_vllm/char-rp-reasoning # and restart litellm (~52s). # @@ -337,7 +337,7 @@ model_list: - model_name: sec litellm_params: model: hosted_vllm/mog-sec-27b - api_base: http://10.250.50.54:8019/v1 + api_base: http://10.251.50.54:8019/v1 api_key: os.environ/VLLM_API_KEY temperature: 0.7 top_p: 0.8 @@ -356,7 +356,7 @@ model_list: - model_name: sec-reasoning litellm_params: model: hosted_vllm/mog-sec-27b-thinking - api_base: http://10.250.50.54:8019/v1 + api_base: http://10.251.50.54:8019/v1 api_key: os.environ/VLLM_API_KEY temperature: 1.0 top_p: 0.95 @@ -393,7 +393,7 @@ model_list: - model_name: qwen3-embedding litellm_params: model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B - api_base: http://10.250.50.54:8001/v1 + api_base: http://10.251.50.54:8001/v1 api_key: os.environ/VLLM_API_KEY model_info: mode: embedding @@ -423,7 +423,7 @@ model_list: - model_name: chat-judge litellm_params: model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.250.50.54:8015/v1 + api_base: http://10.251.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY temperature: 0 top_p: 1.0 @@ -438,7 +438,7 @@ model_list: - model_name: reranker litellm_params: model: hosted_vllm/BAAI/bge-reranker-v2-m3 - api_base: http://10.250.50.54:8013/v1 + api_base: http://10.251.50.54:8013/v1 api_key: os.environ/VLLM_API_KEY model_info: mode: rerank @@ -456,7 +456,7 @@ model_list: - model_name: coder-fast litellm_params: model: hosted_vllm/qwen2.5-coder-1.5b - api_base: http://10.250.50.54:8020/v1 + api_base: http://10.251.50.54:8020/v1 api_key: os.environ/VLLM_API_KEY model_info: mode: completion @@ -681,7 +681,7 @@ model_list: - model_name: erp-tune-v2 litellm_params: model: hosted_vllm/erp-tune-v2 - api_base: http://10.250.50.54:8098/v1 + api_base: http://10.251.50.54:8098/v1 api_key: os.environ/VLLM_API_KEY # gemma4-26b-a4b-it-base -> the UNMODIFIED upstream instruct release, bf16 @@ -743,7 +743,7 @@ model_list: - model_name: gemma4-26b-a4b-it-base litellm_params: model: hosted_vllm/gemma4-26b-a4b-it-base - api_base: http://10.250.50.54:8099/v1 + api_base: http://10.251.50.54:8099/v1 api_key: os.environ/VLLM_API_KEY # merged-run03 -> the in-house ERP/RP SFT, RUN 3, MERGED bf16 (:8099, vLLM, ana-ml2 @@ -848,7 +848,7 @@ model_list: # - model_name: merged-run03 # litellm_params: # model: hosted_vllm/merged-run03 - # api_base: http://10.250.50.54:8099/v1 + # api_base: http://10.251.50.54:8099/v1 # api_key: os.environ/VLLM_API_KEY # trial -> ERP/RP SFT RUN 7, NVFP4A16 (weight-only) quant of the merged LoRA, served on @@ -879,9 +879,10 @@ model_list: # # THIS GATEWAY IS THE SHARED-KEY SURFACE: `all-agents-local` reaches every model here, # in every session and project. Removing this alias does not remove the operator's - # access -- the direct endpoint http://10.250.50.54:8021/v1 is unaffected. + # access -- the direct endpoint http://10.251.50.54:8021/v1 is unaffected. # - # Same-site: seat and gateway are both at Anaheim (local hop, no mesh crossing). + # CROSS-SITE since 2026-09-12: gateway on ana-docker (Anaheim), seat on fv-ml1 +# (Fountain Valley) -- metro hop over the headscale mesh, ~6ms. # char-rp-fast — the MeroMero A4B MoE, on ana-ml2 GPU1 (:8021). Operator, 2026-09-10: # "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias @@ -919,7 +920,7 @@ model_list: - model_name: char-rp-fast litellm_params: model: hosted_vllm/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 - api_base: http://10.250.50.54:8021/v1 + api_base: http://10.251.50.54:8021/v1 api_key: os.environ/VLLM_API_KEY extra_body: min_p: 0.05 @@ -950,14 +951,14 @@ general_settings: maximum_spend_logs_retention_period: "7d" maximum_spend_logs_retention_interval: "1d" # scalar-judge → Skywork-Reward-V2 (scalar reward model; vLLM pooling on - # ana-ml2:8003). LiteLLM has no reward/pooling MODE, so this is a passthrough, + # fv-ml1:8003). LiteLLM has no reward/pooling MODE, so this is a passthrough, # not a model_list alias. Gateway-key-gated. Consumers POST the reward body to # /scalar-judge/ (e.g. /pooling or /classify), forwarded to :8003. # SWAP-SENSITIVE: a different reward model shifts the score scale, so consumers # must recalibrate thresholds after a backing swap. pass_through_endpoints: - path: "/scalar-judge" - target: "http://10.250.50.54:8003" + target: "http://10.251.50.54:8003" forward_headers: true include_subpath: true diff --git a/stacks/llama-swap/compose.yaml b/stacks/llama-swap/compose.yaml index 2cec60c..584f36d 100644 --- a/stacks/llama-swap/compose.yaml +++ b/stacks/llama-swap/compose.yaml @@ -2,7 +2,7 @@ # # Proxies OpenAI-compatible API requests to llama.cpp server instances # and swaps which model is loaded into VRAM per request. Runs on -# ana-ml2 using both GPUs dynamically (no explicit device pinning — +# fv-ml1 using both GPUs dynamically (no explicit device pinning — # llama-swap picks per-model-definition). # # Model definitions live in /opt/docker/conf/llama-swap/config.yaml on @@ -47,8 +47,8 @@ services: - homepage.group=AI Systems - homepage.name=llama-swap - homepage.icon=mdi-swap-horizontal - - homepage.description=GGUF model swapper (llama.cpp; ana-ml2) - - homepage.href=http://10.250.50.54:${LLAMA_SWAP_PORT} + - homepage.description=GGUF model swapper (llama.cpp; fv-ml1) + - homepage.href=http://10.251.50.54:${LLAMA_SWAP_PORT} networks: tnet: diff --git a/stacks/meromero-charrp/compose.yaml b/stacks/meromero-charrp/compose.yaml index 04308ac..46cc02b 100644 --- a/stacks/meromero-charrp/compose.yaml +++ b/stacks/meromero-charrp/compose.yaml @@ -1,5 +1,5 @@ # meromero-charrp — G4-MeroMero-v2-31B NVFP4A16 char-rp PROSE seat (non-thinking, MULTIMODAL) -# on ana-ml2 GPU0. Replaces the retired Magidonia GGUF seat (char-rp-gguf / llama-charrp). +# on fv-ml1 GPU0. Replaces the retired Magidonia GGUF seat (char-rp-gguf / llama-charrp). # # Gemma-4 dense but with SLIDING-WINDOW attention -> KV-efficient, serves the full native 256K # at ~2x concurrency. Vision enabled: preprocessor_config.json was materialized from @@ -93,8 +93,8 @@ services: - homepage.group=AI - Dormant - homepage.name=char-rp (MeroMero-v2 NVFP4) — rollback - homepage.icon=mdi-drama-masks - - homepage.description=Superseded 2026-08-24 by the Gemma-4 26B-A4B MoE seat; dense 31B NVFP4A16, kept for rollback (ana-ml2 GPU0) - - homepage.href=http://10.250.50.54:${MEROMERO_PORT:-8016}/docs + - homepage.description=Superseded 2026-08-24 by the Gemma-4 26B-A4B MoE seat; dense 31B NVFP4A16, kept for rollback (fv-ml1 GPU0) + - homepage.href=http://10.251.50.54:${MEROMERO_PORT:-8016}/docs networks: tnet: diff --git a/stacks/sglang/compose.yaml b/stacks/sglang/compose.yaml index cb44d97..51ae8da 100644 --- a/stacks/sglang/compose.yaml +++ b/stacks/sglang/compose.yaml @@ -1,4 +1,4 @@ -# SGLang — alternative LLM serving engine, stood up on ana-ml2 to bench +# SGLang — alternative LLM serving engine, stood up on fv-ml1 to bench # head-to-head against vLLM on the same model + hardware. # # Bench-oriented (not yet a permanent service): point it at the SAME checkpoint @@ -68,8 +68,8 @@ services: - homepage.group=AI Systems - homepage.name=SGLang (bench) - homepage.icon=mdi-speedometer - - homepage.description=SGLang serving — vLLM bench comparison (ana-ml2) - - homepage.href=http://10.250.50.54:${SGLANG_PORT} + - homepage.description=SGLang serving — vLLM bench comparison (fv-ml1) + - homepage.href=http://10.251.50.54:${SGLANG_PORT} networks: tnet: