vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

fv-ml1: complete the cutover — rename, renumber, DNS, and the LiteLLM repoint

Browse Source 
The box is physically at Fountain Valley, renamed, renumbered onto 10.251/16,
and serving inference again. This lands the repo half of that.

Host: hostname ana-ml2 -> fv-ml1, pinned to 10.251.50.54 by a dnsmasq
reservation so the address the runbook, DNS and LiteLLM all assume is the
address it actually has. Its headscale node is renamed too.

The sweep ran from scripts/fv-ml1-rename-sweep.sh, whose allowlist is the
reason this diff touches current-state files and not the record. Dated
persistent-memory entries, archival-memory and incident notes still say
ana-ml2 in 31 and 62 places respectively, because that is what the box was
when those things happened. Rewriting them would make the history lie.

LiteLLM was the load-bearing piece and needed more than the api_base sed the
runbook describes. Twenty api_base entries repointed, but a grep-and-verify
pass also caught a LIVE pass_through_endpoints target for the scalar-judge
reward route still on the old address -- an api_base-only substitution would
have left it dead. Four prose references describing current state were
repointed as well; one historical note recording where a hand-test was run
is deliberately left pointing at 10.250.50.54.

Two facts in the server tables were wrong and are corrected here. The site is
Fountain Valley, not Anaheim. And the box has FOUR RTX PRO 6000 Blackwell
Max-Q, not two -- verified by nvidia-smi -L and independently by PCI
enumeration of four GB202GL devices. That is 391 GB of VRAM rather than 196,
which changes what fits on it.

DNS: fv-ml1, fv-ml1-bmc and fv-gw added under the fv site via the piggyback
approach, scriberr re-homed, and the ana-ml2 records removed. Applied to all
three resolvers. The BMC record carries a warning that its 802.1q VLAN tag
must stay disabled -- it shipped tagging VLAN 250 into an untagged port,
which made it invisible to every network-side diagnostic and is the reason
it appeared dead through several cable changes.

Verified end to end: summarizer and sec both answer through the Anaheim
gateway across the mesh to FV seats on different ports.
This commit is contained in:
Vuong Hoang
2026-09-12 22:00:50 -07:00
parent fde7834635
commit 91bda3c480
31 changed files with 115 additions and 112 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+3 -3
View File
@@ -199,7 +199,7 @@ Observed and standardized across servers:
- **Restart policy:** `restart: unless-stopped` for daemons
- **Homepage labels** on user-facing services. The dashboard runs on
`esh-docker-vm` and reads the Docker API of **every** host in
`stacks/homepage/conf/docker.yaml` (ana-docker, ana-ml2, nh3-docker,
`stacks/homepage/conf/docker.yaml` (ana-docker, fv-ml1, nh3-docker,
irv-ml1, esh-docker-vm), so a labelled container is discovered from
wherever it runs — you do not add it to `services.yaml` as well. Doing both
renders it twice.
@@ -229,7 +229,7 @@ Observed and standardized across servers:
| Name | IP | Site | Role | Details |
|------|-----|------|------|---------|
| ana-ml2 | 10.250.50.54 | Anaheim (`10.250.0.0/16`) | GPU / AI inference (bare metal, dual RTX PRO 6000 Blackwell Max-Q, 96 GB each) | `servers/ana-ml2/README.md` |
| fv-ml1 | 10.251.50.54 | Fountain Valley (`10.251.0.0/16`) | GPU / AI inference (bare metal, **4× RTX PRO 6000 Blackwell Max-Q, 96 GB each = 391 GB VRAM**) | `servers/fv-ml1/README.md` |
| irv-ml1 | 100.64.0.6 (headscale mesh); 10.6.110.50 (Irvine LAN, routed via mesh) | Irvine — via headscale mesh; wg0 lifeline (10.100.79.3) dormant post-cutover | GPU / AI inference (bare metal, RTX 3090 + RTX A6000, native stacks) | `servers/irv-ml1/README.md` |
| ana-docker | 10.250.50.70 | Anaheim | General-purpose Docker host (non-GPU VM on pfi-pve) | `servers/ana-docker/README.md` |
| pfi-ana-webhost | 10.250.50.52 | Anaheim | VM on pfi-pve (VMID 110) — web workload | `servers/pfi-ana-webhost/README.md` |
@@ -257,7 +257,7 @@ Observed and standardized across servers:
| esh-vm-db | 10.0.50.60 | ESH home lab | DB VM — PostgreSQL (paperless-ng) + MongoDB; bare-metal VM, no Docker | `servers/esh-vm-db/README.md` |
**Placement rules:**
- GPU-required stacks → `ana-ml2` (primary, Anaheim) or `irv-ml1`
- GPU-required stacks → `fv-ml1` (primary, Fountain Valley) or `irv-ml1`
(secondary, Irvine — bigger VRAM ceiling at 72 GB total). Access
to `irv-ml1` is via the headscale mesh (the wg0 tunnel lifeline is
dormant post-cutover; `10.100.79.3` no longer answers).
README.md
+4 -4
View File
@@ -10,7 +10,7 @@ See **[CLAUDE.md](CLAUDE.md)** for the full set of conventions and the rules Cla
| Host | IP | Site | Role |
|---|---|---|---|
| ana-ml2 | `10.250.50.54` | Anaheim (`10.250.0.0/16`) | GPU / AI inference (bare metal) |
| fv-ml1 | `10.251.50.54` | Fountain Valley (`10.251.0.0/16`) | GPU / AI inference (bare metal, 4× RTX PRO 6000 Blackwell Max-Q) |
| ana-docker | `10.250.50.70` | Anaheim | General-purpose Docker + cross-site hubs (VM on pfi-pve) |
| nh3-docker | `10.100.50.40` | NH3 (`10.100.0.0/16`) | General-purpose Docker (VM on nh3-pve) |
| esh-docker-vm | `10.0.50.45` | ESH home lab (`esteban.net`) | Home-lab Docker (VM on esh-pve, non-PFI scope) |
@@ -64,7 +64,7 @@ Per-host snapshots of the running system live under `servers/<host>/system-detai
## Current stacks
**GPU (ana-ml2):**
**GPU (fv-ml1):**
- `llama-swap` — GGUF model swapper via llama.cpp (port 9292)
- `vllm` — embeddings (8001) + reranker (8002) + Skywork reward classifier (8003) via vLLM
@@ -76,7 +76,7 @@ Per-host snapshots of the running system live under `servers/<host>/system-detai
- Fleet services: `beszel` (metrics hub, port 8090), `dozzle-hub` (log viewer, 8088), `backrest` (restic UI, 9898)
- Backup target: `rest-server-ana` on port 8000
**GPU ana-ml2 (non-canonical for now):**
**GPU fv-ml1 (non-canonical for now):**
- `comfyui`, `kokoro`, `parakeet`, `vibevoice` alongside the canonical `llama-swap` + `vllm`
**NH3 (nh3-docker):**
@@ -170,7 +170,7 @@ Goal: **every Docker host + configs + every database** covered, not just VM imag
|---|---|
| VM-level (Proxmox vzdump) | ✅ All running guests covered across pfi-pve / nh3-pve / esh-pve-nas; esh-pve has VMID 108 uncovered |
| ana-docker restic (host files + DBs) | ✅ `configs/restic/ana-docker/` with pre-backup hooks for synapse / seafile / vaultwarden-pg / gitea (native dump) / openwebui |
| ana-ml2 restic | ✅ `configs/restic/ana-ml2/` — bare-metal host files (no DB hooks needed) |
| fv-ml1 restic | ✅ `configs/restic/fv-ml1/` — bare-metal host files (no DB hooks needed) |
| nh3-docker restic | ✅ Light — no DB hooks needed |
| esh-docker-vm restic | ✅ With DB hooks for paperless-postgres (external), home-assistant + pgadmin + uptime-kuma (host-side sqlite3), calibre-web-automated (in-container sqlite3) |
| vm-esh-nas restic | ✅ Light — NFS mounts explicitly excluded |
dns/internal.yaml
+5 -3
View File
@@ -54,7 +54,8 @@ sites:
hosts:
# ---- ana: Anaheim colo ----
- {name: ana-docker, site: ana, v4: 10.250.50.70, note: general-purpose docker host}
- {name: ana-ml2, site: ana, v4: 10.250.50.54, note: GPU inference, dual RTX PRO 6000}
# ---- fv: Fountain Valley colo (piggyback DNS until FV has its own resolver) ----
- {name: fv-ml1, site: fv, v4: 10.251.50.54, note: GPU inference, 4x RTX PRO 6000 Blackwell Max-Q (ex ana-ml2, relocated 2026-09-12)}
- {name: ana-nas, site: ana, v4: 10.250.50.50, note: CT109 on pfi-pve — NFS/SMB}
- {name: ana-filebot, site: ana, v4: 10.250.50.53, note: file-task automation}
- {name: ana-wg, site: ana, v4: 10.250.50.252, note: WireGuard host}
@@ -67,7 +68,8 @@ hosts:
- {name: pfi-pve, site: ana, v4: 10.250.250.31, note: Proxmox hypervisor}
- {name: ana-gw, site: ana, v4: 10.250.0.1, note: FortiGate-80F edge}
- {name: pfi-pve-idrac, site: ana, v4: 10.250.250.30, note: iDRAC — OOB for pfi-pve}
- {name: ana-ml2-bmc, site: ana, v4: 10.250.250.50, note: BMC for ana-ml2}
- {name: fv-ml1-bmc, site: fv, v4: 10.251.250.50, note: BMC for fv-ml1 (Supermicro/ATEN; 802.1q VLAN tag must stay DISABLED)}
- {name: fv-gw, site: fv, v4: 10.251.50.1, note: OPNsense gateway vb-gateway, mesh subnet-router for 10.251.0.0/16; public 172.83.89.66 = fv.phasefinal.com}
# SureFire tenant hardware — PFI-managed under the hosting agreement.
- {name: sfsrv-ana, site: ana, v4: 10.250.250.115, note: SureFire tenant hypervisor}
- {name: sf-ana-container, site: ana, v4: 10.250.150.100, note: SureFire tenant container host}
@@ -120,4 +122,4 @@ aliases:
- {name: booth, site: nh3, target: nh3-dev, note: The Booth :8090}
- {name: wherethef, site: nh3, target: nh3-dev, note: WhereTF :8093}
- {name: homepage, site: esh, target: esh-docker-vm, note: fleet dashboard :5100}
- {name: scriberr, site: ana, target: ana-ml2, note: transcription + diarization :8080 (GPU1)}
- {name: scriberr, site: fv, target: fv-ml1, note: transcription + diarization :8080 (GPU1)}
docs/pfi/docker-stack.md
+1 -1
View File
@@ -131,7 +131,7 @@ group memberships) is in
[`stacks/llama-swap/conf/config.yaml`](../../stacks/llama-swap/conf/config.yaml).
Don't mirror the table here — it drifted last time and immediately
went stale. Live HTTP listing:
`curl http://10.250.50.54:9292/v1/models`.
`curl http://10.251.50.54:9292/v1/models`.
**Model Groups**:
docs/pfi/proxmox-vms.md
+1 -1
View File
@@ -73,5 +73,5 @@ Reverse this for a graceful host shutdown (done 2026-09-05 for the PERC swap).
## Not on this hypervisor
`ana-ml2` (10.250.50.54) is bare-metal Supermicro; `sfsrv-ana` (10.250.250.115) is the
`fv-ml1` (10.251.50.54) is bare-metal Supermicro; `sfsrv-ana` (10.250.250.115) is the
SureFire client's own Dell R630; `pfi-gx10` is at NH3.
docs/pfi/reranker-selection-ledger.md
+12 -12
View File
@@ -18,9 +18,9 @@ implementation.
**CLEARED (green):**
- Execute Brokkr's reranker-selection recommendations unattended.
- Bring **down the prod reranker** at `ana-ml2:8002` (qwen3-reranker-0.6B) —
- Bring **down the prod reranker** at `fv-ml1:8002` (qwen3-reranker-0.6B) —
**temporarily OR permanently**.
- Down **ONE** of the RP (roleplay) seats on ana-ml2 **temporarily** to free
- Down **ONE** of the RP (roleplay) seats on fv-ml1 **temporarily** to free
GPU/VRAM for testing.
- Temporarily clear space for the smoke/bench.
- Pull models, stand up side-port vLLM benches, run the harness — whatever the
@@ -30,7 +30,7 @@ implementation.
- **NO permanent deletion of anything** (no `rm`/`docker volume rm`/model-weight
deletion/data destruction). Downing ≠ deleting.
- **NO taking anything else offline** beyond (a) the prod reranker and (b) ONE
ana-ml2 RP seat. (Not granite/embed/reward/coder/gen/a second RP seat/muninn/etc.)
fv-ml1 RP seat. (Not granite/embed/reward/coder/gen/a second RP seat/muninn/etc.)
- **NO rebooting machines.**
**Process:** accumulate assumptions here; operator reverses at the end.
@@ -54,7 +54,7 @@ implementation.
## Current board at handoff
- **Prod reranker:** `ana-ml2:8002` = `vllm-rerank` (Qwen/Qwen3-Reranker-0.6B),
- **Prod reranker:** `fv-ml1:8002` = `vllm-rerank` (Qwen/Qwen3-Reranker-0.6B),
reverted to baseline `classifier_from_token:["no","yes"]`, healthy. Compose:
`/opt/docker/compose/vllm/compose.yaml` (canonical mirror
`stacks/vllm/compose.yaml`). Gateway alias `reranker`/`qwen3-reranker` →
@@ -75,7 +75,7 @@ implementation.
worldtree-dev signals the lever run is done).
- **Dual query-set requirement (hard):** score bare-name anchor queries AND
question-shaped; bar = recovering the name-lookup class.
- **VRAM:** 4B ≈ 4–5 GB fp8, 8B ≈ 9 GB; ana-ml2 Blackwell has headroom.
- **VRAM:** 4B ≈ 4–5 GB fp8, 8B ≈ 9 GB; fv-ml1 Blackwell has headroom.
---
@@ -84,12 +84,12 @@ implementation.
### 2026-08-06 — A2 brought up (Brokkr thread 01KZBSTSJA…)
- **Backend:** `vllm-rerank-a2` — standalone `docker run` (NOT in the vllm compose
stack), on ana-ml2 **GPU1**, host port **:8012** → container 8000. Image
stack), on fv-ml1 **GPU1**, host port **:8012** → container 8000. Image
`vllm/vllm-openai:latest` (=0.24.0). Args: model
`tomaarsen/Qwen3-Reranker-0.6B-seq-cls`, `--runner pooling`, `--gpu-memory-utilization
0.03`, `--max-model-len 8192`, `--dtype auto`, `--restart no`. Native
`Qwen3ForSequenceClassification` — NO hf-overrides. Routes /rerank /score /classify.
- **Gateway alias:** `reranker-a2-qwen3-seqcls` → `http://10.250.50.54:8012/v1`,
- **Gateway alias:** `reranker-a2-qwen3-seqcls` → `http://10.251.50.54:8012/v1`,
mode rerank. Added via LiteLLM **`/model/new`** (DB-backed, `store_model_in_db:true`)
— **no gateway restart** (respects the "nothing else offline" line). Verified 200
through the gateway.
@@ -106,7 +106,7 @@ implementation.
- D4 — did NOT down any RP seat (A2 is 0.6B / 3.5 GB; no VRAM pressure).
**Cleanup for A2 (run at end / on reversal):**
- `ssh infra-ops@10.250.50.54 'sudo docker stop vllm-rerank-a2 && sudo docker rm vllm-rerank-a2'`
- `ssh infra-ops@10.251.50.54 'sudo docker stop vllm-rerank-a2 && sudo docker rm vllm-rerank-a2'`
- Delete gateway alias: `POST /model/delete {"id": <model_id>}` (id via `/model/info?model_name=reranker-a2-qwen3-seqcls`), infra-ops admin key. (DB-persisted, so it survives a restart — must be explicitly deleted.)
- No weights deleted (red line); HF cache under /tank/aimodels/huggingface retains the 0.6B-seq-cls download.
@@ -114,11 +114,11 @@ implementation.
### 2026-08-06 — A3 + A4 pre-staged (Brokkr said pre-stage in parallel, hold A5)
- **A3** `vllm-rerank-a3` — ana-ml2 GPU1 :8013, `BAAI/bge-reranker-v2-m3`
- **A3** `vllm-rerank-a3` — fv-ml1 GPU1 :8013, `BAAI/bge-reranker-v2-m3`
(XLMRobertaForSequenceClassification), same run pattern, util 0.03. VRAM ≈ **2.3 GB**.
Latency (20-doc, ~1500-char, shared GPU1): single p50 **105 ms**; 8-conc p50 214 ms, ~34 req/s.
Gateway alias `reranker-a3-bge-v2-m3` via /model/new (200, verified).
- **A4** `vllm-rerank-a4` — ana-ml2 GPU1 :8014, `Alibaba-NLP/gte-reranker-modernbert-base`
- **A4** `vllm-rerank-a4` — fv-ml1 GPU1 :8014, `Alibaba-NLP/gte-reranker-modernbert-base`
(ModernBertForSequenceClassification), util 0.02. VRAM ≈ **1.4 GB**. Latency: single
p50 **102 ms**; 8-conc p50 153 ms, ~51 req/s. Gateway alias `reranker-a4-gte-modernbert`
via /model/new (200, verified).
@@ -129,7 +129,7 @@ implementation.
If A5 (4B, ~4–5 GB) is greenlit: fits GPU1 tight or GPU0 (~9 GB free) — no RP-seat downing expected.
**Cleanup for A3/A4 (same pattern as A2):** `docker stop/rm vllm-rerank-a3 vllm-rerank-a4`
on ana-ml2; `/model/delete` the two aliases (DB-persisted); weights retained in HF cache.
on fv-ml1; `/model/delete` the two aliases (DB-persisted); weights retained in HF cache.
### 2026-08-06 — A2 verdict (Brokkr full grid): training-prior confirmed
@@ -202,7 +202,7 @@ on ana-ml2; `/model/delete` the two aliases (DB-persisted); weights retained in
Brokkr signals the bake-off is closed (`docker stop/rm` + `/model/delete`).
- **A4 (`vllm-rerank-a4` :8014)** + alias — KEEP for now (Brokkr's documented throughput fallback).
- **A3 (`vllm-rerank-a3` :8013)** — now PRODUCTION (backs the `reranker` alias). Hardened
2026-08-06: `docker update --restart unless-stopped` (survives ana-ml2 reboot, no recreate).
2026-08-06: `docker update --restart unless-stopped` (survives fv-ml1 reboot, no recreate).
A4 given the same. **Remaining follow-up (not urgent): promote A3 from throwaway `docker run`
to a canonical compose service** (`stacks/vllm/`) for config-managed consistency — a recreate,
so do it in a window since it briefly drops `reranker`.
docs/runbooks/disaster-recovery.md
+3 -3
View File
@@ -37,7 +37,7 @@ pfi-pve during morning backup window. Backup suspected as trigger
**Blast radius:**
- ~~pfi-postgres (VM 105) — PGDATA on `/mnt/db`~~ — **migrated to local disk 2026-04-23**. vaultwarden, gitea, paperless-ng, zammad no longer cascade on ana-nas outage. Left in history for the recovery pre-migration.
- ana-docker rest-server-ana — repo data on `/mnt/backup` → all ana-side restic clients fail (ana-docker, ana-ml2, esh-docker-vm, vm-esh-nas)
- ana-docker rest-server-ana — repo data on `/mnt/backup` → all ana-side restic clients fail (ana-docker, fv-ml1, esh-docker-vm, vm-esh-nas)
- PBS-ANA datastore — NFS-backed on `/mnt/backup/pbs-ana` → fleet vzdumps fail, PBS-NH3 sync fails
- ana-docker NFS mounts for `/mnt/docker`, `/mnt/compose`, `/mnt/pve-VMStorage` if used → various stack misbehavior
@@ -207,12 +207,12 @@ Essentially **all Anaheim primary services** go offline. Because ana-nas lives h
## Tier 4 — Specialty workloads
### ana-ml2 (bare metal Supermicro, 10.250.50.54, BMC 10.250.250.50)
### fv-ml1 (bare metal Supermicro, 10.251.50.54, BMC 10.251.250.50)
**Blast radius:** AI inference services (llama-swap, vllm). Consumer-facing chat/embedding/reward-scoring endpoints fail.
**Recovery:**
1. Check OS via SSH. If unresponsive, BMC console at <https://10.250.250.50>.
1. Check OS via SSH. If unresponsive, BMC console at <https://10.251.250.50>.
2. If hardware issue: BMC logs, power cycle via IPMI, check GPU health (`nvidia-smi`).
3. Docker stacks auto-start via compose `restart: unless-stopped`.
docs/runbooks/fv-ml1-cutover.md
+1 -1
View File
@@ -49,7 +49,7 @@ path the fleet lacked on 2026-09-11. Do NOT tie the mesh node to fv-ml1 itself.
- [ ] `dns/internal.yaml` — add the two records below (piggyback; NOT a `sites:` entry), comment out the old `ana-ml2` / `ana-ml2-bmc` lines. Then `scripts/dns-sync.py --dry-run` → apply.
```yaml
# ---- fv: Fountain Valley colo (piggyback DNS until FV has its own resolver) ----
- {name: fv-ml1, site: fv, v4: 10.251.50.54, note: GPU inference, dual RTX PRO 6000 (ex ana-ml2, relocated 2026-09)}
- {name: fv-ml1, site: fv, v4: 10.251.50.54, note: GPU inference, 4x RTX PRO 6000 Blackwell Max-Q (ex ana-ml2, relocated 2026-09)}
- {name: fv-ml1-bmc, site: fv, v4: 10.251.250.50, note: BMC for fv-ml1}
```
- [ ] mesh route to 10.251.0.0/16 approved in headscale (OPNsense subnet-router).
playbooks/ana-ml2-mesh-routes.yaml → playbooks/fv-ml1-mesh-routes.yaml
View File
playbooks/ana-ml2-overcommit-memory.yaml → playbooks/fv-ml1-overcommit-memory.yaml
View File
playbooks/ana-ml2-pool-health.yaml → playbooks/fv-ml1-pool-health.yaml
View File
playbooks/ana-ml2-training-window-close.yaml → playbooks/fv-ml1-training-window-close.yaml
View File
playbooks/ana-ml2-training-window-open.yaml → playbooks/fv-ml1-training-window-open.yaml
View File
servers/ana-ml2/ssh-target
-1
View File
@@ -1 +0,0 @@
10.250.50.54
servers/ana-ml2/README.md → servers/fv-ml1/README.md
+8 -8
View File
@@ -1,13 +1,13 @@
# ana-ml2
# fv-ml1
Primary AI inference host for PFI.
## Network
- **LAN IP:** 10.250.50.54 (in-band, OS-side)
- **BMC (OOB):** 10.250.250.50 — Supermicro IPMI web UI
at <https://10.250.250.50> (homepage card: *PFI-ANA-ML2 BMC*)
- **SSH:** standard port 22 on 10.250.50.54
- **LAN IP:** 10.251.50.54 (in-band, OS-side)
- **BMC (OOB):** 10.251.250.50 — Supermicro IPMI web UI
at <https://10.251.250.50> (homepage card: *PFI-ANA-ML2 BMC*)
- **SSH:** standard port 22 on 10.251.50.54
## Hardware
@@ -54,9 +54,9 @@ history, not an active fault, and the 2 CKSUM were the stale-block artefact of t
replace on growth** (`zpool replace tank nvme7n1 <new>`; any PM1725b 1.6 TB or
larger). Slot 0-5 itself deserves a reseat / cable check at the next hands-on
visit — a bay that dropped a drive for 4½ months is the likelier fault than the
drive. Playbook: `playbooks/ana-ml2-pool-health.yaml` (idempotent; rerunning is a
drive. Playbook: `playbooks/fv-ml1-pool-health.yaml` (idempotent; rerunning is a
no-op). ⚠ **Nothing alerts on this** — see the open follow-up in
`persistent-memory.d/2026-09-09-ana-ml2-pool-actions-done.md`.
`persistent-memory.d/2026-09-09-fv-ml1-pool-actions-done.md`.
## Key paths
@@ -119,7 +119,7 @@ seats, safe to leave: `mistral-medium-3.5`, `mistral-small-4(-heretic)`,
## Refresh state
```bash
scripts/refresh-server-info.sh ana-ml2
scripts/refresh-server-info.sh fv-ml1
```
Latest snapshot: `system-details.txt` (regenerate as needed).
servers/fv-ml1/ssh-target
+1
View File
@@ -0,0 +1 @@
10.251.50.54
servers/ana-ml2/system-details.txt → servers/fv-ml1/system-details.txt
View File
servers/pfi-pve/README.md
+2 -2
View File
@@ -40,7 +40,7 @@ hypervisor):
- **CT 112** — `ana-filebot` (file-task automation) — `servers/ana-filebot/`
- **CT 113** — `ana-wg` (WireGuard VPN) — `servers/ana-wg/`
`ana-ml2` (`10.250.50.54`) is a separate bare-metal Supermicro box,
`fv-ml1` (`10.251.50.54`) is a separate bare-metal Supermicro box,
not on this hypervisor. `sfsrv-ana` (`10.250.250.115`) runs on a
separate Dell R630, also not on this hypervisor.
@@ -54,4 +54,4 @@ The snapshot captures host/hardware/filesystem/network/ports but **not** VM-leve
## Placement rule
This is a hypervisor, not a target for Docker stacks. Add new workloads by standing up VMs here (or on ana-docker / ana-ml2, whichever fits) rather than deploying containers directly on the PVE host.
This is a hypervisor, not a target for Docker stacks. Add new workloads by standing up VMs here (or on ana-docker / fv-ml1, whichever fits) rather than deploying containers directly on the PVE host.
stacks/beszel/README.md
+8 -8
View File
@@ -2,7 +2,7 @@
Priority 2 completed 2026-09-11: pfi-postgres, esh-vm-db, pbs-ana and pbs-nh3
added with 16 alerts. Live samples verified, including both PBS datastores.
Fleet 17/18 up (known ana-ml2 outage). See `configs/beszel-agent/PRIORITY2.md`.
Fleet 17/18 up (known fv-ml1 outage). See `configs/beszel-agent/PRIORITY2.md`.
Hub: http://10.250.50.70:8090 (ana-docker), version 0.18.7 at the
2026-09-10 wiring. The hub also retains corviduo-dev's existing registration.
@@ -12,12 +12,12 @@ esh-pve and esh-pve-nas); 13 systems total and 20 additional alert rules.
See `configs/beszel-agent/README.md`. Subsequently nh3-nas was installed via
DSM Docker and registered with four alerts (14 registrations total). Recovery
verification at 2026-09-12 01:51Z: all six priority-1 hosts up, Synology live
filesystem samples verified; fleet 13/14 up with known ana-ml2 outage.
filesystem samples verified; fleet 13/14 up with known fv-ml1 outage.
| Host | Compose directory under `/opt/docker/compose/` | Additional filesystems |
|---|---|---|
| ana-docker | beszel | /mnt/backup |
| ana-ml2 | beszel-agent-ana | /tank, /home |
| fv-ml1 | beszel-agent-ana | /tank, /home |
| nh3-docker | beszel-agent-nh3 | none |
| esh-docker-vm (hub name esh-vm-docker) | beszel-agent-esh | /mnt/backup, /mnt/books |
| irv-ml1 | beszel-agent-irv | /worktank, /storetank, /mnt/smithy |
@@ -37,14 +37,14 @@ and `DEPLOY_SUDO=1` for root-owned directories. Example:
```sh
DEPLOY_SUDO=1 DEPLOY_DEST_STACK=beszel-agent-ana \
scripts/deploy-stack.sh infra-ops@10.250.50.54 beszel --compose
scripts/deploy-stack.sh infra-ops@10.251.50.54 beszel --compose
```
Agents use host-specific overrides selected by live `.env`:
```dotenv
COMPOSE_PROFILES=agent
COMPOSE_FILE=compose.yaml:hosts/ana-ml2.yaml
COMPOSE_FILE=compose.yaml:hosts/fv-ml1.yaml
BESZEL_EXTRA_FS=/extra-filesystems/tank,/extra-filesystems/home
```
@@ -71,7 +71,7 @@ hub public key. Never copy live tokens into version control.
## GPU telemetry
ana-ml2 and irv-ml1 use `henrygd/beszel-agent-nvidia:0.18.7` with NVIDIA
fv-ml1 and irv-ml1 use `henrygd/beszel-agent-nvidia:0.18.7` with NVIDIA
`utility` access to all GPUs. Both hosts already have NVIDIA Container Toolkit.
This collects per-card utilization, VRAM, temperature, and power draw without
changing the serving containers or GPU power limits. Verified hub samples
@@ -111,7 +111,7 @@ Thirty rules cover the seven hosts above under the existing operator user:
| CPU | >95% | 15 minutes |
| Memory | >90% | 10 minutes |
| Offline | down | 2 minutes |
| Temperature (ana-ml2 and irv-ml1) | >85 C | 5 minutes |
| Temperature (fv-ml1 and irv-ml1) | >85 C | 5 minutes |
CPU thresholds are sustained-load warnings; expected long-running compute may
need tuning. GPU utilization alone is not an alarm because busy GPUs are normal.
@@ -125,7 +125,7 @@ The bridge at `services/beszel-althing/` forwards through `postbox` to the
was replaced with this verified route. Miranda is a later cutover, not enabled.
See that service's README for operation and recipient changes.
Acceptance on 2026-09-10: ana-ml2 Disk was temporarily lowered to 1%/1 minute;
Acceptance on 2026-09-10: fv-ml1 Disk was temporarily lowered to 1%/1 minute;
the real alert reached althing at 15:29:45Z, thread
`01M25Z0WFDJM92GPTJQF769HJ7`. The threshold was then restored to 85%/5 minutes.
Verification uses `postbox thread`, which does not consume the inbox.
stacks/dockge/.env.example
+2 -2
View File
@@ -10,13 +10,13 @@ DOCKGE_VERSION=latest
DOCKGE_PORT=5001
# Human-readable host name for the homepage card, e.g.
# ana-docker, nh3-docker, esh-docker-vm, vm-esh-nas, ana-ml2
# ana-docker, nh3-docker, esh-docker-vm, vm-esh-nas, fv-ml1
DOCKGE_HOST_LABEL=
# IP (or hostname) the browser uses to reach this Dockge. Used in
# homepage.href — must be reachable from wherever you view homepage.
# ana-docker → 10.250.50.70
# ana-ml2 → 10.250.50.54
# fv-ml1 → 10.251.50.54
# nh3-docker → 10.100.50.40
# esh-docker-vm → 10.0.50.45
# vm-esh-nas → 10.0.50.154
stacks/erp-seat/compose.yaml
+4 -4
View File
@@ -1,4 +1,4 @@
# erp-seat — the RP seat on ana-ml2 GPU1. Serves the **MeroMero A4B MoE** NVFP4A16 quant
# erp-seat — the RP seat on fv-ml1 GPU1. Serves the **MeroMero A4B MoE** NVFP4A16 quant
# (G4-MeroMero-26B-A4B-it-uncensored-heretic) behind the gateway alias `char-rp-fast`.
#
# ⚠ THE STACK NAME IS HISTORICAL. It served Pfish-6 (the run-6 ERP-tune LoRA merge) until
@@ -109,7 +109,7 @@ services:
# SPEED (2026-09-09). ⚠ NVFP4A16 is WEIGHT-ONLY: activations stay 16-bit, so
# there is no FP4 GEMM to issue and the FlashInfer/CUTLASS NVFP4 MoE kernels
# do not apply — vLLM's "your GPU does not have native support for FP4"
# warning is about the SCHEME, not the card. ana-ml2 is sm_120 and DOES have
# warning is about the SCHEME, not the card. fv-ml1 is sm_120 and DOES have
# native FP4; a W4A4 build would use it, at a fidelity cost the operator
# already settled against for this seat. Marlin is the correct kernel here.
# PROVEN, not assumed: forcing moe_backend=flashinfer_cutedsl fails at engine
@@ -151,8 +151,8 @@ services:
- homepage.group=AI - Inference
- homepage.name=char-rp-fast (MeroMero 26B-A4B, NVFP4A16 MoE)
- homepage.icon=mdi-fire
- homepage.description=MeroMero A4B abliterated RP seat, NVFP4A16 weight-only, vision intact (ana-ml2 GPU1)
- homepage.href=http://10.250.50.54:${ERP_PORT:-8021}/docs
- homepage.description=MeroMero A4B abliterated RP seat, NVFP4A16 weight-only, vision intact (fv-ml1 GPU1)
- homepage.href=http://10.251.50.54:${ERP_PORT:-8021}/docs
networks:
tnet:
stacks/gemma4-charrp/README.md
+3 -3
View File
@@ -1,11 +1,11 @@
# gemma4-charrp — the char-rp seat (ana-ml2 GPU0)
# gemma4-charrp — the char-rp seat (fv-ml1 GPU0)
`google/gemma-4-26B-A4B-it`, NVFP4, serving both halves of the char-rp pair on
`:8016`. Replaced the dense `G4-MeroMero-v2-31B-NVFP4A16` seat on 2026-08-24.
```
char-rp non-thinking -> http://10.250.50.54:8016/v1
char-rp-reasoning thinking -> http://10.250.50.54:8016/v1
char-rp non-thinking -> http://10.251.50.54:8016/v1
char-rp-reasoning thinking -> http://10.251.50.54:8016/v1
```
Two LiteLLM aliases, **one backend**. They are not two seats — this trips people
stacks/gemma4-charrp/compose.yaml
+4 -4
View File
@@ -1,4 +1,4 @@
# gemma4-charrp — google/gemma-4-26B-A4B-it NVFP4 char-rp seat on ana-ml2 GPU0.
# gemma4-charrp — google/gemma-4-26B-A4B-it NVFP4 char-rp seat on fv-ml1 GPU0.
#
# Straight-across replacement for `meromero-charrp` (operator, 2026-08-24): same
# port, same served-model-names, so every gateway route and consumer is
@@ -46,7 +46,7 @@ services:
- --quantization
- compressed-tensors
# UNCHANGED FROM THE OUTGOING SEAT, ON PURPOSE. Both names are live
# LiteLLM routes at http://10.250.50.54:8016/v1 — `char-rp` ->
# LiteLLM routes at http://10.251.50.54:8016/v1 — `char-rp` ->
# hosted_vllm/char-rp and `char-rp-reasoning` -> hosted_vllm/char-rp-thinking.
# They are two aliases onto ONE seat, not two seats. Renaming either
# breaks the gateway silently.
@@ -128,8 +128,8 @@ services:
- homepage.group=AI - Inference
- homepage.name=char-rp (Gemma-4 26B-A4B NVFP4, MoE)
- homepage.icon=mdi-drama-masks
- homepage.description=gemma-4-26B-A4B-it NVFP4 MoE prose+CoT seat, 256K (ana-ml2 GPU0)
- homepage.href=http://10.250.50.54:${GEMMA4_PORT:-8016}/docs
- homepage.description=gemma-4-26B-A4B-it NVFP4 MoE prose+CoT seat, 256K (fv-ml1 GPU0)
- homepage.href=http://10.251.50.54:${GEMMA4_PORT:-8016}/docs
networks:
tnet:
stacks/heretic2-charrp-reasoning/README.md
+5 -5
View File
@@ -1,12 +1,12 @@
# heretic2-charrp-reasoning — NVFP4 + native MTP reasoning seat (ana-ml2)
# heretic2-charrp-reasoning — NVFP4 + native MTP reasoning seat (fv-ml1)
The `char-rp-reasoning` seat: NEO-CODE Heretic2 27B quantized to modelopt NVFP4
with a grafted BF16 MTP head, served by vLLM on **ana-ml2 GPU0**, port **8018**.
with a grafted BF16 MTP head, served by vLLM on **fv-ml1 GPU0**, port **8018**.
Replaces the retired GGUF seat (`llama-charrp-reasoning`) at roughly **77 tok/s
(~1.3x)** via `qwen3_5_mtp` speculative decode.
- **Served model name:** `char-rp-reasoning` — the gateway alias consumers use
- **Endpoint:** `http://10.250.50.54:8018` (`/docs` for the card link)
- **Endpoint:** `http://10.251.50.54:8018` (`/docs` for the card link)
## ⚠️ It does not boot without the MTP workaround
@@ -39,8 +39,8 @@ OOM the neighbours, not just this container.** Tunables are in `.env`; see
## Deploy
```bash
scripts/deploy-stack.sh ana-ml2 heretic2-charrp-reasoning
ssh infra-ops@10.250.50.54 \
scripts/deploy-stack.sh fv-ml1 heretic2-charrp-reasoning
ssh infra-ops@10.251.50.54 \
'cd /opt/docker/compose/heretic2-charrp-reasoning && sudo docker compose up -d'
```
stacks/homepage/README.md
+2 -2
View File
@@ -18,7 +18,7 @@ Two paths, and mixing them is the classic failure:
`homepage.group=` / `homepage.name=` / `homepage.icon=` / `homepage.description=`
/ `homepage.href=` labels and appears automatically. `conf/docker.yaml`
wires **five** engines over plaintext `:2375`: esh-docker-vm, ana-docker,
nh3-docker, ana-ml2, irv-ml1.
nh3-docker, fv-ml1, irv-ml1.
2. **Manual entries in `conf/services.yaml`** — for anything that is not a
labelled container on one of those five hosts: hardware, BMCs, hypervisors,
printers, and user-level systemd services (The Booth, Voice Design Studio).
@@ -63,7 +63,7 @@ scripts/elway infra-ops@10.250.50.70 --playbook playbooks/homepage-regroup-ana-d
scripts/elway infra-ops@10.0.50.45 --playbook playbooks/homepage-regroup-esh-docker-vm.yaml
scripts/elway infra-ops@10.100.50.40 --playbook playbooks/homepage-regroup-nh3-docker.yaml
scripts/elway infra-ops@10.100.79.3 --playbook playbooks/homepage-regroup-irv-ml1.yaml
scripts/elway infra-ops@10.250.50.54 --playbook playbooks/homepage-regroup-ana-ml2.yaml
scripts/elway infra-ops@10.251.50.54 --playbook playbooks/homepage-regroup-fv-ml1.yaml
```
⚠ **The GPU-backed model seats are deliberately still named `AI - Inference`,
stacks/homepage/conf/docker.yaml
+6 -6
View File
@@ -17,13 +17,13 @@ nh3-pfi-docker:
host: 10.100.50.40
port: 2375
# ana-ml2-docker — reached via a socat proxy on ana-docker (stacks/ana-ml2-proxy).
# ana-ml2's own :2375 is up but not reachable cross-site from ESH (a return-route
# gap on ana-ml2 itself; no SSH access from the ops seats to fix its routing).
# fv-ml1-docker — reached via a socat proxy on ana-docker (stacks/fv-ml1-proxy).
# fv-ml1's own :2375 is up but not reachable cross-site from ESH (a return-route
# gap on fv-ml1 itself; no SSH access from the ops seats to fix its routing).
# ana-docker is on the same Anaheim subnet and IS reachable from ESH, so it
# relays 10.250.50.70:2354 -> 10.250.50.54:2375. Point back at 10.250.50.54:2375
# and `docker compose down` the proxy once ana-ml2 has a return route to ESH.
ana-ml2-docker:
# relays 10.250.50.70:2354 -> 10.251.50.54:2375. Point back at 10.251.50.54:2375
# and `docker compose down` the proxy once fv-ml1 has a return route to ESH.
fv-ml1-docker:
host: 10.250.50.70
port: 2354
stacks/homepage/conf/services.yaml
+2 -2
View File
@@ -118,14 +118,14 @@
siteMonitor: http://10.250.50.70:5001
description: Docker VM (ana-docker, Dockge at :5001)
- PFI-ANA-ML2:
ping: 10.250.50.54
ping: 10.251.50.54
icon: mdi-brain
description: GPU host (bare-metal)
- PFI-ANA-ML2 BMC:
href: https://10.250.250.50
icon: mdi-brain
siteMonitor: https://10.250.250.50
description: BMC (ana-ml2)
description: BMC (fv-ml1)
- SFsrv-ANA:
href: https://10.250.250.115:8006
icon: si-proxmox
stacks/litellm/conf/config.yaml
+29 -28
View File
@@ -1,5 +1,5 @@
# LiteLLM gateway config — fronts the vLLM services on ana-ml2
# (10.250.50.54) and logs every request + response so they're
# (10.251.50.54) and logs every request + response so they're
# inspectable in the Logs UI at http://10.250.50.70:4000/ui.
#
# Deploys to /opt/docker/conf/litellm/config.yaml (mounted read-only
@@ -22,7 +22,7 @@ model_list:
# ⚠ ALIAS COLLISION — SEVERAL NAMES, ONE SET OF WEIGHTS
#
# As of 2026-08-23 these SEVEN aliases all resolve to the same backend
# (qwen3.8-27b-uncensored @ 10.250.50.54:8015):
# (qwen3.8-27b-uncensored @ 10.251.50.54:8015):
#
# chat-judge classifier gen image-judge
# qwen-image-bench summarizer summarizer-large
@@ -59,7 +59,7 @@ model_list:
# - model_name: granite-4.1-8b
# litellm_params:
# model: hosted_vllm/granite-4.1-8b
# api_base: http://10.250.50.54:8004/v1
# api_base: http://10.251.50.54:8004/v1
# api_key: os.environ/VLLM_API_KEY
# temperature: 0
# model_info:
@@ -71,7 +71,7 @@ model_list:
- model_name: summarizer
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.250.50.54:8015/v1
api_base: http://10.251.50.54:8015/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0
extra_body:
@@ -85,7 +85,7 @@ model_list:
- model_name: classifier
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.250.50.54:8015/v1
api_base: http://10.251.50.54:8015/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0
extra_body:
@@ -100,7 +100,7 @@ model_list:
- model_name: summarizer-large
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.250.50.54:8015/v1
api_base: http://10.251.50.54:8015/v1
api_key: os.environ/VLLM_API_KEY
presence_penalty: 1.5
temperature: 0.7
@@ -123,7 +123,7 @@ model_list:
- model_name: qwen-image-bench
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.250.50.54:8015/v1
api_base: http://10.251.50.54:8015/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0
top_p: 1.0
@@ -137,7 +137,7 @@ model_list:
- model_name: image-judge
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.250.50.54:8015/v1
api_base: http://10.251.50.54:8015/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0
top_p: 1.0
@@ -182,7 +182,7 @@ model_list:
- model_name: gen
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.250.50.54:8015/v1
api_base: http://10.251.50.54:8015/v1
api_key: os.environ/VLLM_API_KEY
presence_penalty: 1.5
temperature: 0.7
@@ -200,7 +200,7 @@ model_list:
# Distinct served-name so a thinking-off `gen` request can't mutate this deployment's
# enable_thinking (shared-config-mutation footgun). Same backend :8015, different model id.
model: hosted_vllm/qwen3.8-27b-uncensored-thinking
api_base: http://10.250.50.54:8015/v1
api_base: http://10.251.50.54:8015/v1
api_key: os.environ/VLLM_API_KEY
# CANONICAL Qwen3.8 THINKING sampling set (Qwen + unsloth "Best Practices"
# §1, identical in both): temperature=1.0, top_p=0.95, top_k=20,
@@ -248,7 +248,7 @@ model_list:
- model_name: char-rp
litellm_params:
model: hosted_vllm/char-rp
api_base: http://10.250.50.54:8016/v1
api_base: http://10.251.50.54:8016/v1
api_key: os.environ/VLLM_API_KEY
temperature: 1.1
top_p: 0.95
@@ -274,7 +274,7 @@ model_list:
- model_name: char-rp-reasoning
litellm_params:
model: hosted_vllm/char-rp-thinking
api_base: http://10.250.50.54:8016/v1
api_base: http://10.251.50.54:8016/v1
api_key: os.environ/VLLM_API_KEY
temperature: 1.1
top_p: 0.95
@@ -310,8 +310,8 @@ model_list:
# character card both sit at ~0%. Fable-Fusion is Heretic-abliterated.
#
# ROLLBACK (restores Dark-Scarlett):
# ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe && sudo docker compose down'
# ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && sudo docker compose up -d'
# ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe && sudo docker compose down'
# ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && sudo docker compose up -d'
# then revert this block to api_base :8018 / model hosted_vllm/char-rp-reasoning
# and restart litellm (~52s).
#
@@ -337,7 +337,7 @@ model_list:
- model_name: sec
litellm_params:
model: hosted_vllm/mog-sec-27b
api_base: http://10.250.50.54:8019/v1
api_base: http://10.251.50.54:8019/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0.7
top_p: 0.8
@@ -356,7 +356,7 @@ model_list:
- model_name: sec-reasoning
litellm_params:
model: hosted_vllm/mog-sec-27b-thinking
api_base: http://10.250.50.54:8019/v1
api_base: http://10.251.50.54:8019/v1
api_key: os.environ/VLLM_API_KEY
temperature: 1.0
top_p: 0.95
@@ -393,7 +393,7 @@ model_list:
- model_name: qwen3-embedding
litellm_params:
model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
api_base: http://10.250.50.54:8001/v1
api_base: http://10.251.50.54:8001/v1
api_key: os.environ/VLLM_API_KEY
model_info:
mode: embedding
@@ -423,7 +423,7 @@ model_list:
- model_name: chat-judge
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.250.50.54:8015/v1
api_base: http://10.251.50.54:8015/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0
top_p: 1.0
@@ -438,7 +438,7 @@ model_list:
- model_name: reranker
litellm_params:
model: hosted_vllm/BAAI/bge-reranker-v2-m3
api_base: http://10.250.50.54:8013/v1
api_base: http://10.251.50.54:8013/v1
api_key: os.environ/VLLM_API_KEY
model_info:
mode: rerank
@@ -456,7 +456,7 @@ model_list:
- model_name: coder-fast
litellm_params:
model: hosted_vllm/qwen2.5-coder-1.5b
api_base: http://10.250.50.54:8020/v1
api_base: http://10.251.50.54:8020/v1
api_key: os.environ/VLLM_API_KEY
model_info:
mode: completion
@@ -681,7 +681,7 @@ model_list:
- model_name: erp-tune-v2
litellm_params:
model: hosted_vllm/erp-tune-v2
api_base: http://10.250.50.54:8098/v1
api_base: http://10.251.50.54:8098/v1
api_key: os.environ/VLLM_API_KEY
# gemma4-26b-a4b-it-base -> the UNMODIFIED upstream instruct release, bf16
@@ -743,7 +743,7 @@ model_list:
- model_name: gemma4-26b-a4b-it-base
litellm_params:
model: hosted_vllm/gemma4-26b-a4b-it-base
api_base: http://10.250.50.54:8099/v1
api_base: http://10.251.50.54:8099/v1
api_key: os.environ/VLLM_API_KEY
# merged-run03 -> the in-house ERP/RP SFT, RUN 3, MERGED bf16 (:8099, vLLM, ana-ml2
@@ -848,7 +848,7 @@ model_list:
# - model_name: merged-run03
# litellm_params:
# model: hosted_vllm/merged-run03
# api_base: http://10.250.50.54:8099/v1
# api_base: http://10.251.50.54:8099/v1
# api_key: os.environ/VLLM_API_KEY
# trial -> ERP/RP SFT RUN 7, NVFP4A16 (weight-only) quant of the merged LoRA, served on
@@ -879,9 +879,10 @@ model_list:
#
# THIS GATEWAY IS THE SHARED-KEY SURFACE: `all-agents-local` reaches every model here,
# in every session and project. Removing this alias does not remove the operator's
# access -- the direct endpoint http://10.250.50.54:8021/v1 is unaffected.
# access -- the direct endpoint http://10.251.50.54:8021/v1 is unaffected.
#
# Same-site: seat and gateway are both at Anaheim (local hop, no mesh crossing).
# CROSS-SITE since 2026-09-12: gateway on ana-docker (Anaheim), seat on fv-ml1
# (Fountain Valley) -- metro hop over the headscale mesh, ~6ms.
# char-rp-fast — the MeroMero A4B MoE, on ana-ml2 GPU1 (:8021). Operator, 2026-09-10:
# "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias
@@ -919,7 +920,7 @@ model_list:
- model_name: char-rp-fast
litellm_params:
model: hosted_vllm/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16
api_base: http://10.250.50.54:8021/v1
api_base: http://10.251.50.54:8021/v1
api_key: os.environ/VLLM_API_KEY
extra_body:
min_p: 0.05
@@ -950,14 +951,14 @@ general_settings:
maximum_spend_logs_retention_period: "7d"
maximum_spend_logs_retention_interval: "1d"
# scalar-judge → Skywork-Reward-V2 (scalar reward model; vLLM pooling on
# ana-ml2:8003). LiteLLM has no reward/pooling MODE, so this is a passthrough,
# fv-ml1:8003). LiteLLM has no reward/pooling MODE, so this is a passthrough,
# not a model_list alias. Gateway-key-gated. Consumers POST the reward body to
# /scalar-judge/<route> (e.g. /pooling or /classify), forwarded to :8003.
# SWAP-SENSITIVE: a different reward model shifts the score scale, so consumers
# must recalibrate thresholds after a backing swap.
pass_through_endpoints:
- path: "/scalar-judge"
target: "http://10.250.50.54:8003"
target: "http://10.251.50.54:8003"
forward_headers: true
include_subpath: true
stacks/llama-swap/compose.yaml
+3 -3
View File
@@ -2,7 +2,7 @@
#
# Proxies OpenAI-compatible API requests to llama.cpp server instances
# and swaps which model is loaded into VRAM per request. Runs on
# ana-ml2 using both GPUs dynamically (no explicit device pinning —
# fv-ml1 using both GPUs dynamically (no explicit device pinning —
# llama-swap picks per-model-definition).
#
# Model definitions live in /opt/docker/conf/llama-swap/config.yaml on
@@ -47,8 +47,8 @@ services:
- homepage.group=AI Systems
- homepage.name=llama-swap
- homepage.icon=mdi-swap-horizontal
- homepage.description=GGUF model swapper (llama.cpp; ana-ml2)
- homepage.href=http://10.250.50.54:${LLAMA_SWAP_PORT}
- homepage.description=GGUF model swapper (llama.cpp; fv-ml1)
- homepage.href=http://10.251.50.54:${LLAMA_SWAP_PORT}
networks:
tnet:
stacks/meromero-charrp/compose.yaml
+3 -3
View File
@@ -1,5 +1,5 @@
# meromero-charrp — G4-MeroMero-v2-31B NVFP4A16 char-rp PROSE seat (non-thinking, MULTIMODAL)
# on ana-ml2 GPU0. Replaces the retired Magidonia GGUF seat (char-rp-gguf / llama-charrp).
# on fv-ml1 GPU0. Replaces the retired Magidonia GGUF seat (char-rp-gguf / llama-charrp).
#
# Gemma-4 dense but with SLIDING-WINDOW attention -> KV-efficient, serves the full native 256K
# at ~2x concurrency. Vision enabled: preprocessor_config.json was materialized from
@@ -93,8 +93,8 @@ services:
- homepage.group=AI - Dormant
- homepage.name=char-rp (MeroMero-v2 NVFP4) — rollback
- homepage.icon=mdi-drama-masks
- homepage.description=Superseded 2026-08-24 by the Gemma-4 26B-A4B MoE seat; dense 31B NVFP4A16, kept for rollback (ana-ml2 GPU0)
- homepage.href=http://10.250.50.54:${MEROMERO_PORT:-8016}/docs
- homepage.description=Superseded 2026-08-24 by the Gemma-4 26B-A4B MoE seat; dense 31B NVFP4A16, kept for rollback (fv-ml1 GPU0)
- homepage.href=http://10.251.50.54:${MEROMERO_PORT:-8016}/docs
networks:
tnet:
stacks/sglang/compose.yaml
+3 -3
View File
@@ -1,4 +1,4 @@
# SGLang — alternative LLM serving engine, stood up on ana-ml2 to bench
# SGLang — alternative LLM serving engine, stood up on fv-ml1 to bench
# head-to-head against vLLM on the same model + hardware.
#
# Bench-oriented (not yet a permanent service): point it at the SAME checkpoint
@@ -68,8 +68,8 @@ services:
- homepage.group=AI Systems
- homepage.name=SGLang (bench)
- homepage.icon=mdi-speedometer
- homepage.description=SGLang serving — vLLM bench comparison (ana-ml2)
- homepage.href=http://10.250.50.54:${SGLANG_PORT}
- homepage.description=SGLang serving — vLLM bench comparison (fv-ml1)
- homepage.href=http://10.251.50.54:${SGLANG_PORT}
networks:
tnet: