fv-ml1: complete the cutover — rename, renumber, DNS, and the LiteLLM repoint
The box is physically at Fountain Valley, renamed, renumbered onto 10.251/16, and serving inference again. This lands the repo half of that. Host: hostname ana-ml2 -> fv-ml1, pinned to 10.251.50.54 by a dnsmasq reservation so the address the runbook, DNS and LiteLLM all assume is the address it actually has. Its headscale node is renamed too. The sweep ran from scripts/fv-ml1-rename-sweep.sh, whose allowlist is the reason this diff touches current-state files and not the record. Dated persistent-memory entries, archival-memory and incident notes still say ana-ml2 in 31 and 62 places respectively, because that is what the box was when those things happened. Rewriting them would make the history lie. LiteLLM was the load-bearing piece and needed more than the api_base sed the runbook describes. Twenty api_base entries repointed, but a grep-and-verify pass also caught a LIVE pass_through_endpoints target for the scalar-judge reward route still on the old address -- an api_base-only substitution would have left it dead. Four prose references describing current state were repointed as well; one historical note recording where a hand-test was run is deliberately left pointing at 10.250.50.54. Two facts in the server tables were wrong and are corrected here. The site is Fountain Valley, not Anaheim. And the box has FOUR RTX PRO 6000 Blackwell Max-Q, not two -- verified by nvidia-smi -L and independently by PCI enumeration of four GB202GL devices. That is 391 GB of VRAM rather than 196, which changes what fits on it. DNS: fv-ml1, fv-ml1-bmc and fv-gw added under the fv site via the piggyback approach, scriberr re-homed, and the ana-ml2 records removed. Applied to all three resolvers. The BMC record carries a warning that its 802.1q VLAN tag must stay disabled -- it shipped tagging VLAN 250 into an untagged port, which made it invisible to every network-side diagnostic and is the reason it appeared dead through several cable changes. Verified end to end: summarizer and sec both answer through the Anaheim gateway across the mesh to FV seats on different ports.
This commit is contained in:
@@ -199,7 +199,7 @@ Observed and standardized across servers:
|
|||||||
- **Restart policy:** `restart: unless-stopped` for daemons
|
- **Restart policy:** `restart: unless-stopped` for daemons
|
||||||
- **Homepage labels** on user-facing services. The dashboard runs on
|
- **Homepage labels** on user-facing services. The dashboard runs on
|
||||||
`esh-docker-vm` and reads the Docker API of **every** host in
|
`esh-docker-vm` and reads the Docker API of **every** host in
|
||||||
`stacks/homepage/conf/docker.yaml` (ana-docker, ana-ml2, nh3-docker,
|
`stacks/homepage/conf/docker.yaml` (ana-docker, fv-ml1, nh3-docker,
|
||||||
irv-ml1, esh-docker-vm), so a labelled container is discovered from
|
irv-ml1, esh-docker-vm), so a labelled container is discovered from
|
||||||
wherever it runs — you do not add it to `services.yaml` as well. Doing both
|
wherever it runs — you do not add it to `services.yaml` as well. Doing both
|
||||||
renders it twice.
|
renders it twice.
|
||||||
@@ -229,7 +229,7 @@ Observed and standardized across servers:
|
|||||||
|
|
||||||
| Name | IP | Site | Role | Details |
|
| Name | IP | Site | Role | Details |
|
||||||
|------|-----|------|------|---------|
|
|------|-----|------|------|---------|
|
||||||
| ana-ml2 | 10.250.50.54 | Anaheim (`10.250.0.0/16`) | GPU / AI inference (bare metal, dual RTX PRO 6000 Blackwell Max-Q, 96 GB each) | `servers/ana-ml2/README.md` |
|
| fv-ml1 | 10.251.50.54 | Fountain Valley (`10.251.0.0/16`) | GPU / AI inference (bare metal, **4× RTX PRO 6000 Blackwell Max-Q, 96 GB each = 391 GB VRAM**) | `servers/fv-ml1/README.md` |
|
||||||
| irv-ml1 | 100.64.0.6 (headscale mesh); 10.6.110.50 (Irvine LAN, routed via mesh) | Irvine — via headscale mesh; wg0 lifeline (10.100.79.3) dormant post-cutover | GPU / AI inference (bare metal, RTX 3090 + RTX A6000, native stacks) | `servers/irv-ml1/README.md` |
|
| irv-ml1 | 100.64.0.6 (headscale mesh); 10.6.110.50 (Irvine LAN, routed via mesh) | Irvine — via headscale mesh; wg0 lifeline (10.100.79.3) dormant post-cutover | GPU / AI inference (bare metal, RTX 3090 + RTX A6000, native stacks) | `servers/irv-ml1/README.md` |
|
||||||
| ana-docker | 10.250.50.70 | Anaheim | General-purpose Docker host (non-GPU VM on pfi-pve) | `servers/ana-docker/README.md` |
|
| ana-docker | 10.250.50.70 | Anaheim | General-purpose Docker host (non-GPU VM on pfi-pve) | `servers/ana-docker/README.md` |
|
||||||
| pfi-ana-webhost | 10.250.50.52 | Anaheim | VM on pfi-pve (VMID 110) — web workload | `servers/pfi-ana-webhost/README.md` |
|
| pfi-ana-webhost | 10.250.50.52 | Anaheim | VM on pfi-pve (VMID 110) — web workload | `servers/pfi-ana-webhost/README.md` |
|
||||||
@@ -257,7 +257,7 @@ Observed and standardized across servers:
|
|||||||
| esh-vm-db | 10.0.50.60 | ESH home lab | DB VM — PostgreSQL (paperless-ng) + MongoDB; bare-metal VM, no Docker | `servers/esh-vm-db/README.md` |
|
| esh-vm-db | 10.0.50.60 | ESH home lab | DB VM — PostgreSQL (paperless-ng) + MongoDB; bare-metal VM, no Docker | `servers/esh-vm-db/README.md` |
|
||||||
|
|
||||||
**Placement rules:**
|
**Placement rules:**
|
||||||
- GPU-required stacks → `ana-ml2` (primary, Anaheim) or `irv-ml1`
|
- GPU-required stacks → `fv-ml1` (primary, Fountain Valley) or `irv-ml1`
|
||||||
(secondary, Irvine — bigger VRAM ceiling at 72 GB total). Access
|
(secondary, Irvine — bigger VRAM ceiling at 72 GB total). Access
|
||||||
to `irv-ml1` is via the headscale mesh (the wg0 tunnel lifeline is
|
to `irv-ml1` is via the headscale mesh (the wg0 tunnel lifeline is
|
||||||
dormant post-cutover; `10.100.79.3` no longer answers).
|
dormant post-cutover; `10.100.79.3` no longer answers).
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ See **[CLAUDE.md](CLAUDE.md)** for the full set of conventions and the rules Cla
|
|||||||
|
|
||||||
| Host | IP | Site | Role |
|
| Host | IP | Site | Role |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| ana-ml2 | `10.250.50.54` | Anaheim (`10.250.0.0/16`) | GPU / AI inference (bare metal) |
|
| fv-ml1 | `10.251.50.54` | Fountain Valley (`10.251.0.0/16`) | GPU / AI inference (bare metal, 4× RTX PRO 6000 Blackwell Max-Q) |
|
||||||
| ana-docker | `10.250.50.70` | Anaheim | General-purpose Docker + cross-site hubs (VM on pfi-pve) |
|
| ana-docker | `10.250.50.70` | Anaheim | General-purpose Docker + cross-site hubs (VM on pfi-pve) |
|
||||||
| nh3-docker | `10.100.50.40` | NH3 (`10.100.0.0/16`) | General-purpose Docker (VM on nh3-pve) |
|
| nh3-docker | `10.100.50.40` | NH3 (`10.100.0.0/16`) | General-purpose Docker (VM on nh3-pve) |
|
||||||
| esh-docker-vm | `10.0.50.45` | ESH home lab (`esteban.net`) | Home-lab Docker (VM on esh-pve, non-PFI scope) |
|
| esh-docker-vm | `10.0.50.45` | ESH home lab (`esteban.net`) | Home-lab Docker (VM on esh-pve, non-PFI scope) |
|
||||||
@@ -64,7 +64,7 @@ Per-host snapshots of the running system live under `servers/<host>/system-detai
|
|||||||
|
|
||||||
## Current stacks
|
## Current stacks
|
||||||
|
|
||||||
**GPU (ana-ml2):**
|
**GPU (fv-ml1):**
|
||||||
- `llama-swap` — GGUF model swapper via llama.cpp (port 9292)
|
- `llama-swap` — GGUF model swapper via llama.cpp (port 9292)
|
||||||
- `vllm` — embeddings (8001) + reranker (8002) + Skywork reward classifier (8003) via vLLM
|
- `vllm` — embeddings (8001) + reranker (8002) + Skywork reward classifier (8003) via vLLM
|
||||||
|
|
||||||
@@ -76,7 +76,7 @@ Per-host snapshots of the running system live under `servers/<host>/system-detai
|
|||||||
- Fleet services: `beszel` (metrics hub, port 8090), `dozzle-hub` (log viewer, 8088), `backrest` (restic UI, 9898)
|
- Fleet services: `beszel` (metrics hub, port 8090), `dozzle-hub` (log viewer, 8088), `backrest` (restic UI, 9898)
|
||||||
- Backup target: `rest-server-ana` on port 8000
|
- Backup target: `rest-server-ana` on port 8000
|
||||||
|
|
||||||
**GPU ana-ml2 (non-canonical for now):**
|
**GPU fv-ml1 (non-canonical for now):**
|
||||||
- `comfyui`, `kokoro`, `parakeet`, `vibevoice` alongside the canonical `llama-swap` + `vllm`
|
- `comfyui`, `kokoro`, `parakeet`, `vibevoice` alongside the canonical `llama-swap` + `vllm`
|
||||||
|
|
||||||
**NH3 (nh3-docker):**
|
**NH3 (nh3-docker):**
|
||||||
@@ -170,7 +170,7 @@ Goal: **every Docker host + configs + every database** covered, not just VM imag
|
|||||||
|---|---|
|
|---|---|
|
||||||
| VM-level (Proxmox vzdump) | ✅ All running guests covered across pfi-pve / nh3-pve / esh-pve-nas; esh-pve has VMID 108 uncovered |
|
| VM-level (Proxmox vzdump) | ✅ All running guests covered across pfi-pve / nh3-pve / esh-pve-nas; esh-pve has VMID 108 uncovered |
|
||||||
| ana-docker restic (host files + DBs) | ✅ `configs/restic/ana-docker/` with pre-backup hooks for synapse / seafile / vaultwarden-pg / gitea (native dump) / openwebui |
|
| ana-docker restic (host files + DBs) | ✅ `configs/restic/ana-docker/` with pre-backup hooks for synapse / seafile / vaultwarden-pg / gitea (native dump) / openwebui |
|
||||||
| ana-ml2 restic | ✅ `configs/restic/ana-ml2/` — bare-metal host files (no DB hooks needed) |
|
| fv-ml1 restic | ✅ `configs/restic/fv-ml1/` — bare-metal host files (no DB hooks needed) |
|
||||||
| nh3-docker restic | ✅ Light — no DB hooks needed |
|
| nh3-docker restic | ✅ Light — no DB hooks needed |
|
||||||
| esh-docker-vm restic | ✅ With DB hooks for paperless-postgres (external), home-assistant + pgadmin + uptime-kuma (host-side sqlite3), calibre-web-automated (in-container sqlite3) |
|
| esh-docker-vm restic | ✅ With DB hooks for paperless-postgres (external), home-assistant + pgadmin + uptime-kuma (host-side sqlite3), calibre-web-automated (in-container sqlite3) |
|
||||||
| vm-esh-nas restic | ✅ Light — NFS mounts explicitly excluded |
|
| vm-esh-nas restic | ✅ Light — NFS mounts explicitly excluded |
|
||||||
|
|||||||
+5
-3
@@ -54,7 +54,8 @@ sites:
|
|||||||
hosts:
|
hosts:
|
||||||
# ---- ana: Anaheim colo ----
|
# ---- ana: Anaheim colo ----
|
||||||
- {name: ana-docker, site: ana, v4: 10.250.50.70, note: general-purpose docker host}
|
- {name: ana-docker, site: ana, v4: 10.250.50.70, note: general-purpose docker host}
|
||||||
- {name: ana-ml2, site: ana, v4: 10.250.50.54, note: GPU inference, dual RTX PRO 6000}
|
# ---- fv: Fountain Valley colo (piggyback DNS until FV has its own resolver) ----
|
||||||
|
- {name: fv-ml1, site: fv, v4: 10.251.50.54, note: GPU inference, 4x RTX PRO 6000 Blackwell Max-Q (ex ana-ml2, relocated 2026-09-12)}
|
||||||
- {name: ana-nas, site: ana, v4: 10.250.50.50, note: CT109 on pfi-pve — NFS/SMB}
|
- {name: ana-nas, site: ana, v4: 10.250.50.50, note: CT109 on pfi-pve — NFS/SMB}
|
||||||
- {name: ana-filebot, site: ana, v4: 10.250.50.53, note: file-task automation}
|
- {name: ana-filebot, site: ana, v4: 10.250.50.53, note: file-task automation}
|
||||||
- {name: ana-wg, site: ana, v4: 10.250.50.252, note: WireGuard host}
|
- {name: ana-wg, site: ana, v4: 10.250.50.252, note: WireGuard host}
|
||||||
@@ -67,7 +68,8 @@ hosts:
|
|||||||
- {name: pfi-pve, site: ana, v4: 10.250.250.31, note: Proxmox hypervisor}
|
- {name: pfi-pve, site: ana, v4: 10.250.250.31, note: Proxmox hypervisor}
|
||||||
- {name: ana-gw, site: ana, v4: 10.250.0.1, note: FortiGate-80F edge}
|
- {name: ana-gw, site: ana, v4: 10.250.0.1, note: FortiGate-80F edge}
|
||||||
- {name: pfi-pve-idrac, site: ana, v4: 10.250.250.30, note: iDRAC — OOB for pfi-pve}
|
- {name: pfi-pve-idrac, site: ana, v4: 10.250.250.30, note: iDRAC — OOB for pfi-pve}
|
||||||
- {name: ana-ml2-bmc, site: ana, v4: 10.250.250.50, note: BMC for ana-ml2}
|
- {name: fv-ml1-bmc, site: fv, v4: 10.251.250.50, note: BMC for fv-ml1 (Supermicro/ATEN; 802.1q VLAN tag must stay DISABLED)}
|
||||||
|
- {name: fv-gw, site: fv, v4: 10.251.50.1, note: OPNsense gateway vb-gateway, mesh subnet-router for 10.251.0.0/16; public 172.83.89.66 = fv.phasefinal.com}
|
||||||
# SureFire tenant hardware — PFI-managed under the hosting agreement.
|
# SureFire tenant hardware — PFI-managed under the hosting agreement.
|
||||||
- {name: sfsrv-ana, site: ana, v4: 10.250.250.115, note: SureFire tenant hypervisor}
|
- {name: sfsrv-ana, site: ana, v4: 10.250.250.115, note: SureFire tenant hypervisor}
|
||||||
- {name: sf-ana-container, site: ana, v4: 10.250.150.100, note: SureFire tenant container host}
|
- {name: sf-ana-container, site: ana, v4: 10.250.150.100, note: SureFire tenant container host}
|
||||||
@@ -120,4 +122,4 @@ aliases:
|
|||||||
- {name: booth, site: nh3, target: nh3-dev, note: The Booth :8090}
|
- {name: booth, site: nh3, target: nh3-dev, note: The Booth :8090}
|
||||||
- {name: wherethef, site: nh3, target: nh3-dev, note: WhereTF :8093}
|
- {name: wherethef, site: nh3, target: nh3-dev, note: WhereTF :8093}
|
||||||
- {name: homepage, site: esh, target: esh-docker-vm, note: fleet dashboard :5100}
|
- {name: homepage, site: esh, target: esh-docker-vm, note: fleet dashboard :5100}
|
||||||
- {name: scriberr, site: ana, target: ana-ml2, note: transcription + diarization :8080 (GPU1)}
|
- {name: scriberr, site: fv, target: fv-ml1, note: transcription + diarization :8080 (GPU1)}
|
||||||
|
|||||||
@@ -131,7 +131,7 @@ group memberships) is in
|
|||||||
[`stacks/llama-swap/conf/config.yaml`](../../stacks/llama-swap/conf/config.yaml).
|
[`stacks/llama-swap/conf/config.yaml`](../../stacks/llama-swap/conf/config.yaml).
|
||||||
Don't mirror the table here — it drifted last time and immediately
|
Don't mirror the table here — it drifted last time and immediately
|
||||||
went stale. Live HTTP listing:
|
went stale. Live HTTP listing:
|
||||||
`curl http://10.250.50.54:9292/v1/models`.
|
`curl http://10.251.50.54:9292/v1/models`.
|
||||||
|
|
||||||
**Model Groups**:
|
**Model Groups**:
|
||||||
|
|
||||||
|
|||||||
@@ -73,5 +73,5 @@ Reverse this for a graceful host shutdown (done 2026-09-05 for the PERC swap).
|
|||||||
|
|
||||||
## Not on this hypervisor
|
## Not on this hypervisor
|
||||||
|
|
||||||
`ana-ml2` (10.250.50.54) is bare-metal Supermicro; `sfsrv-ana` (10.250.250.115) is the
|
`fv-ml1` (10.251.50.54) is bare-metal Supermicro; `sfsrv-ana` (10.250.250.115) is the
|
||||||
SureFire client's own Dell R630; `pfi-gx10` is at NH3.
|
SureFire client's own Dell R630; `pfi-gx10` is at NH3.
|
||||||
|
|||||||
@@ -18,9 +18,9 @@ implementation.
|
|||||||
|
|
||||||
**CLEARED (green):**
|
**CLEARED (green):**
|
||||||
- Execute Brokkr's reranker-selection recommendations unattended.
|
- Execute Brokkr's reranker-selection recommendations unattended.
|
||||||
- Bring **down the prod reranker** at `ana-ml2:8002` (qwen3-reranker-0.6B) —
|
- Bring **down the prod reranker** at `fv-ml1:8002` (qwen3-reranker-0.6B) —
|
||||||
**temporarily OR permanently**.
|
**temporarily OR permanently**.
|
||||||
- Down **ONE** of the RP (roleplay) seats on ana-ml2 **temporarily** to free
|
- Down **ONE** of the RP (roleplay) seats on fv-ml1 **temporarily** to free
|
||||||
GPU/VRAM for testing.
|
GPU/VRAM for testing.
|
||||||
- Temporarily clear space for the smoke/bench.
|
- Temporarily clear space for the smoke/bench.
|
||||||
- Pull models, stand up side-port vLLM benches, run the harness — whatever the
|
- Pull models, stand up side-port vLLM benches, run the harness — whatever the
|
||||||
@@ -30,7 +30,7 @@ implementation.
|
|||||||
- **NO permanent deletion of anything** (no `rm`/`docker volume rm`/model-weight
|
- **NO permanent deletion of anything** (no `rm`/`docker volume rm`/model-weight
|
||||||
deletion/data destruction). Downing ≠ deleting.
|
deletion/data destruction). Downing ≠ deleting.
|
||||||
- **NO taking anything else offline** beyond (a) the prod reranker and (b) ONE
|
- **NO taking anything else offline** beyond (a) the prod reranker and (b) ONE
|
||||||
ana-ml2 RP seat. (Not granite/embed/reward/coder/gen/a second RP seat/muninn/etc.)
|
fv-ml1 RP seat. (Not granite/embed/reward/coder/gen/a second RP seat/muninn/etc.)
|
||||||
- **NO rebooting machines.**
|
- **NO rebooting machines.**
|
||||||
|
|
||||||
**Process:** accumulate assumptions here; operator reverses at the end.
|
**Process:** accumulate assumptions here; operator reverses at the end.
|
||||||
@@ -54,7 +54,7 @@ implementation.
|
|||||||
|
|
||||||
## Current board at handoff
|
## Current board at handoff
|
||||||
|
|
||||||
- **Prod reranker:** `ana-ml2:8002` = `vllm-rerank` (Qwen/Qwen3-Reranker-0.6B),
|
- **Prod reranker:** `fv-ml1:8002` = `vllm-rerank` (Qwen/Qwen3-Reranker-0.6B),
|
||||||
reverted to baseline `classifier_from_token:["no","yes"]`, healthy. Compose:
|
reverted to baseline `classifier_from_token:["no","yes"]`, healthy. Compose:
|
||||||
`/opt/docker/compose/vllm/compose.yaml` (canonical mirror
|
`/opt/docker/compose/vllm/compose.yaml` (canonical mirror
|
||||||
`stacks/vllm/compose.yaml`). Gateway alias `reranker`/`qwen3-reranker` →
|
`stacks/vllm/compose.yaml`). Gateway alias `reranker`/`qwen3-reranker` →
|
||||||
@@ -75,7 +75,7 @@ implementation.
|
|||||||
worldtree-dev signals the lever run is done).
|
worldtree-dev signals the lever run is done).
|
||||||
- **Dual query-set requirement (hard):** score bare-name anchor queries AND
|
- **Dual query-set requirement (hard):** score bare-name anchor queries AND
|
||||||
question-shaped; bar = recovering the name-lookup class.
|
question-shaped; bar = recovering the name-lookup class.
|
||||||
- **VRAM:** 4B ≈ 4–5 GB fp8, 8B ≈ 9 GB; ana-ml2 Blackwell has headroom.
|
- **VRAM:** 4B ≈ 4–5 GB fp8, 8B ≈ 9 GB; fv-ml1 Blackwell has headroom.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -84,12 +84,12 @@ implementation.
|
|||||||
### 2026-08-06 — A2 brought up (Brokkr thread 01KZBSTSJA…)
|
### 2026-08-06 — A2 brought up (Brokkr thread 01KZBSTSJA…)
|
||||||
|
|
||||||
- **Backend:** `vllm-rerank-a2` — standalone `docker run` (NOT in the vllm compose
|
- **Backend:** `vllm-rerank-a2` — standalone `docker run` (NOT in the vllm compose
|
||||||
stack), on ana-ml2 **GPU1**, host port **:8012** → container 8000. Image
|
stack), on fv-ml1 **GPU1**, host port **:8012** → container 8000. Image
|
||||||
`vllm/vllm-openai:latest` (=0.24.0). Args: model
|
`vllm/vllm-openai:latest` (=0.24.0). Args: model
|
||||||
`tomaarsen/Qwen3-Reranker-0.6B-seq-cls`, `--runner pooling`, `--gpu-memory-utilization
|
`tomaarsen/Qwen3-Reranker-0.6B-seq-cls`, `--runner pooling`, `--gpu-memory-utilization
|
||||||
0.03`, `--max-model-len 8192`, `--dtype auto`, `--restart no`. Native
|
0.03`, `--max-model-len 8192`, `--dtype auto`, `--restart no`. Native
|
||||||
`Qwen3ForSequenceClassification` — NO hf-overrides. Routes /rerank /score /classify.
|
`Qwen3ForSequenceClassification` — NO hf-overrides. Routes /rerank /score /classify.
|
||||||
- **Gateway alias:** `reranker-a2-qwen3-seqcls` → `http://10.250.50.54:8012/v1`,
|
- **Gateway alias:** `reranker-a2-qwen3-seqcls` → `http://10.251.50.54:8012/v1`,
|
||||||
mode rerank. Added via LiteLLM **`/model/new`** (DB-backed, `store_model_in_db:true`)
|
mode rerank. Added via LiteLLM **`/model/new`** (DB-backed, `store_model_in_db:true`)
|
||||||
— **no gateway restart** (respects the "nothing else offline" line). Verified 200
|
— **no gateway restart** (respects the "nothing else offline" line). Verified 200
|
||||||
through the gateway.
|
through the gateway.
|
||||||
@@ -106,7 +106,7 @@ implementation.
|
|||||||
- D4 — did NOT down any RP seat (A2 is 0.6B / 3.5 GB; no VRAM pressure).
|
- D4 — did NOT down any RP seat (A2 is 0.6B / 3.5 GB; no VRAM pressure).
|
||||||
|
|
||||||
**Cleanup for A2 (run at end / on reversal):**
|
**Cleanup for A2 (run at end / on reversal):**
|
||||||
- `ssh infra-ops@10.250.50.54 'sudo docker stop vllm-rerank-a2 && sudo docker rm vllm-rerank-a2'`
|
- `ssh infra-ops@10.251.50.54 'sudo docker stop vllm-rerank-a2 && sudo docker rm vllm-rerank-a2'`
|
||||||
- Delete gateway alias: `POST /model/delete {"id": <model_id>}` (id via `/model/info?model_name=reranker-a2-qwen3-seqcls`), infra-ops admin key. (DB-persisted, so it survives a restart — must be explicitly deleted.)
|
- Delete gateway alias: `POST /model/delete {"id": <model_id>}` (id via `/model/info?model_name=reranker-a2-qwen3-seqcls`), infra-ops admin key. (DB-persisted, so it survives a restart — must be explicitly deleted.)
|
||||||
- No weights deleted (red line); HF cache under /tank/aimodels/huggingface retains the 0.6B-seq-cls download.
|
- No weights deleted (red line); HF cache under /tank/aimodels/huggingface retains the 0.6B-seq-cls download.
|
||||||
|
|
||||||
@@ -114,11 +114,11 @@ implementation.
|
|||||||
|
|
||||||
### 2026-08-06 — A3 + A4 pre-staged (Brokkr said pre-stage in parallel, hold A5)
|
### 2026-08-06 — A3 + A4 pre-staged (Brokkr said pre-stage in parallel, hold A5)
|
||||||
|
|
||||||
- **A3** `vllm-rerank-a3` — ana-ml2 GPU1 :8013, `BAAI/bge-reranker-v2-m3`
|
- **A3** `vllm-rerank-a3` — fv-ml1 GPU1 :8013, `BAAI/bge-reranker-v2-m3`
|
||||||
(XLMRobertaForSequenceClassification), same run pattern, util 0.03. VRAM ≈ **2.3 GB**.
|
(XLMRobertaForSequenceClassification), same run pattern, util 0.03. VRAM ≈ **2.3 GB**.
|
||||||
Latency (20-doc, ~1500-char, shared GPU1): single p50 **105 ms**; 8-conc p50 214 ms, ~34 req/s.
|
Latency (20-doc, ~1500-char, shared GPU1): single p50 **105 ms**; 8-conc p50 214 ms, ~34 req/s.
|
||||||
Gateway alias `reranker-a3-bge-v2-m3` via /model/new (200, verified).
|
Gateway alias `reranker-a3-bge-v2-m3` via /model/new (200, verified).
|
||||||
- **A4** `vllm-rerank-a4` — ana-ml2 GPU1 :8014, `Alibaba-NLP/gte-reranker-modernbert-base`
|
- **A4** `vllm-rerank-a4` — fv-ml1 GPU1 :8014, `Alibaba-NLP/gte-reranker-modernbert-base`
|
||||||
(ModernBertForSequenceClassification), util 0.02. VRAM ≈ **1.4 GB**. Latency: single
|
(ModernBertForSequenceClassification), util 0.02. VRAM ≈ **1.4 GB**. Latency: single
|
||||||
p50 **102 ms**; 8-conc p50 153 ms, ~51 req/s. Gateway alias `reranker-a4-gte-modernbert`
|
p50 **102 ms**; 8-conc p50 153 ms, ~51 req/s. Gateway alias `reranker-a4-gte-modernbert`
|
||||||
via /model/new (200, verified).
|
via /model/new (200, verified).
|
||||||
@@ -129,7 +129,7 @@ implementation.
|
|||||||
If A5 (4B, ~4–5 GB) is greenlit: fits GPU1 tight or GPU0 (~9 GB free) — no RP-seat downing expected.
|
If A5 (4B, ~4–5 GB) is greenlit: fits GPU1 tight or GPU0 (~9 GB free) — no RP-seat downing expected.
|
||||||
|
|
||||||
**Cleanup for A3/A4 (same pattern as A2):** `docker stop/rm vllm-rerank-a3 vllm-rerank-a4`
|
**Cleanup for A3/A4 (same pattern as A2):** `docker stop/rm vllm-rerank-a3 vllm-rerank-a4`
|
||||||
on ana-ml2; `/model/delete` the two aliases (DB-persisted); weights retained in HF cache.
|
on fv-ml1; `/model/delete` the two aliases (DB-persisted); weights retained in HF cache.
|
||||||
|
|
||||||
### 2026-08-06 — A2 verdict (Brokkr full grid): training-prior confirmed
|
### 2026-08-06 — A2 verdict (Brokkr full grid): training-prior confirmed
|
||||||
|
|
||||||
@@ -202,7 +202,7 @@ on ana-ml2; `/model/delete` the two aliases (DB-persisted); weights retained in
|
|||||||
Brokkr signals the bake-off is closed (`docker stop/rm` + `/model/delete`).
|
Brokkr signals the bake-off is closed (`docker stop/rm` + `/model/delete`).
|
||||||
- **A4 (`vllm-rerank-a4` :8014)** + alias — KEEP for now (Brokkr's documented throughput fallback).
|
- **A4 (`vllm-rerank-a4` :8014)** + alias — KEEP for now (Brokkr's documented throughput fallback).
|
||||||
- **A3 (`vllm-rerank-a3` :8013)** — now PRODUCTION (backs the `reranker` alias). Hardened
|
- **A3 (`vllm-rerank-a3` :8013)** — now PRODUCTION (backs the `reranker` alias). Hardened
|
||||||
2026-08-06: `docker update --restart unless-stopped` (survives ana-ml2 reboot, no recreate).
|
2026-08-06: `docker update --restart unless-stopped` (survives fv-ml1 reboot, no recreate).
|
||||||
A4 given the same. **Remaining follow-up (not urgent): promote A3 from throwaway `docker run`
|
A4 given the same. **Remaining follow-up (not urgent): promote A3 from throwaway `docker run`
|
||||||
to a canonical compose service** (`stacks/vllm/`) for config-managed consistency — a recreate,
|
to a canonical compose service** (`stacks/vllm/`) for config-managed consistency — a recreate,
|
||||||
so do it in a window since it briefly drops `reranker`.
|
so do it in a window since it briefly drops `reranker`.
|
||||||
|
|||||||
@@ -37,7 +37,7 @@ pfi-pve during morning backup window. Backup suspected as trigger
|
|||||||
|
|
||||||
**Blast radius:**
|
**Blast radius:**
|
||||||
- ~~pfi-postgres (VM 105) — PGDATA on `/mnt/db`~~ — **migrated to local disk 2026-04-23**. vaultwarden, gitea, paperless-ng, zammad no longer cascade on ana-nas outage. Left in history for the recovery pre-migration.
|
- ~~pfi-postgres (VM 105) — PGDATA on `/mnt/db`~~ — **migrated to local disk 2026-04-23**. vaultwarden, gitea, paperless-ng, zammad no longer cascade on ana-nas outage. Left in history for the recovery pre-migration.
|
||||||
- ana-docker rest-server-ana — repo data on `/mnt/backup` → all ana-side restic clients fail (ana-docker, ana-ml2, esh-docker-vm, vm-esh-nas)
|
- ana-docker rest-server-ana — repo data on `/mnt/backup` → all ana-side restic clients fail (ana-docker, fv-ml1, esh-docker-vm, vm-esh-nas)
|
||||||
- PBS-ANA datastore — NFS-backed on `/mnt/backup/pbs-ana` → fleet vzdumps fail, PBS-NH3 sync fails
|
- PBS-ANA datastore — NFS-backed on `/mnt/backup/pbs-ana` → fleet vzdumps fail, PBS-NH3 sync fails
|
||||||
- ana-docker NFS mounts for `/mnt/docker`, `/mnt/compose`, `/mnt/pve-VMStorage` if used → various stack misbehavior
|
- ana-docker NFS mounts for `/mnt/docker`, `/mnt/compose`, `/mnt/pve-VMStorage` if used → various stack misbehavior
|
||||||
|
|
||||||
@@ -207,12 +207,12 @@ Essentially **all Anaheim primary services** go offline. Because ana-nas lives h
|
|||||||
|
|
||||||
## Tier 4 — Specialty workloads
|
## Tier 4 — Specialty workloads
|
||||||
|
|
||||||
### ana-ml2 (bare metal Supermicro, 10.250.50.54, BMC 10.250.250.50)
|
### fv-ml1 (bare metal Supermicro, 10.251.50.54, BMC 10.251.250.50)
|
||||||
|
|
||||||
**Blast radius:** AI inference services (llama-swap, vllm). Consumer-facing chat/embedding/reward-scoring endpoints fail.
|
**Blast radius:** AI inference services (llama-swap, vllm). Consumer-facing chat/embedding/reward-scoring endpoints fail.
|
||||||
|
|
||||||
**Recovery:**
|
**Recovery:**
|
||||||
1. Check OS via SSH. If unresponsive, BMC console at <https://10.250.250.50>.
|
1. Check OS via SSH. If unresponsive, BMC console at <https://10.251.250.50>.
|
||||||
2. If hardware issue: BMC logs, power cycle via IPMI, check GPU health (`nvidia-smi`).
|
2. If hardware issue: BMC logs, power cycle via IPMI, check GPU health (`nvidia-smi`).
|
||||||
3. Docker stacks auto-start via compose `restart: unless-stopped`.
|
3. Docker stacks auto-start via compose `restart: unless-stopped`.
|
||||||
|
|
||||||
|
|||||||
@@ -49,7 +49,7 @@ path the fleet lacked on 2026-09-11. Do NOT tie the mesh node to fv-ml1 itself.
|
|||||||
- [ ] `dns/internal.yaml` — add the two records below (piggyback; NOT a `sites:` entry), comment out the old `ana-ml2` / `ana-ml2-bmc` lines. Then `scripts/dns-sync.py --dry-run` → apply.
|
- [ ] `dns/internal.yaml` — add the two records below (piggyback; NOT a `sites:` entry), comment out the old `ana-ml2` / `ana-ml2-bmc` lines. Then `scripts/dns-sync.py --dry-run` → apply.
|
||||||
```yaml
|
```yaml
|
||||||
# ---- fv: Fountain Valley colo (piggyback DNS until FV has its own resolver) ----
|
# ---- fv: Fountain Valley colo (piggyback DNS until FV has its own resolver) ----
|
||||||
- {name: fv-ml1, site: fv, v4: 10.251.50.54, note: GPU inference, dual RTX PRO 6000 (ex ana-ml2, relocated 2026-09)}
|
- {name: fv-ml1, site: fv, v4: 10.251.50.54, note: GPU inference, 4x RTX PRO 6000 Blackwell Max-Q (ex ana-ml2, relocated 2026-09)}
|
||||||
- {name: fv-ml1-bmc, site: fv, v4: 10.251.250.50, note: BMC for fv-ml1}
|
- {name: fv-ml1-bmc, site: fv, v4: 10.251.250.50, note: BMC for fv-ml1}
|
||||||
```
|
```
|
||||||
- [ ] mesh route to 10.251.0.0/16 approved in headscale (OPNsense subnet-router).
|
- [ ] mesh route to 10.251.0.0/16 approved in headscale (OPNsense subnet-router).
|
||||||
|
|||||||
@@ -1 +0,0 @@
|
|||||||
10.250.50.54
|
|
||||||
@@ -1,13 +1,13 @@
|
|||||||
# ana-ml2
|
# fv-ml1
|
||||||
|
|
||||||
Primary AI inference host for PFI.
|
Primary AI inference host for PFI.
|
||||||
|
|
||||||
## Network
|
## Network
|
||||||
|
|
||||||
- **LAN IP:** 10.250.50.54 (in-band, OS-side)
|
- **LAN IP:** 10.251.50.54 (in-band, OS-side)
|
||||||
- **BMC (OOB):** 10.250.250.50 — Supermicro IPMI web UI
|
- **BMC (OOB):** 10.251.250.50 — Supermicro IPMI web UI
|
||||||
at <https://10.250.250.50> (homepage card: *PFI-ANA-ML2 BMC*)
|
at <https://10.251.250.50> (homepage card: *PFI-ANA-ML2 BMC*)
|
||||||
- **SSH:** standard port 22 on 10.250.50.54
|
- **SSH:** standard port 22 on 10.251.50.54
|
||||||
|
|
||||||
## Hardware
|
## Hardware
|
||||||
|
|
||||||
@@ -54,9 +54,9 @@ history, not an active fault, and the 2 CKSUM were the stale-block artefact of t
|
|||||||
replace on growth** (`zpool replace tank nvme7n1 <new>`; any PM1725b 1.6 TB or
|
replace on growth** (`zpool replace tank nvme7n1 <new>`; any PM1725b 1.6 TB or
|
||||||
larger). Slot 0-5 itself deserves a reseat / cable check at the next hands-on
|
larger). Slot 0-5 itself deserves a reseat / cable check at the next hands-on
|
||||||
visit — a bay that dropped a drive for 4½ months is the likelier fault than the
|
visit — a bay that dropped a drive for 4½ months is the likelier fault than the
|
||||||
drive. Playbook: `playbooks/ana-ml2-pool-health.yaml` (idempotent; rerunning is a
|
drive. Playbook: `playbooks/fv-ml1-pool-health.yaml` (idempotent; rerunning is a
|
||||||
no-op). ⚠ **Nothing alerts on this** — see the open follow-up in
|
no-op). ⚠ **Nothing alerts on this** — see the open follow-up in
|
||||||
`persistent-memory.d/2026-09-09-ana-ml2-pool-actions-done.md`.
|
`persistent-memory.d/2026-09-09-fv-ml1-pool-actions-done.md`.
|
||||||
|
|
||||||
## Key paths
|
## Key paths
|
||||||
|
|
||||||
@@ -119,7 +119,7 @@ seats, safe to leave: `mistral-medium-3.5`, `mistral-small-4(-heretic)`,
|
|||||||
## Refresh state
|
## Refresh state
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
scripts/refresh-server-info.sh ana-ml2
|
scripts/refresh-server-info.sh fv-ml1
|
||||||
```
|
```
|
||||||
|
|
||||||
Latest snapshot: `system-details.txt` (regenerate as needed).
|
Latest snapshot: `system-details.txt` (regenerate as needed).
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
10.251.50.54
|
||||||
@@ -40,7 +40,7 @@ hypervisor):
|
|||||||
- **CT 112** — `ana-filebot` (file-task automation) — `servers/ana-filebot/`
|
- **CT 112** — `ana-filebot` (file-task automation) — `servers/ana-filebot/`
|
||||||
- **CT 113** — `ana-wg` (WireGuard VPN) — `servers/ana-wg/`
|
- **CT 113** — `ana-wg` (WireGuard VPN) — `servers/ana-wg/`
|
||||||
|
|
||||||
`ana-ml2` (`10.250.50.54`) is a separate bare-metal Supermicro box,
|
`fv-ml1` (`10.251.50.54`) is a separate bare-metal Supermicro box,
|
||||||
not on this hypervisor. `sfsrv-ana` (`10.250.250.115`) runs on a
|
not on this hypervisor. `sfsrv-ana` (`10.250.250.115`) runs on a
|
||||||
separate Dell R630, also not on this hypervisor.
|
separate Dell R630, also not on this hypervisor.
|
||||||
|
|
||||||
@@ -54,4 +54,4 @@ The snapshot captures host/hardware/filesystem/network/ports but **not** VM-leve
|
|||||||
|
|
||||||
## Placement rule
|
## Placement rule
|
||||||
|
|
||||||
This is a hypervisor, not a target for Docker stacks. Add new workloads by standing up VMs here (or on ana-docker / ana-ml2, whichever fits) rather than deploying containers directly on the PVE host.
|
This is a hypervisor, not a target for Docker stacks. Add new workloads by standing up VMs here (or on ana-docker / fv-ml1, whichever fits) rather than deploying containers directly on the PVE host.
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
Priority 2 completed 2026-09-11: pfi-postgres, esh-vm-db, pbs-ana and pbs-nh3
|
Priority 2 completed 2026-09-11: pfi-postgres, esh-vm-db, pbs-ana and pbs-nh3
|
||||||
added with 16 alerts. Live samples verified, including both PBS datastores.
|
added with 16 alerts. Live samples verified, including both PBS datastores.
|
||||||
Fleet 17/18 up (known ana-ml2 outage). See `configs/beszel-agent/PRIORITY2.md`.
|
Fleet 17/18 up (known fv-ml1 outage). See `configs/beszel-agent/PRIORITY2.md`.
|
||||||
|
|
||||||
Hub: http://10.250.50.70:8090 (ana-docker), version 0.18.7 at the
|
Hub: http://10.250.50.70:8090 (ana-docker), version 0.18.7 at the
|
||||||
2026-09-10 wiring. The hub also retains corviduo-dev's existing registration.
|
2026-09-10 wiring. The hub also retains corviduo-dev's existing registration.
|
||||||
@@ -12,12 +12,12 @@ esh-pve and esh-pve-nas); 13 systems total and 20 additional alert rules.
|
|||||||
See `configs/beszel-agent/README.md`. Subsequently nh3-nas was installed via
|
See `configs/beszel-agent/README.md`. Subsequently nh3-nas was installed via
|
||||||
DSM Docker and registered with four alerts (14 registrations total). Recovery
|
DSM Docker and registered with four alerts (14 registrations total). Recovery
|
||||||
verification at 2026-09-12 01:51Z: all six priority-1 hosts up, Synology live
|
verification at 2026-09-12 01:51Z: all six priority-1 hosts up, Synology live
|
||||||
filesystem samples verified; fleet 13/14 up with known ana-ml2 outage.
|
filesystem samples verified; fleet 13/14 up with known fv-ml1 outage.
|
||||||
|
|
||||||
| Host | Compose directory under `/opt/docker/compose/` | Additional filesystems |
|
| Host | Compose directory under `/opt/docker/compose/` | Additional filesystems |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| ana-docker | beszel | /mnt/backup |
|
| ana-docker | beszel | /mnt/backup |
|
||||||
| ana-ml2 | beszel-agent-ana | /tank, /home |
|
| fv-ml1 | beszel-agent-ana | /tank, /home |
|
||||||
| nh3-docker | beszel-agent-nh3 | none |
|
| nh3-docker | beszel-agent-nh3 | none |
|
||||||
| esh-docker-vm (hub name esh-vm-docker) | beszel-agent-esh | /mnt/backup, /mnt/books |
|
| esh-docker-vm (hub name esh-vm-docker) | beszel-agent-esh | /mnt/backup, /mnt/books |
|
||||||
| irv-ml1 | beszel-agent-irv | /worktank, /storetank, /mnt/smithy |
|
| irv-ml1 | beszel-agent-irv | /worktank, /storetank, /mnt/smithy |
|
||||||
@@ -37,14 +37,14 @@ and `DEPLOY_SUDO=1` for root-owned directories. Example:
|
|||||||
|
|
||||||
```sh
|
```sh
|
||||||
DEPLOY_SUDO=1 DEPLOY_DEST_STACK=beszel-agent-ana \
|
DEPLOY_SUDO=1 DEPLOY_DEST_STACK=beszel-agent-ana \
|
||||||
scripts/deploy-stack.sh infra-ops@10.250.50.54 beszel --compose
|
scripts/deploy-stack.sh infra-ops@10.251.50.54 beszel --compose
|
||||||
```
|
```
|
||||||
|
|
||||||
Agents use host-specific overrides selected by live `.env`:
|
Agents use host-specific overrides selected by live `.env`:
|
||||||
|
|
||||||
```dotenv
|
```dotenv
|
||||||
COMPOSE_PROFILES=agent
|
COMPOSE_PROFILES=agent
|
||||||
COMPOSE_FILE=compose.yaml:hosts/ana-ml2.yaml
|
COMPOSE_FILE=compose.yaml:hosts/fv-ml1.yaml
|
||||||
BESZEL_EXTRA_FS=/extra-filesystems/tank,/extra-filesystems/home
|
BESZEL_EXTRA_FS=/extra-filesystems/tank,/extra-filesystems/home
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -71,7 +71,7 @@ hub public key. Never copy live tokens into version control.
|
|||||||
|
|
||||||
## GPU telemetry
|
## GPU telemetry
|
||||||
|
|
||||||
ana-ml2 and irv-ml1 use `henrygd/beszel-agent-nvidia:0.18.7` with NVIDIA
|
fv-ml1 and irv-ml1 use `henrygd/beszel-agent-nvidia:0.18.7` with NVIDIA
|
||||||
`utility` access to all GPUs. Both hosts already have NVIDIA Container Toolkit.
|
`utility` access to all GPUs. Both hosts already have NVIDIA Container Toolkit.
|
||||||
This collects per-card utilization, VRAM, temperature, and power draw without
|
This collects per-card utilization, VRAM, temperature, and power draw without
|
||||||
changing the serving containers or GPU power limits. Verified hub samples
|
changing the serving containers or GPU power limits. Verified hub samples
|
||||||
@@ -111,7 +111,7 @@ Thirty rules cover the seven hosts above under the existing operator user:
|
|||||||
| CPU | >95% | 15 minutes |
|
| CPU | >95% | 15 minutes |
|
||||||
| Memory | >90% | 10 minutes |
|
| Memory | >90% | 10 minutes |
|
||||||
| Offline | down | 2 minutes |
|
| Offline | down | 2 minutes |
|
||||||
| Temperature (ana-ml2 and irv-ml1) | >85 C | 5 minutes |
|
| Temperature (fv-ml1 and irv-ml1) | >85 C | 5 minutes |
|
||||||
|
|
||||||
CPU thresholds are sustained-load warnings; expected long-running compute may
|
CPU thresholds are sustained-load warnings; expected long-running compute may
|
||||||
need tuning. GPU utilization alone is not an alarm because busy GPUs are normal.
|
need tuning. GPU utilization alone is not an alarm because busy GPUs are normal.
|
||||||
@@ -125,7 +125,7 @@ The bridge at `services/beszel-althing/` forwards through `postbox` to the
|
|||||||
was replaced with this verified route. Miranda is a later cutover, not enabled.
|
was replaced with this verified route. Miranda is a later cutover, not enabled.
|
||||||
See that service's README for operation and recipient changes.
|
See that service's README for operation and recipient changes.
|
||||||
|
|
||||||
Acceptance on 2026-09-10: ana-ml2 Disk was temporarily lowered to 1%/1 minute;
|
Acceptance on 2026-09-10: fv-ml1 Disk was temporarily lowered to 1%/1 minute;
|
||||||
the real alert reached althing at 15:29:45Z, thread
|
the real alert reached althing at 15:29:45Z, thread
|
||||||
`01M25Z0WFDJM92GPTJQF769HJ7`. The threshold was then restored to 85%/5 minutes.
|
`01M25Z0WFDJM92GPTJQF769HJ7`. The threshold was then restored to 85%/5 minutes.
|
||||||
Verification uses `postbox thread`, which does not consume the inbox.
|
Verification uses `postbox thread`, which does not consume the inbox.
|
||||||
|
|||||||
@@ -10,13 +10,13 @@ DOCKGE_VERSION=latest
|
|||||||
DOCKGE_PORT=5001
|
DOCKGE_PORT=5001
|
||||||
|
|
||||||
# Human-readable host name for the homepage card, e.g.
|
# Human-readable host name for the homepage card, e.g.
|
||||||
# ana-docker, nh3-docker, esh-docker-vm, vm-esh-nas, ana-ml2
|
# ana-docker, nh3-docker, esh-docker-vm, vm-esh-nas, fv-ml1
|
||||||
DOCKGE_HOST_LABEL=
|
DOCKGE_HOST_LABEL=
|
||||||
|
|
||||||
# IP (or hostname) the browser uses to reach this Dockge. Used in
|
# IP (or hostname) the browser uses to reach this Dockge. Used in
|
||||||
# homepage.href — must be reachable from wherever you view homepage.
|
# homepage.href — must be reachable from wherever you view homepage.
|
||||||
# ana-docker → 10.250.50.70
|
# ana-docker → 10.250.50.70
|
||||||
# ana-ml2 → 10.250.50.54
|
# fv-ml1 → 10.251.50.54
|
||||||
# nh3-docker → 10.100.50.40
|
# nh3-docker → 10.100.50.40
|
||||||
# esh-docker-vm → 10.0.50.45
|
# esh-docker-vm → 10.0.50.45
|
||||||
# vm-esh-nas → 10.0.50.154
|
# vm-esh-nas → 10.0.50.154
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
# erp-seat — the RP seat on ana-ml2 GPU1. Serves the **MeroMero A4B MoE** NVFP4A16 quant
|
# erp-seat — the RP seat on fv-ml1 GPU1. Serves the **MeroMero A4B MoE** NVFP4A16 quant
|
||||||
# (G4-MeroMero-26B-A4B-it-uncensored-heretic) behind the gateway alias `char-rp-fast`.
|
# (G4-MeroMero-26B-A4B-it-uncensored-heretic) behind the gateway alias `char-rp-fast`.
|
||||||
#
|
#
|
||||||
# ⚠ THE STACK NAME IS HISTORICAL. It served Pfish-6 (the run-6 ERP-tune LoRA merge) until
|
# ⚠ THE STACK NAME IS HISTORICAL. It served Pfish-6 (the run-6 ERP-tune LoRA merge) until
|
||||||
@@ -109,7 +109,7 @@ services:
|
|||||||
# SPEED (2026-09-09). ⚠ NVFP4A16 is WEIGHT-ONLY: activations stay 16-bit, so
|
# SPEED (2026-09-09). ⚠ NVFP4A16 is WEIGHT-ONLY: activations stay 16-bit, so
|
||||||
# there is no FP4 GEMM to issue and the FlashInfer/CUTLASS NVFP4 MoE kernels
|
# there is no FP4 GEMM to issue and the FlashInfer/CUTLASS NVFP4 MoE kernels
|
||||||
# do not apply — vLLM's "your GPU does not have native support for FP4"
|
# do not apply — vLLM's "your GPU does not have native support for FP4"
|
||||||
# warning is about the SCHEME, not the card. ana-ml2 is sm_120 and DOES have
|
# warning is about the SCHEME, not the card. fv-ml1 is sm_120 and DOES have
|
||||||
# native FP4; a W4A4 build would use it, at a fidelity cost the operator
|
# native FP4; a W4A4 build would use it, at a fidelity cost the operator
|
||||||
# already settled against for this seat. Marlin is the correct kernel here.
|
# already settled against for this seat. Marlin is the correct kernel here.
|
||||||
# PROVEN, not assumed: forcing moe_backend=flashinfer_cutedsl fails at engine
|
# PROVEN, not assumed: forcing moe_backend=flashinfer_cutedsl fails at engine
|
||||||
@@ -151,8 +151,8 @@ services:
|
|||||||
- homepage.group=AI - Inference
|
- homepage.group=AI - Inference
|
||||||
- homepage.name=char-rp-fast (MeroMero 26B-A4B, NVFP4A16 MoE)
|
- homepage.name=char-rp-fast (MeroMero 26B-A4B, NVFP4A16 MoE)
|
||||||
- homepage.icon=mdi-fire
|
- homepage.icon=mdi-fire
|
||||||
- homepage.description=MeroMero A4B abliterated RP seat, NVFP4A16 weight-only, vision intact (ana-ml2 GPU1)
|
- homepage.description=MeroMero A4B abliterated RP seat, NVFP4A16 weight-only, vision intact (fv-ml1 GPU1)
|
||||||
- homepage.href=http://10.250.50.54:${ERP_PORT:-8021}/docs
|
- homepage.href=http://10.251.50.54:${ERP_PORT:-8021}/docs
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
tnet:
|
tnet:
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
# gemma4-charrp — the char-rp seat (ana-ml2 GPU0)
|
# gemma4-charrp — the char-rp seat (fv-ml1 GPU0)
|
||||||
|
|
||||||
`google/gemma-4-26B-A4B-it`, NVFP4, serving both halves of the char-rp pair on
|
`google/gemma-4-26B-A4B-it`, NVFP4, serving both halves of the char-rp pair on
|
||||||
`:8016`. Replaced the dense `G4-MeroMero-v2-31B-NVFP4A16` seat on 2026-08-24.
|
`:8016`. Replaced the dense `G4-MeroMero-v2-31B-NVFP4A16` seat on 2026-08-24.
|
||||||
|
|
||||||
```
|
```
|
||||||
char-rp non-thinking -> http://10.250.50.54:8016/v1
|
char-rp non-thinking -> http://10.251.50.54:8016/v1
|
||||||
char-rp-reasoning thinking -> http://10.250.50.54:8016/v1
|
char-rp-reasoning thinking -> http://10.251.50.54:8016/v1
|
||||||
```
|
```
|
||||||
|
|
||||||
Two LiteLLM aliases, **one backend**. They are not two seats — this trips people
|
Two LiteLLM aliases, **one backend**. They are not two seats — this trips people
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
# gemma4-charrp — google/gemma-4-26B-A4B-it NVFP4 char-rp seat on ana-ml2 GPU0.
|
# gemma4-charrp — google/gemma-4-26B-A4B-it NVFP4 char-rp seat on fv-ml1 GPU0.
|
||||||
#
|
#
|
||||||
# Straight-across replacement for `meromero-charrp` (operator, 2026-08-24): same
|
# Straight-across replacement for `meromero-charrp` (operator, 2026-08-24): same
|
||||||
# port, same served-model-names, so every gateway route and consumer is
|
# port, same served-model-names, so every gateway route and consumer is
|
||||||
@@ -46,7 +46,7 @@ services:
|
|||||||
- --quantization
|
- --quantization
|
||||||
- compressed-tensors
|
- compressed-tensors
|
||||||
# UNCHANGED FROM THE OUTGOING SEAT, ON PURPOSE. Both names are live
|
# UNCHANGED FROM THE OUTGOING SEAT, ON PURPOSE. Both names are live
|
||||||
# LiteLLM routes at http://10.250.50.54:8016/v1 — `char-rp` ->
|
# LiteLLM routes at http://10.251.50.54:8016/v1 — `char-rp` ->
|
||||||
# hosted_vllm/char-rp and `char-rp-reasoning` -> hosted_vllm/char-rp-thinking.
|
# hosted_vllm/char-rp and `char-rp-reasoning` -> hosted_vllm/char-rp-thinking.
|
||||||
# They are two aliases onto ONE seat, not two seats. Renaming either
|
# They are two aliases onto ONE seat, not two seats. Renaming either
|
||||||
# breaks the gateway silently.
|
# breaks the gateway silently.
|
||||||
@@ -128,8 +128,8 @@ services:
|
|||||||
- homepage.group=AI - Inference
|
- homepage.group=AI - Inference
|
||||||
- homepage.name=char-rp (Gemma-4 26B-A4B NVFP4, MoE)
|
- homepage.name=char-rp (Gemma-4 26B-A4B NVFP4, MoE)
|
||||||
- homepage.icon=mdi-drama-masks
|
- homepage.icon=mdi-drama-masks
|
||||||
- homepage.description=gemma-4-26B-A4B-it NVFP4 MoE prose+CoT seat, 256K (ana-ml2 GPU0)
|
- homepage.description=gemma-4-26B-A4B-it NVFP4 MoE prose+CoT seat, 256K (fv-ml1 GPU0)
|
||||||
- homepage.href=http://10.250.50.54:${GEMMA4_PORT:-8016}/docs
|
- homepage.href=http://10.251.50.54:${GEMMA4_PORT:-8016}/docs
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
tnet:
|
tnet:
|
||||||
|
|||||||
@@ -1,12 +1,12 @@
|
|||||||
# heretic2-charrp-reasoning — NVFP4 + native MTP reasoning seat (ana-ml2)
|
# heretic2-charrp-reasoning — NVFP4 + native MTP reasoning seat (fv-ml1)
|
||||||
|
|
||||||
The `char-rp-reasoning` seat: NEO-CODE Heretic2 27B quantized to modelopt NVFP4
|
The `char-rp-reasoning` seat: NEO-CODE Heretic2 27B quantized to modelopt NVFP4
|
||||||
with a grafted BF16 MTP head, served by vLLM on **ana-ml2 GPU0**, port **8018**.
|
with a grafted BF16 MTP head, served by vLLM on **fv-ml1 GPU0**, port **8018**.
|
||||||
Replaces the retired GGUF seat (`llama-charrp-reasoning`) at roughly **77 tok/s
|
Replaces the retired GGUF seat (`llama-charrp-reasoning`) at roughly **77 tok/s
|
||||||
(~1.3x)** via `qwen3_5_mtp` speculative decode.
|
(~1.3x)** via `qwen3_5_mtp` speculative decode.
|
||||||
|
|
||||||
- **Served model name:** `char-rp-reasoning` — the gateway alias consumers use
|
- **Served model name:** `char-rp-reasoning` — the gateway alias consumers use
|
||||||
- **Endpoint:** `http://10.250.50.54:8018` (`/docs` for the card link)
|
- **Endpoint:** `http://10.251.50.54:8018` (`/docs` for the card link)
|
||||||
|
|
||||||
## ⚠️ It does not boot without the MTP workaround
|
## ⚠️ It does not boot without the MTP workaround
|
||||||
|
|
||||||
@@ -39,8 +39,8 @@ OOM the neighbours, not just this container.** Tunables are in `.env`; see
|
|||||||
## Deploy
|
## Deploy
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
scripts/deploy-stack.sh ana-ml2 heretic2-charrp-reasoning
|
scripts/deploy-stack.sh fv-ml1 heretic2-charrp-reasoning
|
||||||
ssh infra-ops@10.250.50.54 \
|
ssh infra-ops@10.251.50.54 \
|
||||||
'cd /opt/docker/compose/heretic2-charrp-reasoning && sudo docker compose up -d'
|
'cd /opt/docker/compose/heretic2-charrp-reasoning && sudo docker compose up -d'
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
@@ -18,7 +18,7 @@ Two paths, and mixing them is the classic failure:
|
|||||||
`homepage.group=` / `homepage.name=` / `homepage.icon=` / `homepage.description=`
|
`homepage.group=` / `homepage.name=` / `homepage.icon=` / `homepage.description=`
|
||||||
/ `homepage.href=` labels and appears automatically. `conf/docker.yaml`
|
/ `homepage.href=` labels and appears automatically. `conf/docker.yaml`
|
||||||
wires **five** engines over plaintext `:2375`: esh-docker-vm, ana-docker,
|
wires **five** engines over plaintext `:2375`: esh-docker-vm, ana-docker,
|
||||||
nh3-docker, ana-ml2, irv-ml1.
|
nh3-docker, fv-ml1, irv-ml1.
|
||||||
2. **Manual entries in `conf/services.yaml`** — for anything that is not a
|
2. **Manual entries in `conf/services.yaml`** — for anything that is not a
|
||||||
labelled container on one of those five hosts: hardware, BMCs, hypervisors,
|
labelled container on one of those five hosts: hardware, BMCs, hypervisors,
|
||||||
printers, and user-level systemd services (The Booth, Voice Design Studio).
|
printers, and user-level systemd services (The Booth, Voice Design Studio).
|
||||||
@@ -63,7 +63,7 @@ scripts/elway infra-ops@10.250.50.70 --playbook playbooks/homepage-regroup-ana-d
|
|||||||
scripts/elway infra-ops@10.0.50.45 --playbook playbooks/homepage-regroup-esh-docker-vm.yaml
|
scripts/elway infra-ops@10.0.50.45 --playbook playbooks/homepage-regroup-esh-docker-vm.yaml
|
||||||
scripts/elway infra-ops@10.100.50.40 --playbook playbooks/homepage-regroup-nh3-docker.yaml
|
scripts/elway infra-ops@10.100.50.40 --playbook playbooks/homepage-regroup-nh3-docker.yaml
|
||||||
scripts/elway infra-ops@10.100.79.3 --playbook playbooks/homepage-regroup-irv-ml1.yaml
|
scripts/elway infra-ops@10.100.79.3 --playbook playbooks/homepage-regroup-irv-ml1.yaml
|
||||||
scripts/elway infra-ops@10.250.50.54 --playbook playbooks/homepage-regroup-ana-ml2.yaml
|
scripts/elway infra-ops@10.251.50.54 --playbook playbooks/homepage-regroup-fv-ml1.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
⚠ **The GPU-backed model seats are deliberately still named `AI - Inference`,
|
⚠ **The GPU-backed model seats are deliberately still named `AI - Inference`,
|
||||||
|
|||||||
@@ -17,13 +17,13 @@ nh3-pfi-docker:
|
|||||||
host: 10.100.50.40
|
host: 10.100.50.40
|
||||||
port: 2375
|
port: 2375
|
||||||
|
|
||||||
# ana-ml2-docker — reached via a socat proxy on ana-docker (stacks/ana-ml2-proxy).
|
# fv-ml1-docker — reached via a socat proxy on ana-docker (stacks/fv-ml1-proxy).
|
||||||
# ana-ml2's own :2375 is up but not reachable cross-site from ESH (a return-route
|
# fv-ml1's own :2375 is up but not reachable cross-site from ESH (a return-route
|
||||||
# gap on ana-ml2 itself; no SSH access from the ops seats to fix its routing).
|
# gap on fv-ml1 itself; no SSH access from the ops seats to fix its routing).
|
||||||
# ana-docker is on the same Anaheim subnet and IS reachable from ESH, so it
|
# ana-docker is on the same Anaheim subnet and IS reachable from ESH, so it
|
||||||
# relays 10.250.50.70:2354 -> 10.250.50.54:2375. Point back at 10.250.50.54:2375
|
# relays 10.250.50.70:2354 -> 10.251.50.54:2375. Point back at 10.251.50.54:2375
|
||||||
# and `docker compose down` the proxy once ana-ml2 has a return route to ESH.
|
# and `docker compose down` the proxy once fv-ml1 has a return route to ESH.
|
||||||
ana-ml2-docker:
|
fv-ml1-docker:
|
||||||
host: 10.250.50.70
|
host: 10.250.50.70
|
||||||
port: 2354
|
port: 2354
|
||||||
|
|
||||||
|
|||||||
@@ -118,14 +118,14 @@
|
|||||||
siteMonitor: http://10.250.50.70:5001
|
siteMonitor: http://10.250.50.70:5001
|
||||||
description: Docker VM (ana-docker, Dockge at :5001)
|
description: Docker VM (ana-docker, Dockge at :5001)
|
||||||
- PFI-ANA-ML2:
|
- PFI-ANA-ML2:
|
||||||
ping: 10.250.50.54
|
ping: 10.251.50.54
|
||||||
icon: mdi-brain
|
icon: mdi-brain
|
||||||
description: GPU host (bare-metal)
|
description: GPU host (bare-metal)
|
||||||
- PFI-ANA-ML2 BMC:
|
- PFI-ANA-ML2 BMC:
|
||||||
href: https://10.250.250.50
|
href: https://10.250.250.50
|
||||||
icon: mdi-brain
|
icon: mdi-brain
|
||||||
siteMonitor: https://10.250.250.50
|
siteMonitor: https://10.250.250.50
|
||||||
description: BMC (ana-ml2)
|
description: BMC (fv-ml1)
|
||||||
- SFsrv-ANA:
|
- SFsrv-ANA:
|
||||||
href: https://10.250.250.115:8006
|
href: https://10.250.250.115:8006
|
||||||
icon: si-proxmox
|
icon: si-proxmox
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
# LiteLLM gateway config — fronts the vLLM services on ana-ml2
|
# LiteLLM gateway config — fronts the vLLM services on ana-ml2
|
||||||
# (10.250.50.54) and logs every request + response so they're
|
# (10.251.50.54) and logs every request + response so they're
|
||||||
# inspectable in the Logs UI at http://10.250.50.70:4000/ui.
|
# inspectable in the Logs UI at http://10.250.50.70:4000/ui.
|
||||||
#
|
#
|
||||||
# Deploys to /opt/docker/conf/litellm/config.yaml (mounted read-only
|
# Deploys to /opt/docker/conf/litellm/config.yaml (mounted read-only
|
||||||
@@ -22,7 +22,7 @@ model_list:
|
|||||||
# ⚠ ALIAS COLLISION — SEVERAL NAMES, ONE SET OF WEIGHTS
|
# ⚠ ALIAS COLLISION — SEVERAL NAMES, ONE SET OF WEIGHTS
|
||||||
#
|
#
|
||||||
# As of 2026-08-23 these SEVEN aliases all resolve to the same backend
|
# As of 2026-08-23 these SEVEN aliases all resolve to the same backend
|
||||||
# (qwen3.8-27b-uncensored @ 10.250.50.54:8015):
|
# (qwen3.8-27b-uncensored @ 10.251.50.54:8015):
|
||||||
#
|
#
|
||||||
# chat-judge classifier gen image-judge
|
# chat-judge classifier gen image-judge
|
||||||
# qwen-image-bench summarizer summarizer-large
|
# qwen-image-bench summarizer summarizer-large
|
||||||
@@ -59,7 +59,7 @@ model_list:
|
|||||||
# - model_name: granite-4.1-8b
|
# - model_name: granite-4.1-8b
|
||||||
# litellm_params:
|
# litellm_params:
|
||||||
# model: hosted_vllm/granite-4.1-8b
|
# model: hosted_vllm/granite-4.1-8b
|
||||||
# api_base: http://10.250.50.54:8004/v1
|
# api_base: http://10.251.50.54:8004/v1
|
||||||
# api_key: os.environ/VLLM_API_KEY
|
# api_key: os.environ/VLLM_API_KEY
|
||||||
# temperature: 0
|
# temperature: 0
|
||||||
# model_info:
|
# model_info:
|
||||||
@@ -71,7 +71,7 @@ model_list:
|
|||||||
- model_name: summarizer
|
- model_name: summarizer
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: hosted_vllm/qwen3.8-27b-uncensored
|
model: hosted_vllm/qwen3.8-27b-uncensored
|
||||||
api_base: http://10.250.50.54:8015/v1
|
api_base: http://10.251.50.54:8015/v1
|
||||||
api_key: os.environ/VLLM_API_KEY
|
api_key: os.environ/VLLM_API_KEY
|
||||||
temperature: 0
|
temperature: 0
|
||||||
extra_body:
|
extra_body:
|
||||||
@@ -85,7 +85,7 @@ model_list:
|
|||||||
- model_name: classifier
|
- model_name: classifier
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: hosted_vllm/qwen3.8-27b-uncensored
|
model: hosted_vllm/qwen3.8-27b-uncensored
|
||||||
api_base: http://10.250.50.54:8015/v1
|
api_base: http://10.251.50.54:8015/v1
|
||||||
api_key: os.environ/VLLM_API_KEY
|
api_key: os.environ/VLLM_API_KEY
|
||||||
temperature: 0
|
temperature: 0
|
||||||
extra_body:
|
extra_body:
|
||||||
@@ -100,7 +100,7 @@ model_list:
|
|||||||
- model_name: summarizer-large
|
- model_name: summarizer-large
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: hosted_vllm/qwen3.8-27b-uncensored
|
model: hosted_vllm/qwen3.8-27b-uncensored
|
||||||
api_base: http://10.250.50.54:8015/v1
|
api_base: http://10.251.50.54:8015/v1
|
||||||
api_key: os.environ/VLLM_API_KEY
|
api_key: os.environ/VLLM_API_KEY
|
||||||
presence_penalty: 1.5
|
presence_penalty: 1.5
|
||||||
temperature: 0.7
|
temperature: 0.7
|
||||||
@@ -123,7 +123,7 @@ model_list:
|
|||||||
- model_name: qwen-image-bench
|
- model_name: qwen-image-bench
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: hosted_vllm/qwen3.8-27b-uncensored
|
model: hosted_vllm/qwen3.8-27b-uncensored
|
||||||
api_base: http://10.250.50.54:8015/v1
|
api_base: http://10.251.50.54:8015/v1
|
||||||
api_key: os.environ/VLLM_API_KEY
|
api_key: os.environ/VLLM_API_KEY
|
||||||
temperature: 0
|
temperature: 0
|
||||||
top_p: 1.0
|
top_p: 1.0
|
||||||
@@ -137,7 +137,7 @@ model_list:
|
|||||||
- model_name: image-judge
|
- model_name: image-judge
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: hosted_vllm/qwen3.8-27b-uncensored
|
model: hosted_vllm/qwen3.8-27b-uncensored
|
||||||
api_base: http://10.250.50.54:8015/v1
|
api_base: http://10.251.50.54:8015/v1
|
||||||
api_key: os.environ/VLLM_API_KEY
|
api_key: os.environ/VLLM_API_KEY
|
||||||
temperature: 0
|
temperature: 0
|
||||||
top_p: 1.0
|
top_p: 1.0
|
||||||
@@ -182,7 +182,7 @@ model_list:
|
|||||||
- model_name: gen
|
- model_name: gen
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: hosted_vllm/qwen3.8-27b-uncensored
|
model: hosted_vllm/qwen3.8-27b-uncensored
|
||||||
api_base: http://10.250.50.54:8015/v1
|
api_base: http://10.251.50.54:8015/v1
|
||||||
api_key: os.environ/VLLM_API_KEY
|
api_key: os.environ/VLLM_API_KEY
|
||||||
presence_penalty: 1.5
|
presence_penalty: 1.5
|
||||||
temperature: 0.7
|
temperature: 0.7
|
||||||
@@ -200,7 +200,7 @@ model_list:
|
|||||||
# Distinct served-name so a thinking-off `gen` request can't mutate this deployment's
|
# Distinct served-name so a thinking-off `gen` request can't mutate this deployment's
|
||||||
# enable_thinking (shared-config-mutation footgun). Same backend :8015, different model id.
|
# enable_thinking (shared-config-mutation footgun). Same backend :8015, different model id.
|
||||||
model: hosted_vllm/qwen3.8-27b-uncensored-thinking
|
model: hosted_vllm/qwen3.8-27b-uncensored-thinking
|
||||||
api_base: http://10.250.50.54:8015/v1
|
api_base: http://10.251.50.54:8015/v1
|
||||||
api_key: os.environ/VLLM_API_KEY
|
api_key: os.environ/VLLM_API_KEY
|
||||||
# CANONICAL Qwen3.8 THINKING sampling set (Qwen + unsloth "Best Practices"
|
# CANONICAL Qwen3.8 THINKING sampling set (Qwen + unsloth "Best Practices"
|
||||||
# §1, identical in both): temperature=1.0, top_p=0.95, top_k=20,
|
# §1, identical in both): temperature=1.0, top_p=0.95, top_k=20,
|
||||||
@@ -248,7 +248,7 @@ model_list:
|
|||||||
- model_name: char-rp
|
- model_name: char-rp
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: hosted_vllm/char-rp
|
model: hosted_vllm/char-rp
|
||||||
api_base: http://10.250.50.54:8016/v1
|
api_base: http://10.251.50.54:8016/v1
|
||||||
api_key: os.environ/VLLM_API_KEY
|
api_key: os.environ/VLLM_API_KEY
|
||||||
temperature: 1.1
|
temperature: 1.1
|
||||||
top_p: 0.95
|
top_p: 0.95
|
||||||
@@ -274,7 +274,7 @@ model_list:
|
|||||||
- model_name: char-rp-reasoning
|
- model_name: char-rp-reasoning
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: hosted_vllm/char-rp-thinking
|
model: hosted_vllm/char-rp-thinking
|
||||||
api_base: http://10.250.50.54:8016/v1
|
api_base: http://10.251.50.54:8016/v1
|
||||||
api_key: os.environ/VLLM_API_KEY
|
api_key: os.environ/VLLM_API_KEY
|
||||||
temperature: 1.1
|
temperature: 1.1
|
||||||
top_p: 0.95
|
top_p: 0.95
|
||||||
@@ -310,8 +310,8 @@ model_list:
|
|||||||
# character card both sit at ~0%. Fable-Fusion is Heretic-abliterated.
|
# character card both sit at ~0%. Fable-Fusion is Heretic-abliterated.
|
||||||
#
|
#
|
||||||
# ROLLBACK (restores Dark-Scarlett):
|
# ROLLBACK (restores Dark-Scarlett):
|
||||||
# ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe && sudo docker compose down'
|
# ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe && sudo docker compose down'
|
||||||
# ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && sudo docker compose up -d'
|
# ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && sudo docker compose up -d'
|
||||||
# then revert this block to api_base :8018 / model hosted_vllm/char-rp-reasoning
|
# then revert this block to api_base :8018 / model hosted_vllm/char-rp-reasoning
|
||||||
# and restart litellm (~52s).
|
# and restart litellm (~52s).
|
||||||
#
|
#
|
||||||
@@ -337,7 +337,7 @@ model_list:
|
|||||||
- model_name: sec
|
- model_name: sec
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: hosted_vllm/mog-sec-27b
|
model: hosted_vllm/mog-sec-27b
|
||||||
api_base: http://10.250.50.54:8019/v1
|
api_base: http://10.251.50.54:8019/v1
|
||||||
api_key: os.environ/VLLM_API_KEY
|
api_key: os.environ/VLLM_API_KEY
|
||||||
temperature: 0.7
|
temperature: 0.7
|
||||||
top_p: 0.8
|
top_p: 0.8
|
||||||
@@ -356,7 +356,7 @@ model_list:
|
|||||||
- model_name: sec-reasoning
|
- model_name: sec-reasoning
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: hosted_vllm/mog-sec-27b-thinking
|
model: hosted_vllm/mog-sec-27b-thinking
|
||||||
api_base: http://10.250.50.54:8019/v1
|
api_base: http://10.251.50.54:8019/v1
|
||||||
api_key: os.environ/VLLM_API_KEY
|
api_key: os.environ/VLLM_API_KEY
|
||||||
temperature: 1.0
|
temperature: 1.0
|
||||||
top_p: 0.95
|
top_p: 0.95
|
||||||
@@ -393,7 +393,7 @@ model_list:
|
|||||||
- model_name: qwen3-embedding
|
- model_name: qwen3-embedding
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
|
model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
|
||||||
api_base: http://10.250.50.54:8001/v1
|
api_base: http://10.251.50.54:8001/v1
|
||||||
api_key: os.environ/VLLM_API_KEY
|
api_key: os.environ/VLLM_API_KEY
|
||||||
model_info:
|
model_info:
|
||||||
mode: embedding
|
mode: embedding
|
||||||
@@ -423,7 +423,7 @@ model_list:
|
|||||||
- model_name: chat-judge
|
- model_name: chat-judge
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: hosted_vllm/qwen3.8-27b-uncensored
|
model: hosted_vllm/qwen3.8-27b-uncensored
|
||||||
api_base: http://10.250.50.54:8015/v1
|
api_base: http://10.251.50.54:8015/v1
|
||||||
api_key: os.environ/VLLM_API_KEY
|
api_key: os.environ/VLLM_API_KEY
|
||||||
temperature: 0
|
temperature: 0
|
||||||
top_p: 1.0
|
top_p: 1.0
|
||||||
@@ -438,7 +438,7 @@ model_list:
|
|||||||
- model_name: reranker
|
- model_name: reranker
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: hosted_vllm/BAAI/bge-reranker-v2-m3
|
model: hosted_vllm/BAAI/bge-reranker-v2-m3
|
||||||
api_base: http://10.250.50.54:8013/v1
|
api_base: http://10.251.50.54:8013/v1
|
||||||
api_key: os.environ/VLLM_API_KEY
|
api_key: os.environ/VLLM_API_KEY
|
||||||
model_info:
|
model_info:
|
||||||
mode: rerank
|
mode: rerank
|
||||||
@@ -456,7 +456,7 @@ model_list:
|
|||||||
- model_name: coder-fast
|
- model_name: coder-fast
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: hosted_vllm/qwen2.5-coder-1.5b
|
model: hosted_vllm/qwen2.5-coder-1.5b
|
||||||
api_base: http://10.250.50.54:8020/v1
|
api_base: http://10.251.50.54:8020/v1
|
||||||
api_key: os.environ/VLLM_API_KEY
|
api_key: os.environ/VLLM_API_KEY
|
||||||
model_info:
|
model_info:
|
||||||
mode: completion
|
mode: completion
|
||||||
@@ -681,7 +681,7 @@ model_list:
|
|||||||
- model_name: erp-tune-v2
|
- model_name: erp-tune-v2
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: hosted_vllm/erp-tune-v2
|
model: hosted_vllm/erp-tune-v2
|
||||||
api_base: http://10.250.50.54:8098/v1
|
api_base: http://10.251.50.54:8098/v1
|
||||||
api_key: os.environ/VLLM_API_KEY
|
api_key: os.environ/VLLM_API_KEY
|
||||||
|
|
||||||
# gemma4-26b-a4b-it-base -> the UNMODIFIED upstream instruct release, bf16
|
# gemma4-26b-a4b-it-base -> the UNMODIFIED upstream instruct release, bf16
|
||||||
@@ -743,7 +743,7 @@ model_list:
|
|||||||
- model_name: gemma4-26b-a4b-it-base
|
- model_name: gemma4-26b-a4b-it-base
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: hosted_vllm/gemma4-26b-a4b-it-base
|
model: hosted_vllm/gemma4-26b-a4b-it-base
|
||||||
api_base: http://10.250.50.54:8099/v1
|
api_base: http://10.251.50.54:8099/v1
|
||||||
api_key: os.environ/VLLM_API_KEY
|
api_key: os.environ/VLLM_API_KEY
|
||||||
|
|
||||||
# merged-run03 -> the in-house ERP/RP SFT, RUN 3, MERGED bf16 (:8099, vLLM, ana-ml2
|
# merged-run03 -> the in-house ERP/RP SFT, RUN 3, MERGED bf16 (:8099, vLLM, ana-ml2
|
||||||
@@ -848,7 +848,7 @@ model_list:
|
|||||||
# - model_name: merged-run03
|
# - model_name: merged-run03
|
||||||
# litellm_params:
|
# litellm_params:
|
||||||
# model: hosted_vllm/merged-run03
|
# model: hosted_vllm/merged-run03
|
||||||
# api_base: http://10.250.50.54:8099/v1
|
# api_base: http://10.251.50.54:8099/v1
|
||||||
# api_key: os.environ/VLLM_API_KEY
|
# api_key: os.environ/VLLM_API_KEY
|
||||||
|
|
||||||
# trial -> ERP/RP SFT RUN 7, NVFP4A16 (weight-only) quant of the merged LoRA, served on
|
# trial -> ERP/RP SFT RUN 7, NVFP4A16 (weight-only) quant of the merged LoRA, served on
|
||||||
@@ -879,9 +879,10 @@ model_list:
|
|||||||
#
|
#
|
||||||
# THIS GATEWAY IS THE SHARED-KEY SURFACE: `all-agents-local` reaches every model here,
|
# THIS GATEWAY IS THE SHARED-KEY SURFACE: `all-agents-local` reaches every model here,
|
||||||
# in every session and project. Removing this alias does not remove the operator's
|
# in every session and project. Removing this alias does not remove the operator's
|
||||||
# access -- the direct endpoint http://10.250.50.54:8021/v1 is unaffected.
|
# access -- the direct endpoint http://10.251.50.54:8021/v1 is unaffected.
|
||||||
#
|
#
|
||||||
# Same-site: seat and gateway are both at Anaheim (local hop, no mesh crossing).
|
# CROSS-SITE since 2026-09-12: gateway on ana-docker (Anaheim), seat on fv-ml1
|
||||||
|
# (Fountain Valley) -- metro hop over the headscale mesh, ~6ms.
|
||||||
|
|
||||||
# char-rp-fast — the MeroMero A4B MoE, on ana-ml2 GPU1 (:8021). Operator, 2026-09-10:
|
# char-rp-fast — the MeroMero A4B MoE, on ana-ml2 GPU1 (:8021). Operator, 2026-09-10:
|
||||||
# "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias
|
# "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias
|
||||||
@@ -919,7 +920,7 @@ model_list:
|
|||||||
- model_name: char-rp-fast
|
- model_name: char-rp-fast
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: hosted_vllm/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16
|
model: hosted_vllm/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16
|
||||||
api_base: http://10.250.50.54:8021/v1
|
api_base: http://10.251.50.54:8021/v1
|
||||||
api_key: os.environ/VLLM_API_KEY
|
api_key: os.environ/VLLM_API_KEY
|
||||||
extra_body:
|
extra_body:
|
||||||
min_p: 0.05
|
min_p: 0.05
|
||||||
@@ -950,14 +951,14 @@ general_settings:
|
|||||||
maximum_spend_logs_retention_period: "7d"
|
maximum_spend_logs_retention_period: "7d"
|
||||||
maximum_spend_logs_retention_interval: "1d"
|
maximum_spend_logs_retention_interval: "1d"
|
||||||
# scalar-judge → Skywork-Reward-V2 (scalar reward model; vLLM pooling on
|
# scalar-judge → Skywork-Reward-V2 (scalar reward model; vLLM pooling on
|
||||||
# ana-ml2:8003). LiteLLM has no reward/pooling MODE, so this is a passthrough,
|
# fv-ml1:8003). LiteLLM has no reward/pooling MODE, so this is a passthrough,
|
||||||
# not a model_list alias. Gateway-key-gated. Consumers POST the reward body to
|
# not a model_list alias. Gateway-key-gated. Consumers POST the reward body to
|
||||||
# /scalar-judge/<route> (e.g. /pooling or /classify), forwarded to :8003.
|
# /scalar-judge/<route> (e.g. /pooling or /classify), forwarded to :8003.
|
||||||
# SWAP-SENSITIVE: a different reward model shifts the score scale, so consumers
|
# SWAP-SENSITIVE: a different reward model shifts the score scale, so consumers
|
||||||
# must recalibrate thresholds after a backing swap.
|
# must recalibrate thresholds after a backing swap.
|
||||||
pass_through_endpoints:
|
pass_through_endpoints:
|
||||||
- path: "/scalar-judge"
|
- path: "/scalar-judge"
|
||||||
target: "http://10.250.50.54:8003"
|
target: "http://10.251.50.54:8003"
|
||||||
forward_headers: true
|
forward_headers: true
|
||||||
include_subpath: true
|
include_subpath: true
|
||||||
|
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
#
|
#
|
||||||
# Proxies OpenAI-compatible API requests to llama.cpp server instances
|
# Proxies OpenAI-compatible API requests to llama.cpp server instances
|
||||||
# and swaps which model is loaded into VRAM per request. Runs on
|
# and swaps which model is loaded into VRAM per request. Runs on
|
||||||
# ana-ml2 using both GPUs dynamically (no explicit device pinning —
|
# fv-ml1 using both GPUs dynamically (no explicit device pinning —
|
||||||
# llama-swap picks per-model-definition).
|
# llama-swap picks per-model-definition).
|
||||||
#
|
#
|
||||||
# Model definitions live in /opt/docker/conf/llama-swap/config.yaml on
|
# Model definitions live in /opt/docker/conf/llama-swap/config.yaml on
|
||||||
@@ -47,8 +47,8 @@ services:
|
|||||||
- homepage.group=AI Systems
|
- homepage.group=AI Systems
|
||||||
- homepage.name=llama-swap
|
- homepage.name=llama-swap
|
||||||
- homepage.icon=mdi-swap-horizontal
|
- homepage.icon=mdi-swap-horizontal
|
||||||
- homepage.description=GGUF model swapper (llama.cpp; ana-ml2)
|
- homepage.description=GGUF model swapper (llama.cpp; fv-ml1)
|
||||||
- homepage.href=http://10.250.50.54:${LLAMA_SWAP_PORT}
|
- homepage.href=http://10.251.50.54:${LLAMA_SWAP_PORT}
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
tnet:
|
tnet:
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
# meromero-charrp — G4-MeroMero-v2-31B NVFP4A16 char-rp PROSE seat (non-thinking, MULTIMODAL)
|
# meromero-charrp — G4-MeroMero-v2-31B NVFP4A16 char-rp PROSE seat (non-thinking, MULTIMODAL)
|
||||||
# on ana-ml2 GPU0. Replaces the retired Magidonia GGUF seat (char-rp-gguf / llama-charrp).
|
# on fv-ml1 GPU0. Replaces the retired Magidonia GGUF seat (char-rp-gguf / llama-charrp).
|
||||||
#
|
#
|
||||||
# Gemma-4 dense but with SLIDING-WINDOW attention -> KV-efficient, serves the full native 256K
|
# Gemma-4 dense but with SLIDING-WINDOW attention -> KV-efficient, serves the full native 256K
|
||||||
# at ~2x concurrency. Vision enabled: preprocessor_config.json was materialized from
|
# at ~2x concurrency. Vision enabled: preprocessor_config.json was materialized from
|
||||||
@@ -93,8 +93,8 @@ services:
|
|||||||
- homepage.group=AI - Dormant
|
- homepage.group=AI - Dormant
|
||||||
- homepage.name=char-rp (MeroMero-v2 NVFP4) — rollback
|
- homepage.name=char-rp (MeroMero-v2 NVFP4) — rollback
|
||||||
- homepage.icon=mdi-drama-masks
|
- homepage.icon=mdi-drama-masks
|
||||||
- homepage.description=Superseded 2026-08-24 by the Gemma-4 26B-A4B MoE seat; dense 31B NVFP4A16, kept for rollback (ana-ml2 GPU0)
|
- homepage.description=Superseded 2026-08-24 by the Gemma-4 26B-A4B MoE seat; dense 31B NVFP4A16, kept for rollback (fv-ml1 GPU0)
|
||||||
- homepage.href=http://10.250.50.54:${MEROMERO_PORT:-8016}/docs
|
- homepage.href=http://10.251.50.54:${MEROMERO_PORT:-8016}/docs
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
tnet:
|
tnet:
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
# SGLang — alternative LLM serving engine, stood up on ana-ml2 to bench
|
# SGLang — alternative LLM serving engine, stood up on fv-ml1 to bench
|
||||||
# head-to-head against vLLM on the same model + hardware.
|
# head-to-head against vLLM on the same model + hardware.
|
||||||
#
|
#
|
||||||
# Bench-oriented (not yet a permanent service): point it at the SAME checkpoint
|
# Bench-oriented (not yet a permanent service): point it at the SAME checkpoint
|
||||||
@@ -68,8 +68,8 @@ services:
|
|||||||
- homepage.group=AI Systems
|
- homepage.group=AI Systems
|
||||||
- homepage.name=SGLang (bench)
|
- homepage.name=SGLang (bench)
|
||||||
- homepage.icon=mdi-speedometer
|
- homepage.icon=mdi-speedometer
|
||||||
- homepage.description=SGLang serving — vLLM bench comparison (ana-ml2)
|
- homepage.description=SGLang serving — vLLM bench comparison (fv-ml1)
|
||||||
- homepage.href=http://10.250.50.54:${SGLANG_PORT}
|
- homepage.href=http://10.251.50.54:${SGLANG_PORT}
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
tnet:
|
tnet:
|
||||||
|
|||||||
Reference in New Issue
Block a user