vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

fv-ml1: complete the cutover — rename, renumber, DNS, and the LiteLLM repoint

Browse Source 
The box is physically at Fountain Valley, renamed, renumbered onto 10.251/16,
and serving inference again. This lands the repo half of that.

Host: hostname ana-ml2 -> fv-ml1, pinned to 10.251.50.54 by a dnsmasq
reservation so the address the runbook, DNS and LiteLLM all assume is the
address it actually has. Its headscale node is renamed too.

The sweep ran from scripts/fv-ml1-rename-sweep.sh, whose allowlist is the
reason this diff touches current-state files and not the record. Dated
persistent-memory entries, archival-memory and incident notes still say
ana-ml2 in 31 and 62 places respectively, because that is what the box was
when those things happened. Rewriting them would make the history lie.

LiteLLM was the load-bearing piece and needed more than the api_base sed the
runbook describes. Twenty api_base entries repointed, but a grep-and-verify
pass also caught a LIVE pass_through_endpoints target for the scalar-judge
reward route still on the old address -- an api_base-only substitution would
have left it dead. Four prose references describing current state were
repointed as well; one historical note recording where a hand-test was run
is deliberately left pointing at 10.250.50.54.

Two facts in the server tables were wrong and are corrected here. The site is
Fountain Valley, not Anaheim. And the box has FOUR RTX PRO 6000 Blackwell
Max-Q, not two -- verified by nvidia-smi -L and independently by PCI
enumeration of four GB202GL devices. That is 391 GB of VRAM rather than 196,
which changes what fits on it.

DNS: fv-ml1, fv-ml1-bmc and fv-gw added under the fv site via the piggyback
approach, scriberr re-homed, and the ana-ml2 records removed. Applied to all
three resolvers. The BMC record carries a warning that its 802.1q VLAN tag
must stay disabled -- it shipped tagging VLAN 250 into an untagged port,
which made it invisible to every network-side diagnostic and is the reason
it appeared dead through several cable changes.

Verified end to end: summarizer and sec both answer through the Anaheim
gateway across the mesh to FV seats on different ports.
This commit is contained in:
vh
2026-09-12 22:00:50 -07:00
parent fde7834635
commit 91bda3c480
31 changed files with 115 additions and 112 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/beszel/README.md
+8 -8
View File
@@ -2,7 +2,7 @@
Priority 2 completed 2026-09-11: pfi-postgres, esh-vm-db, pbs-ana and pbs-nh3
added with 16 alerts. Live samples verified, including both PBS datastores.
Fleet 17/18 up (known ana-ml2 outage). See `configs/beszel-agent/PRIORITY2.md`.
Fleet 17/18 up (known fv-ml1 outage). See `configs/beszel-agent/PRIORITY2.md`.
Hub: http://10.250.50.70:8090 (ana-docker), version 0.18.7 at the
2026-09-10 wiring. The hub also retains corviduo-dev's existing registration.
@@ -12,12 +12,12 @@ esh-pve and esh-pve-nas); 13 systems total and 20 additional alert rules.
See `configs/beszel-agent/README.md`. Subsequently nh3-nas was installed via
DSM Docker and registered with four alerts (14 registrations total). Recovery
verification at 2026-09-12 01:51Z: all six priority-1 hosts up, Synology live
filesystem samples verified; fleet 13/14 up with known ana-ml2 outage.
filesystem samples verified; fleet 13/14 up with known fv-ml1 outage.
| Host | Compose directory under `/opt/docker/compose/` | Additional filesystems |
|---|---|---|
| ana-docker | beszel | /mnt/backup |
| ana-ml2 | beszel-agent-ana | /tank, /home |
| fv-ml1 | beszel-agent-ana | /tank, /home |
| nh3-docker | beszel-agent-nh3 | none |
| esh-docker-vm (hub name esh-vm-docker) | beszel-agent-esh | /mnt/backup, /mnt/books |
| irv-ml1 | beszel-agent-irv | /worktank, /storetank, /mnt/smithy |
@@ -37,14 +37,14 @@ and `DEPLOY_SUDO=1` for root-owned directories. Example:
```sh
DEPLOY_SUDO=1 DEPLOY_DEST_STACK=beszel-agent-ana \
scripts/deploy-stack.sh infra-ops@10.250.50.54 beszel --compose
scripts/deploy-stack.sh infra-ops@10.251.50.54 beszel --compose
```
Agents use host-specific overrides selected by live `.env`:
```dotenv
COMPOSE_PROFILES=agent
COMPOSE_FILE=compose.yaml:hosts/ana-ml2.yaml
COMPOSE_FILE=compose.yaml:hosts/fv-ml1.yaml
BESZEL_EXTRA_FS=/extra-filesystems/tank,/extra-filesystems/home
```
@@ -71,7 +71,7 @@ hub public key. Never copy live tokens into version control.
## GPU telemetry
ana-ml2 and irv-ml1 use `henrygd/beszel-agent-nvidia:0.18.7` with NVIDIA
fv-ml1 and irv-ml1 use `henrygd/beszel-agent-nvidia:0.18.7` with NVIDIA
`utility` access to all GPUs. Both hosts already have NVIDIA Container Toolkit.
This collects per-card utilization, VRAM, temperature, and power draw without
changing the serving containers or GPU power limits. Verified hub samples
@@ -111,7 +111,7 @@ Thirty rules cover the seven hosts above under the existing operator user:
| CPU | >95% | 15 minutes |
| Memory | >90% | 10 minutes |
| Offline | down | 2 minutes |
| Temperature (ana-ml2 and irv-ml1) | >85 C | 5 minutes |
| Temperature (fv-ml1 and irv-ml1) | >85 C | 5 minutes |
CPU thresholds are sustained-load warnings; expected long-running compute may
need tuning. GPU utilization alone is not an alarm because busy GPUs are normal.
@@ -125,7 +125,7 @@ The bridge at `services/beszel-althing/` forwards through `postbox` to the
was replaced with this verified route. Miranda is a later cutover, not enabled.
See that service's README for operation and recipient changes.
Acceptance on 2026-09-10: ana-ml2 Disk was temporarily lowered to 1%/1 minute;
Acceptance on 2026-09-10: fv-ml1 Disk was temporarily lowered to 1%/1 minute;
the real alert reached althing at 15:29:45Z, thread
`01M25Z0WFDJM92GPTJQF769HJ7`. The threshold was then restored to 85%/5 minutes.
Verification uses `postbox thread`, which does not consume the inbox.
stacks/dockge/.env.example
+2 -2
View File
@@ -10,13 +10,13 @@ DOCKGE_VERSION=latest
DOCKGE_PORT=5001
# Human-readable host name for the homepage card, e.g.
# ana-docker, nh3-docker, esh-docker-vm, vm-esh-nas, ana-ml2
# ana-docker, nh3-docker, esh-docker-vm, vm-esh-nas, fv-ml1
DOCKGE_HOST_LABEL=
# IP (or hostname) the browser uses to reach this Dockge. Used in
# homepage.href — must be reachable from wherever you view homepage.
# ana-docker → 10.250.50.70
# ana-ml2 → 10.250.50.54
# fv-ml1 → 10.251.50.54
# nh3-docker → 10.100.50.40
# esh-docker-vm → 10.0.50.45
# vm-esh-nas → 10.0.50.154
stacks/erp-seat/compose.yaml
+4 -4
View File
@@ -1,4 +1,4 @@
# erp-seat — the RP seat on ana-ml2 GPU1. Serves the **MeroMero A4B MoE** NVFP4A16 quant
# erp-seat — the RP seat on fv-ml1 GPU1. Serves the **MeroMero A4B MoE** NVFP4A16 quant
# (G4-MeroMero-26B-A4B-it-uncensored-heretic) behind the gateway alias `char-rp-fast`.
#
# ⚠ THE STACK NAME IS HISTORICAL. It served Pfish-6 (the run-6 ERP-tune LoRA merge) until
@@ -109,7 +109,7 @@ services:
# SPEED (2026-09-09). ⚠ NVFP4A16 is WEIGHT-ONLY: activations stay 16-bit, so
# there is no FP4 GEMM to issue and the FlashInfer/CUTLASS NVFP4 MoE kernels
# do not apply — vLLM's "your GPU does not have native support for FP4"
# warning is about the SCHEME, not the card. ana-ml2 is sm_120 and DOES have
# warning is about the SCHEME, not the card. fv-ml1 is sm_120 and DOES have
# native FP4; a W4A4 build would use it, at a fidelity cost the operator
# already settled against for this seat. Marlin is the correct kernel here.
# PROVEN, not assumed: forcing moe_backend=flashinfer_cutedsl fails at engine
@@ -151,8 +151,8 @@ services:
- homepage.group=AI - Inference
- homepage.name=char-rp-fast (MeroMero 26B-A4B, NVFP4A16 MoE)
- homepage.icon=mdi-fire
- homepage.description=MeroMero A4B abliterated RP seat, NVFP4A16 weight-only, vision intact (ana-ml2 GPU1)
- homepage.href=http://10.250.50.54:${ERP_PORT:-8021}/docs
- homepage.description=MeroMero A4B abliterated RP seat, NVFP4A16 weight-only, vision intact (fv-ml1 GPU1)
- homepage.href=http://10.251.50.54:${ERP_PORT:-8021}/docs
networks:
tnet:
stacks/gemma4-charrp/README.md
+3 -3
View File
@@ -1,11 +1,11 @@
# gemma4-charrp — the char-rp seat (ana-ml2 GPU0)
# gemma4-charrp — the char-rp seat (fv-ml1 GPU0)
`google/gemma-4-26B-A4B-it`, NVFP4, serving both halves of the char-rp pair on
`:8016`. Replaced the dense `G4-MeroMero-v2-31B-NVFP4A16` seat on 2026-08-24.
```
char-rp non-thinking -> http://10.250.50.54:8016/v1
char-rp-reasoning thinking -> http://10.250.50.54:8016/v1
char-rp non-thinking -> http://10.251.50.54:8016/v1
char-rp-reasoning thinking -> http://10.251.50.54:8016/v1
```
Two LiteLLM aliases, **one backend**. They are not two seats — this trips people
stacks/gemma4-charrp/compose.yaml
+4 -4
View File
@@ -1,4 +1,4 @@
# gemma4-charrp — google/gemma-4-26B-A4B-it NVFP4 char-rp seat on ana-ml2 GPU0.
# gemma4-charrp — google/gemma-4-26B-A4B-it NVFP4 char-rp seat on fv-ml1 GPU0.
#
# Straight-across replacement for `meromero-charrp` (operator, 2026-08-24): same
# port, same served-model-names, so every gateway route and consumer is
@@ -46,7 +46,7 @@ services:
- --quantization
- compressed-tensors
# UNCHANGED FROM THE OUTGOING SEAT, ON PURPOSE. Both names are live
# LiteLLM routes at http://10.250.50.54:8016/v1 — `char-rp` ->
# LiteLLM routes at http://10.251.50.54:8016/v1 — `char-rp` ->
# hosted_vllm/char-rp and `char-rp-reasoning` -> hosted_vllm/char-rp-thinking.
# They are two aliases onto ONE seat, not two seats. Renaming either
# breaks the gateway silently.
@@ -128,8 +128,8 @@ services:
- homepage.group=AI - Inference
- homepage.name=char-rp (Gemma-4 26B-A4B NVFP4, MoE)
- homepage.icon=mdi-drama-masks
- homepage.description=gemma-4-26B-A4B-it NVFP4 MoE prose+CoT seat, 256K (ana-ml2 GPU0)
- homepage.href=http://10.250.50.54:${GEMMA4_PORT:-8016}/docs
- homepage.description=gemma-4-26B-A4B-it NVFP4 MoE prose+CoT seat, 256K (fv-ml1 GPU0)
- homepage.href=http://10.251.50.54:${GEMMA4_PORT:-8016}/docs
networks:
tnet:
stacks/heretic2-charrp-reasoning/README.md
+5 -5
View File
@@ -1,12 +1,12 @@
# heretic2-charrp-reasoning — NVFP4 + native MTP reasoning seat (ana-ml2)
# heretic2-charrp-reasoning — NVFP4 + native MTP reasoning seat (fv-ml1)
The `char-rp-reasoning` seat: NEO-CODE Heretic2 27B quantized to modelopt NVFP4
with a grafted BF16 MTP head, served by vLLM on **ana-ml2 GPU0**, port **8018**.
with a grafted BF16 MTP head, served by vLLM on **fv-ml1 GPU0**, port **8018**.
Replaces the retired GGUF seat (`llama-charrp-reasoning`) at roughly **77 tok/s
(~1.3x)** via `qwen3_5_mtp` speculative decode.
- **Served model name:** `char-rp-reasoning` — the gateway alias consumers use
- **Endpoint:** `http://10.250.50.54:8018` (`/docs` for the card link)
- **Endpoint:** `http://10.251.50.54:8018` (`/docs` for the card link)
## ⚠️ It does not boot without the MTP workaround
@@ -39,8 +39,8 @@ OOM the neighbours, not just this container.** Tunables are in `.env`; see
## Deploy
```bash
scripts/deploy-stack.sh ana-ml2 heretic2-charrp-reasoning
ssh infra-ops@10.250.50.54 \
scripts/deploy-stack.sh fv-ml1 heretic2-charrp-reasoning
ssh infra-ops@10.251.50.54 \
'cd /opt/docker/compose/heretic2-charrp-reasoning && sudo docker compose up -d'
```
stacks/homepage/README.md
+2 -2
View File
@@ -18,7 +18,7 @@ Two paths, and mixing them is the classic failure:
`homepage.group=` / `homepage.name=` / `homepage.icon=` / `homepage.description=`
/ `homepage.href=` labels and appears automatically. `conf/docker.yaml`
wires **five** engines over plaintext `:2375`: esh-docker-vm, ana-docker,
nh3-docker, ana-ml2, irv-ml1.
nh3-docker, fv-ml1, irv-ml1.
2. **Manual entries in `conf/services.yaml`** — for anything that is not a
labelled container on one of those five hosts: hardware, BMCs, hypervisors,
printers, and user-level systemd services (The Booth, Voice Design Studio).
@@ -63,7 +63,7 @@ scripts/elway infra-ops@10.250.50.70 --playbook playbooks/homepage-regroup-ana-d
scripts/elway infra-ops@10.0.50.45 --playbook playbooks/homepage-regroup-esh-docker-vm.yaml
scripts/elway infra-ops@10.100.50.40 --playbook playbooks/homepage-regroup-nh3-docker.yaml
scripts/elway infra-ops@10.100.79.3 --playbook playbooks/homepage-regroup-irv-ml1.yaml
scripts/elway infra-ops@10.250.50.54 --playbook playbooks/homepage-regroup-ana-ml2.yaml
scripts/elway infra-ops@10.251.50.54 --playbook playbooks/homepage-regroup-fv-ml1.yaml
```
⚠ **The GPU-backed model seats are deliberately still named `AI - Inference`,
stacks/homepage/conf/docker.yaml
+6 -6
View File
@@ -17,13 +17,13 @@ nh3-pfi-docker:
host: 10.100.50.40
port: 2375
# ana-ml2-docker — reached via a socat proxy on ana-docker (stacks/ana-ml2-proxy).
# ana-ml2's own :2375 is up but not reachable cross-site from ESH (a return-route
# gap on ana-ml2 itself; no SSH access from the ops seats to fix its routing).
# fv-ml1-docker — reached via a socat proxy on ana-docker (stacks/fv-ml1-proxy).
# fv-ml1's own :2375 is up but not reachable cross-site from ESH (a return-route
# gap on fv-ml1 itself; no SSH access from the ops seats to fix its routing).
# ana-docker is on the same Anaheim subnet and IS reachable from ESH, so it
# relays 10.250.50.70:2354 -> 10.250.50.54:2375. Point back at 10.250.50.54:2375
# and `docker compose down` the proxy once ana-ml2 has a return route to ESH.
ana-ml2-docker:
# relays 10.250.50.70:2354 -> 10.251.50.54:2375. Point back at 10.251.50.54:2375
# and `docker compose down` the proxy once fv-ml1 has a return route to ESH.
fv-ml1-docker:
host: 10.250.50.70
port: 2354
stacks/homepage/conf/services.yaml
+2 -2
View File
@@ -118,14 +118,14 @@
siteMonitor: http://10.250.50.70:5001
description: Docker VM (ana-docker, Dockge at :5001)
- PFI-ANA-ML2:
ping: 10.250.50.54
ping: 10.251.50.54
icon: mdi-brain
description: GPU host (bare-metal)
- PFI-ANA-ML2 BMC:
href: https://10.250.250.50
icon: mdi-brain
siteMonitor: https://10.250.250.50
description: BMC (ana-ml2)
description: BMC (fv-ml1)
- SFsrv-ANA:
href: https://10.250.250.115:8006
icon: si-proxmox
stacks/litellm/conf/config.yaml
+29 -28
View File
@@ -1,5 +1,5 @@
# LiteLLM gateway config — fronts the vLLM services on ana-ml2
# (10.250.50.54) and logs every request + response so they're
# (10.251.50.54) and logs every request + response so they're
# inspectable in the Logs UI at http://10.250.50.70:4000/ui.
#
# Deploys to /opt/docker/conf/litellm/config.yaml (mounted read-only
@@ -22,7 +22,7 @@ model_list:
# ⚠ ALIAS COLLISION — SEVERAL NAMES, ONE SET OF WEIGHTS
#
# As of 2026-08-23 these SEVEN aliases all resolve to the same backend
# (qwen3.8-27b-uncensored @ 10.250.50.54:8015):
# (qwen3.8-27b-uncensored @ 10.251.50.54:8015):
#
# chat-judge classifier gen image-judge
# qwen-image-bench summarizer summarizer-large
@@ -59,7 +59,7 @@ model_list:
# - model_name: granite-4.1-8b
# litellm_params:
# model: hosted_vllm/granite-4.1-8b
# api_base: http://10.250.50.54:8004/v1
# api_base: http://10.251.50.54:8004/v1
# api_key: os.environ/VLLM_API_KEY
# temperature: 0
# model_info:
@@ -71,7 +71,7 @@ model_list:
- model_name: summarizer
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.250.50.54:8015/v1
api_base: http://10.251.50.54:8015/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0
extra_body:
@@ -85,7 +85,7 @@ model_list:
- model_name: classifier
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.250.50.54:8015/v1
api_base: http://10.251.50.54:8015/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0
extra_body:
@@ -100,7 +100,7 @@ model_list:
- model_name: summarizer-large
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.250.50.54:8015/v1
api_base: http://10.251.50.54:8015/v1
api_key: os.environ/VLLM_API_KEY
presence_penalty: 1.5
temperature: 0.7
@@ -123,7 +123,7 @@ model_list:
- model_name: qwen-image-bench
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.250.50.54:8015/v1
api_base: http://10.251.50.54:8015/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0
top_p: 1.0
@@ -137,7 +137,7 @@ model_list:
- model_name: image-judge
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.250.50.54:8015/v1
api_base: http://10.251.50.54:8015/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0
top_p: 1.0
@@ -182,7 +182,7 @@ model_list:
- model_name: gen
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.250.50.54:8015/v1
api_base: http://10.251.50.54:8015/v1
api_key: os.environ/VLLM_API_KEY
presence_penalty: 1.5
temperature: 0.7
@@ -200,7 +200,7 @@ model_list:
# Distinct served-name so a thinking-off `gen` request can't mutate this deployment's
# enable_thinking (shared-config-mutation footgun). Same backend :8015, different model id.
model: hosted_vllm/qwen3.8-27b-uncensored-thinking
api_base: http://10.250.50.54:8015/v1
api_base: http://10.251.50.54:8015/v1
api_key: os.environ/VLLM_API_KEY
# CANONICAL Qwen3.8 THINKING sampling set (Qwen + unsloth "Best Practices"
# §1, identical in both): temperature=1.0, top_p=0.95, top_k=20,
@@ -248,7 +248,7 @@ model_list:
- model_name: char-rp
litellm_params:
model: hosted_vllm/char-rp
api_base: http://10.250.50.54:8016/v1
api_base: http://10.251.50.54:8016/v1
api_key: os.environ/VLLM_API_KEY
temperature: 1.1
top_p: 0.95
@@ -274,7 +274,7 @@ model_list:
- model_name: char-rp-reasoning
litellm_params:
model: hosted_vllm/char-rp-thinking
api_base: http://10.250.50.54:8016/v1
api_base: http://10.251.50.54:8016/v1
api_key: os.environ/VLLM_API_KEY
temperature: 1.1
top_p: 0.95
@@ -310,8 +310,8 @@ model_list:
# character card both sit at ~0%. Fable-Fusion is Heretic-abliterated.
#
# ROLLBACK (restores Dark-Scarlett):
# ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe && sudo docker compose down'
# ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && sudo docker compose up -d'
# ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe && sudo docker compose down'
# ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && sudo docker compose up -d'
# then revert this block to api_base :8018 / model hosted_vllm/char-rp-reasoning
# and restart litellm (~52s).
#
@@ -337,7 +337,7 @@ model_list:
- model_name: sec
litellm_params:
model: hosted_vllm/mog-sec-27b
api_base: http://10.250.50.54:8019/v1
api_base: http://10.251.50.54:8019/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0.7
top_p: 0.8
@@ -356,7 +356,7 @@ model_list:
- model_name: sec-reasoning
litellm_params:
model: hosted_vllm/mog-sec-27b-thinking
api_base: http://10.250.50.54:8019/v1
api_base: http://10.251.50.54:8019/v1
api_key: os.environ/VLLM_API_KEY
temperature: 1.0
top_p: 0.95
@@ -393,7 +393,7 @@ model_list:
- model_name: qwen3-embedding
litellm_params:
model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
api_base: http://10.250.50.54:8001/v1
api_base: http://10.251.50.54:8001/v1
api_key: os.environ/VLLM_API_KEY
model_info:
mode: embedding
@@ -423,7 +423,7 @@ model_list:
- model_name: chat-judge
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.250.50.54:8015/v1
api_base: http://10.251.50.54:8015/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0
top_p: 1.0
@@ -438,7 +438,7 @@ model_list:
- model_name: reranker
litellm_params:
model: hosted_vllm/BAAI/bge-reranker-v2-m3
api_base: http://10.250.50.54:8013/v1
api_base: http://10.251.50.54:8013/v1
api_key: os.environ/VLLM_API_KEY
model_info:
mode: rerank
@@ -456,7 +456,7 @@ model_list:
- model_name: coder-fast
litellm_params:
model: hosted_vllm/qwen2.5-coder-1.5b
api_base: http://10.250.50.54:8020/v1
api_base: http://10.251.50.54:8020/v1
api_key: os.environ/VLLM_API_KEY
model_info:
mode: completion
@@ -681,7 +681,7 @@ model_list:
- model_name: erp-tune-v2
litellm_params:
model: hosted_vllm/erp-tune-v2
api_base: http://10.250.50.54:8098/v1
api_base: http://10.251.50.54:8098/v1
api_key: os.environ/VLLM_API_KEY
# gemma4-26b-a4b-it-base -> the UNMODIFIED upstream instruct release, bf16
@@ -743,7 +743,7 @@ model_list:
- model_name: gemma4-26b-a4b-it-base
litellm_params:
model: hosted_vllm/gemma4-26b-a4b-it-base
api_base: http://10.250.50.54:8099/v1
api_base: http://10.251.50.54:8099/v1
api_key: os.environ/VLLM_API_KEY
# merged-run03 -> the in-house ERP/RP SFT, RUN 3, MERGED bf16 (:8099, vLLM, ana-ml2
@@ -848,7 +848,7 @@ model_list:
# - model_name: merged-run03
# litellm_params:
# model: hosted_vllm/merged-run03
# api_base: http://10.250.50.54:8099/v1
# api_base: http://10.251.50.54:8099/v1
# api_key: os.environ/VLLM_API_KEY
# trial -> ERP/RP SFT RUN 7, NVFP4A16 (weight-only) quant of the merged LoRA, served on
@@ -879,9 +879,10 @@ model_list:
#
# THIS GATEWAY IS THE SHARED-KEY SURFACE: `all-agents-local` reaches every model here,
# in every session and project. Removing this alias does not remove the operator's
# access -- the direct endpoint http://10.250.50.54:8021/v1 is unaffected.
# access -- the direct endpoint http://10.251.50.54:8021/v1 is unaffected.
#
# Same-site: seat and gateway are both at Anaheim (local hop, no mesh crossing).
# CROSS-SITE since 2026-09-12: gateway on ana-docker (Anaheim), seat on fv-ml1
# (Fountain Valley) -- metro hop over the headscale mesh, ~6ms.
# char-rp-fast — the MeroMero A4B MoE, on ana-ml2 GPU1 (:8021). Operator, 2026-09-10:
# "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias
@@ -919,7 +920,7 @@ model_list:
- model_name: char-rp-fast
litellm_params:
model: hosted_vllm/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16
api_base: http://10.250.50.54:8021/v1
api_base: http://10.251.50.54:8021/v1
api_key: os.environ/VLLM_API_KEY
extra_body:
min_p: 0.05
@@ -950,14 +951,14 @@ general_settings:
maximum_spend_logs_retention_period: "7d"
maximum_spend_logs_retention_interval: "1d"
# scalar-judge → Skywork-Reward-V2 (scalar reward model; vLLM pooling on
# ana-ml2:8003). LiteLLM has no reward/pooling MODE, so this is a passthrough,
# fv-ml1:8003). LiteLLM has no reward/pooling MODE, so this is a passthrough,
# not a model_list alias. Gateway-key-gated. Consumers POST the reward body to
# /scalar-judge/<route> (e.g. /pooling or /classify), forwarded to :8003.
# SWAP-SENSITIVE: a different reward model shifts the score scale, so consumers
# must recalibrate thresholds after a backing swap.
pass_through_endpoints:
- path: "/scalar-judge"
target: "http://10.250.50.54:8003"
target: "http://10.251.50.54:8003"
forward_headers: true
include_subpath: true
stacks/llama-swap/compose.yaml
+3 -3
View File
@@ -2,7 +2,7 @@
#
# Proxies OpenAI-compatible API requests to llama.cpp server instances
# and swaps which model is loaded into VRAM per request. Runs on
# ana-ml2 using both GPUs dynamically (no explicit device pinning —
# fv-ml1 using both GPUs dynamically (no explicit device pinning —
# llama-swap picks per-model-definition).
#
# Model definitions live in /opt/docker/conf/llama-swap/config.yaml on
@@ -47,8 +47,8 @@ services:
- homepage.group=AI Systems
- homepage.name=llama-swap
- homepage.icon=mdi-swap-horizontal
- homepage.description=GGUF model swapper (llama.cpp; ana-ml2)
- homepage.href=http://10.250.50.54:${LLAMA_SWAP_PORT}
- homepage.description=GGUF model swapper (llama.cpp; fv-ml1)
- homepage.href=http://10.251.50.54:${LLAMA_SWAP_PORT}
networks:
tnet:
stacks/meromero-charrp/compose.yaml
+3 -3
View File
@@ -1,5 +1,5 @@
# meromero-charrp — G4-MeroMero-v2-31B NVFP4A16 char-rp PROSE seat (non-thinking, MULTIMODAL)
# on ana-ml2 GPU0. Replaces the retired Magidonia GGUF seat (char-rp-gguf / llama-charrp).
# on fv-ml1 GPU0. Replaces the retired Magidonia GGUF seat (char-rp-gguf / llama-charrp).
#
# Gemma-4 dense but with SLIDING-WINDOW attention -> KV-efficient, serves the full native 256K
# at ~2x concurrency. Vision enabled: preprocessor_config.json was materialized from
@@ -93,8 +93,8 @@ services:
- homepage.group=AI - Dormant
- homepage.name=char-rp (MeroMero-v2 NVFP4) — rollback
- homepage.icon=mdi-drama-masks
- homepage.description=Superseded 2026-08-24 by the Gemma-4 26B-A4B MoE seat; dense 31B NVFP4A16, kept for rollback (ana-ml2 GPU0)
- homepage.href=http://10.250.50.54:${MEROMERO_PORT:-8016}/docs
- homepage.description=Superseded 2026-08-24 by the Gemma-4 26B-A4B MoE seat; dense 31B NVFP4A16, kept for rollback (fv-ml1 GPU0)
- homepage.href=http://10.251.50.54:${MEROMERO_PORT:-8016}/docs
networks:
tnet:
stacks/sglang/compose.yaml
+3 -3
View File
@@ -1,4 +1,4 @@
# SGLang — alternative LLM serving engine, stood up on ana-ml2 to bench
# SGLang — alternative LLM serving engine, stood up on fv-ml1 to bench
# head-to-head against vLLM on the same model + hardware.
#
# Bench-oriented (not yet a permanent service): point it at the SAME checkpoint
@@ -68,8 +68,8 @@ services:
- homepage.group=AI Systems
- homepage.name=SGLang (bench)
- homepage.icon=mdi-speedometer
- homepage.description=SGLang serving — vLLM bench comparison (ana-ml2)
- homepage.href=http://10.250.50.54:${SGLANG_PORT}
- homepage.description=SGLang serving — vLLM bench comparison (fv-ml1)
- homepage.href=http://10.251.50.54:${SGLANG_PORT}
networks:
tnet: