vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

fv-ml1: finish the renumber the cutover missed -- 16 dead dashboard links

Browse Source 
Every fv-ml1 link on the Homepage dashboard was broken. Measured against the
live dashboard API before the fix: 16 entries pointing at the dead 10.250.50.54
and zero at the live 10.251.50.54, covering gen, M.O.G.-SEC, Scriberr, Embed,
Rerank, Reward, Coder, Dockge and six dormant seats.

The miss was structural, not careless. fv-ml1-rename-sweep.sh works from an
allowlist assembled from files that mention the HOST, and a homepage.href label
mentions only an IP -- so every stack whose sole stale reference was a label
fell outside it. The allowlist now covers those 24 files, and records how to
derive the list next time (grep the old address, subtract history) rather than
enumerating from memory.

History is still untouched, and the exclusions are now written down with the
reason each one keeps the old address: recorded benchmark results, whose
base_url is part of a measurement's provenance; the one LiteLLM comment
preserving a retired hand-test endpoint; and the cutover runbooks, where the old
address is the subject matter.

Two bugs found while applying it, both fixed here:

  - deploy-stack.sh rejected any stack name containing a dot, so qwen3.5-122b,
    qwopus3.5-122b and mistral-medium-3.5 could not be deployed by the script at
    all. The check exists to stop path traversal, which means rejecting ".." and
    "/" -- not every dot. Traversal is now rejected explicitly and tested.
  - stacks/scriberr/.env.example allowed CORS only from the dead IP and from
    scriberr.ana.internal, which no longer resolves; the box is at the fv site
    and DNS already carries scriberr.fv.internal. The live .env had both stale
    origins, i.e. an allowlist with nothing reachable in it.

Host side, applied separately: canonical pushed for the 16 stacks whose only
difference from the host was this renumber, and an in-place address-only fix for
the nine whose host copy has genuinely drifted or has no canonical copy, so that
drift survives for a deliberate reconciliation instead of being clobbered. Every
compose.yaml on fv-ml1 now reads 10.251.50.54. The labels themselves only take
effect at container creation, so the running containers still need recreating.
This commit is contained in:
vh
2026-09-12 23:05:29 -07:00
parent f964a47262
commit 3132a16ca0
26 changed files with 163 additions and 106 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/char-rp-gguf/README.md
+7 -7
View File
@@ -1,6 +1,6 @@
# char-rp-gguf
Dedicated GGUF **character-RP seat** on ana-ml2 GPU 0, served via llama.cpp
Dedicated GGUF **character-RP seat** on fv-ml1 GPU 0, served via llama.cpp
(`llama-server`) — **never Ollama**. Replaces the broken `ms32-24b-angel` NVFP4
serve (self-quant produced garbage even at greedy).
@@ -47,8 +47,8 @@ Canonical copies live here; authoritative copies live on the host under
`/opt/docker/compose/char-rp-gguf/`.
```bash
# On ana-ml2 (models already pre-pulled to /tank/aimodels/llm/rp/):
scripts/deploy-stack.sh ana-ml2 char-rp-gguf # from this workspace
# On fv-ml1 (models already pre-pulled to /tank/aimodels/llm/rp/):
scripts/deploy-stack.sh fv-ml1 char-rp-gguf # from this workspace
# or, on the host:
cd /opt/docker/compose/char-rp-gguf && cp .env.example .env && docker compose up -d
```
@@ -64,8 +64,8 @@ docker run -d --entrypoint python3 -v /tank:/tank vllm/vllm-openai:latest -c \
## Gateway wiring (ana-docker LiteLLM `:4000`)
`char-rp` → `http://10.250.50.54:8016/v1`, `char-rp-reasoning` →
`http://10.250.50.54:8018/v1`. Sampler defaults (per the model cards):
`char-rp` → `http://10.251.50.54:8016/v1`, `char-rp-reasoning` →
`http://10.251.50.54:8018/v1`. Sampler defaults (per the model cards):
- **char-rp (Magidonia / Mistral):** `temperature 1.0, top_p 0.95, min_p 0.03`;
no repetition penalty (Magistral is stable without one). Callers may override.
@@ -89,7 +89,7 @@ broken — do not re-serve it).
## Verify
```bash
curl -s http://10.250.50.54:8016/v1/models # char-rp up
curl -s http://10.250.50.54:8018/v1/models # char-rp-reasoning up
curl -s http://10.251.50.54:8016/v1/models # char-rp up
curl -s http://10.251.50.54:8018/v1/models # char-rp-reasoning up
nvidia-smi --query-gpu=index,memory.free --format=csv # GPU 0 margin
```
stacks/char-rp-gguf/compose.yaml
+5 -5
View File
@@ -1,4 +1,4 @@
# char-rp-gguf — dedicated GGUF character-RP seat on ana-ml2 GPU 0, REPLACING the
# char-rp-gguf — dedicated GGUF character-RP seat on fv-ml1 GPU 0, REPLACING the
# broken ms32-24b-angel NVFP4 serve (garbage output — bad self-quant W4A4).
#
# Two co-located llama.cpp (llama-server) instances on GPU 0, served alongside the
@@ -91,8 +91,8 @@ services:
- homepage.group=AI - Inference
- homepage.name=char-rp (Magidonia-24B GGUF)
- homepage.icon=mdi-drama-masks
- homepage.description=Dark-romantasy RP prose seat, non-thinking (llama.cpp, ana-ml2 GPU 0)
- homepage.href=http://10.250.50.54:${CHARRP_PORT:-8016}
- homepage.description=Dark-romantasy RP prose seat, non-thinking (llama.cpp, fv-ml1 GPU 0)
- homepage.href=http://10.251.50.54:${CHARRP_PORT:-8016}
# ── REASONING seat — NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) managed thinking. gateway char-rp-reasoning. ──
llama-charrp-reasoning:
@@ -172,8 +172,8 @@ services:
- homepage.group=AI - Dormant
- homepage.name=char-rp-reasoning (QwQ-32B RpR-v4 GGUF)
- homepage.icon=mdi-brain
- homepage.description=Dark-romantasy RP reasoning seat, managed CoT (llama.cpp, ana-ml2 GPU 0)
- homepage.href=http://10.250.50.54:${CHARRP_REASONING_PORT:-8018}
- homepage.description=Dark-romantasy RP reasoning seat, managed CoT (llama.cpp, fv-ml1 GPU 0)
- homepage.href=http://10.251.50.54:${CHARRP_REASONING_PORT:-8018}
networks:
tnet:
stacks/darkscarlett-charrp-reasoning/compose.yaml
+3 -3
View File
@@ -1,5 +1,5 @@
# darkscarlett-charrp-reasoning — Dark-Scarlett-v1.0-27B NVFP4A16 char-rp-reasoning seat
# (THINKING default) on ana-ml2 GPU1. Replaces the retired heretic2-charrp-reasoning seat.
# (THINKING default) on fv-ml1 GPU1. Replaces the retired heretic2-charrp-reasoning seat.
#
# Qwen3.6 hybrid GatedDeltaNet linear-attention -> KV-cheap, serves the full native 256K.
# Quantized with llm-compressor NVFP4A16 loaded through the Qwen3_5ForConditionalGeneration
@@ -67,8 +67,8 @@ services:
- homepage.group=AI - Inference
- homepage.name=char-rp-reasoning (Dark-Scarlett NVFP4)
- homepage.icon=mdi-brain
- homepage.description=Dark-Scarlett-v1.0-27B NVFP4A16 thinking seat, 256K (ana-ml2 GPU1)
- homepage.href=http://10.250.50.54:${DS_PORT:-8018}/docs
- homepage.description=Dark-Scarlett-v1.0-27B NVFP4A16 thinking seat, 256K (fv-ml1 GPU1)
- homepage.href=http://10.251.50.54:${DS_PORT:-8018}/docs
networks:
tnet:
stacks/dockge/README.md
+4 -4
View File
@@ -40,10 +40,10 @@ DOCKGE_HOST_LABEL=ana-docker
DOCKGE_HOST_IP=10.250.50.70
EOF'
# ana-ml2
ssh -t ana-ml2 'sudo tee /opt/docker/compose/dockge/.env > /dev/null <<EOF
DOCKGE_HOST_LABEL=ana-ml2
DOCKGE_HOST_IP=10.250.50.54
# fv-ml1
ssh -t fv-ml1 'sudo tee /opt/docker/compose/dockge/.env > /dev/null <<EOF
DOCKGE_HOST_LABEL=fv-ml1
DOCKGE_HOST_IP=10.251.50.54
EOF'
# nh3-docker
stacks/dozzle/.env.example
+4 -4
View File
@@ -13,11 +13,11 @@ DOZZLE_VERSION=latest
# On ana-docker (hub):
# COMPOSE_PROFILES=hub
# DOZZLE_HOSTNAME=ana-docker
# DOZZLE_REMOTE_AGENT=10.250.50.54:7007
# DOZZLE_REMOTE_AGENT=10.251.50.54:7007
#
# On ana-ml2 (agent):
# On fv-ml1 (agent):
# COMPOSE_PROFILES=agent
# DOZZLE_HOSTNAME=ana-ml2
# DOZZLE_HOSTNAME=fv-ml1
# ------------------------------------------------------------------------
COMPOSE_PROFILES=hub
@@ -32,7 +32,7 @@ DOZZLE_PORT=8088
# Comma-separated list of remote agents the hub should connect to.
# Leave blank if this host only views its own containers.
DOZZLE_REMOTE_AGENT=10.250.50.54:7007,10.100.50.40:7007
DOZZLE_REMOTE_AGENT=10.251.50.54:7007,10.100.50.40:7007
# Auth — `none` is fine behind the LAN / a reverse proxy with auth.
# Switch to `simple` and set USERNAME/PASSWORD to gate the UI itself.
stacks/dozzle/README.md
+9 -9
View File
@@ -4,11 +4,11 @@ Container log viewer. One UI on **ana-docker** aggregates logs from every Docker
**Deploys to:**
- **ana-docker** (hub) — UI at `http://10.250.50.70:8088`
- **ana-ml2** (agent) — listens on `10.250.50.54:7007`
- **fv-ml1** (agent) — listens on `10.251.50.54:7007`
- **nh3-docker** (agent, cross-site) — listens on `10.100.50.40:7007`
- **corviduo-dev** (agent) — listens on `10.250.50.152:7007`. Compose at `/home/vh/docker/compose/dozzle-agent/` (not `/opt/docker/compose/` — see `servers/corviduo-dev/README.md` for why)
One compose.yaml lives on each host. The per-host `.env` sets `COMPOSE_PROFILES=hub` or `COMPOSE_PROFILES=agent` so `docker compose up -d` brings up the right service. On the hub, add every agent to `DOZZLE_REMOTE_AGENT` as a comma-separated list (e.g. `10.250.50.54:7007,10.100.50.40:7007`).
One compose.yaml lives on each host. The per-host `.env` sets `COMPOSE_PROFILES=hub` or `COMPOSE_PROFILES=agent` so `docker compose up -d` brings up the right service. On the hub, add every agent to `DOZZLE_REMOTE_AGENT` as a comma-separated list (e.g. `10.251.50.54:7007,10.100.50.40:7007`).
## Auth / TLS note
@@ -27,7 +27,7 @@ cp .env.example .env
# Ensure:
# COMPOSE_PROFILES=hub
# DOZZLE_HOSTNAME=ana-docker
# DOZZLE_REMOTE_AGENT=10.250.50.54:7007
# DOZZLE_REMOTE_AGENT=10.251.50.54:7007
# DOZZLE_PORT=8088
docker compose config
@@ -35,10 +35,10 @@ docker compose up -d
docker compose logs -f
```
## Deploy — agent (ana-ml2)
## Deploy — agent (fv-ml1)
```bash
ssh ana-ml2
ssh fv-ml1
sudo mkdir -p /opt/docker/compose/dozzle
sudo chown $USER /opt/docker/compose/dozzle
cd /opt/docker/compose/dozzle
@@ -47,7 +47,7 @@ cd /opt/docker/compose/dozzle
cp .env.example .env
# Edit to:
# COMPOSE_PROFILES=agent
# DOZZLE_HOSTNAME=ana-ml2
# DOZZLE_HOSTNAME=fv-ml1
# DOZZLE_AGENT_PORT=7007
docker compose config
@@ -62,14 +62,14 @@ docker compose logs -f
curl -s http://10.250.50.70:8088/healthz
# Agent reachable from the hub's perspective
ssh ana-docker 'nc -zv 10.250.50.54 7007'
ssh ana-docker 'nc -zv 10.251.50.54 7007'
# Open http://10.250.50.70:8088 — you should see two tabs:
# "ana-docker" (local containers) and "ana-ml2" (via agent).
# "ana-docker" (local containers) and "fv-ml1" (via agent).
```
## Troubleshooting
- **Hub shows only local containers:** agent is unreachable. Check firewall rules on ana-ml2 (port 7007 must be open from 10.250.50.70) and that the agent is actually listening (`ss -tlnp | grep 7007`).
- **Hub shows only local containers:** agent is unreachable. Check firewall rules on fv-ml1 (port 7007 must be open from 10.250.50.70) and that the agent is actually listening (`ss -tlnp | grep 7007`).
- **Agent keeps restarting:** verify the docker.sock bind mount is read-only and the socket exists.
- **Certificate mismatch after image upgrade:** delete the `dozzle_data` and `dozzle_agent_data` volumes on both hosts and redeploy to regenerate.
stacks/fablefusion-charrp-probe/README.md
+6 -6
View File
@@ -1,4 +1,4 @@
# fablefusion-charrp-probe — THROWAWAY evaluation seat (ana-ml2 GPU1)
# fablefusion-charrp-probe — THROWAWAY evaluation seat (fv-ml1 GPU1)
Evaluation-only seat for
[`kkuspa/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-NVFP4A16`](https://huggingface.co/kkuspa/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-NVFP4A16),
@@ -40,11 +40,11 @@ at `util 0.44`; this seat needs the same slot. The A/B is **sequential**, and th
```bash
# 1. baseline DS live first (no disruption)
# 2. swap
ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && docker compose down'
ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe && docker compose up -d'
ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && docker compose down'
ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe && docker compose up -d'
# 3. probe :8019, then
ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe && docker compose down'
ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && docker compose up -d'
ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe && docker compose down'
ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && docker compose up -d'
```
## MTP depth: 3, not the card's 5
@@ -59,7 +59,7 @@ until benchmarked at real RP sampling settings.
## Deploy
```bash
scripts/deploy-stack.sh ana-ml2 fablefusion-charrp-probe
scripts/deploy-stack.sh fv-ml1 fablefusion-charrp-probe
# on host: cp .env.example .env (defaults are fine)
```
stacks/gen-seat/README.md
+3 -3
View File
@@ -1,4 +1,4 @@
# gen-seat — the fleet `gen` seat (ana-ml2 GPU0, :8015)
# gen-seat — the fleet `gen` seat (fv-ml1 GPU0, :8015)
Serves **`qwen3.8-27b-uncensored`** (JonathanColetti/Qwen3.8-27B-Uncensored,
Heretic-abliterated Qwen3.8-27B, vision-intact, 262K context) plus the
@@ -27,8 +27,8 @@ budget interaction with `meromero-charrp`, and the rollback command — lives in
## Deploy
```bash
scripts/deploy-stack.sh ana-ml2 gen-seat # diffs vs live, prompts y/N
scripts/deploy-stack.sh fv-ml1 gen-seat # diffs vs live, prompts y/N
# on host:
ssh infra-ops@10.250.50.54
ssh infra-ops@10.251.50.54
cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen
```
stacks/gen-seat/compose.yaml
+3 -3
View File
@@ -1,4 +1,4 @@
# gen-seat — the fleet's general `gen` seat on ana-ml2 GPU 0 (:8015).
# gen-seat — the fleet's general `gen` seat on fv-ml1 GPU 0 (:8015).
# Serves JonathanColetti/Qwen3.8-27B-Uncensored (Heretic abliteration KL 0.12, vision-intact
# Qwen3_5ForConditionalGeneration, grafted MTP head), quantized in-house to NVFP4 W4A16
# (compressed-tensors) with the bf16 MTP grafted back. ⚠ the grafted MTP requires
@@ -159,8 +159,8 @@ services:
- homepage.group=AI - Inference
- homepage.name=Qwen3.8-27B Uncensored (NVFP4, vision, MTP) — gen
- homepage.icon=mdi-creation
- homepage.description=Uncensored Qwen3.8-27B multimodal NVFP4+MTP, the `gen` seat (ana-ml2 GPU 0)
- homepage.href=http://10.250.50.54:${GEN_PORT:-8015}/docs
- homepage.description=Uncensored Qwen3.8-27B multimodal NVFP4+MTP, the `gen` seat (fv-ml1 GPU 0)
- homepage.href=http://10.251.50.54:${GEN_PORT:-8015}/docs
networks:
tnet:
stacks/heretic2-charrp-reasoning/compose.yaml
+3 -3
View File
@@ -1,5 +1,5 @@
# heretic2-charrp-reasoning — modelopt NVFP4 + native MTP fast char-rp-reasoning seat on
# ana-ml2 GPU0, replacing the GGUF NEO-CODE reasoning seat (llama-charrp-reasoning, now retired).
# fv-ml1 GPU0, replacing the GGUF NEO-CODE reasoning seat (llama-charrp-reasoning, now retired).
# Same Heretic2/NEO-CODE model; ~77 tok/s (~1.3x over GGUF) via qwen3_5_mtp spec-decode.
#
# ⚠️ REQUIRES the MTP workaround: vLLM 0.24.0 doesn't propagate modelopt exclude_modules to the
@@ -76,8 +76,8 @@ services:
- homepage.group=AI - Inference
- homepage.name=char-rp-reasoning (Heretic2 NVFP4+MTP)
- homepage.icon=mdi-rocket-launch
- homepage.description=NEO-CODE Heretic2 NVFP4 + native MTP, ~77 tok/s (ana-ml2 GPU0)
- homepage.href=http://10.250.50.54:${REASONING_PORT:-8018}/docs
- homepage.description=NEO-CODE Heretic2 NVFP4 + native MTP, ~77 tok/s (fv-ml1 GPU0)
- homepage.href=http://10.251.50.54:${REASONING_PORT:-8018}/docs
networks:
tnet:
stacks/litellm/README.md
+3 -3
View File
@@ -1,6 +1,6 @@
# litellm
OpenAI-compatible **gateway** in front of the vLLM services on ana-ml2,
OpenAI-compatible **gateway** in front of the vLLM services on fv-ml1,
standing in the request path so every request + response is **logged and
inspectable in a browser**. This is the thing vLLM does not give us:
Dozzle shows vLLM's stdout (connection/request metadata) but not the full
@@ -8,7 +8,7 @@ prompt/completion bodies. LiteLLM captures both, per call, with a Logs UI.
**Server:** ana-docker (`10.250.50.70`)
**Port:** `4000` (proxy API + admin/Logs UI at `/ui`) — configurable in `.env`
**Backs:** the `vllm` stack on ana-ml2 (`10.250.50.54`)
**Backs:** the `vllm` stack on fv-ml1 (`10.251.50.54`)
## Why it exists
@@ -130,7 +130,7 @@ Then open `http://10.250.50.70:4000/ui` (log in with the master key) →
## Notes
- Both boxes are Anaheim (`10.250.0.0/16`) so the ana-docker → ana-ml2 hop
- Both boxes are Anaheim (`10.250.0.0/16`) so the ana-docker → fv-ml1 hop
is LAN-local; negligible added latency.
- `VLLM_API_KEY` is blank by default because the `vllm` stack ships
`API_KEY=` empty. Set it here only if you set it there.
stacks/mistral-small-4-heretic/compose.yaml
+2 -2
View File
@@ -92,8 +92,8 @@ services:
- homepage.group=AI Systems
- homepage.name=Mistral Small 4 (heretic NVFP4)
- homepage.icon=mdi-creation
- homepage.description=Abliterated Mistral-Small-4 (heretic NVFP4) drop-in via vLLM (ana-ml2 GPU 0)
- homepage.href=http://10.250.50.54:${MISTRAL_PORT}/docs
- homepage.description=Abliterated Mistral-Small-4 (heretic NVFP4) drop-in via vLLM (fv-ml1 GPU 0)
- homepage.href=http://10.251.50.54:${MISTRAL_PORT}/docs
networks:
tnet:
stacks/mistral-small-4/compose.yaml
+3 -3
View File
@@ -1,4 +1,4 @@
# mistral-small-4 — Mistral-Small-4-119B-2603 (official NVFP4) on ana-ml2 GPU 0.
# mistral-small-4 — Mistral-Small-4-119B-2603 (official NVFP4) on fv-ml1 GPU 0.
#
# Mistral Small 4 is a 119B-total / 6.5B-active MoE (128 experts, 4 active),
# 256K context, multimodal, Apache-2.0 (released 2026-03). This serves the
@@ -107,8 +107,8 @@ services:
- homepage.group=AI Systems
- homepage.name=Mistral Small 4 (NVFP4)
- homepage.icon=mdi-creation
- homepage.description=Mistral-Small-4-119B-2603 MoE (NVFP4) via vLLM (ana-ml2 GPU 0)
- homepage.href=http://10.250.50.54:${MISTRAL_PORT}/docs
- homepage.description=Mistral-Small-4-119B-2603 MoE (NVFP4) via vLLM (fv-ml1 GPU 0)
- homepage.href=http://10.251.50.54:${MISTRAL_PORT}/docs
networks:
tnet:
stacks/mog-sec/compose.yaml
+3 -3
View File
@@ -1,4 +1,4 @@
# mog-sec — the pen-test seat on ana-ml2 GPU 0 (:8019).
# mog-sec — the pen-test seat on fv-ml1 GPU 0 (:8019).
#
# MOVED GPU 1 -> GPU 0 on 2026-08-28 (operator-directed). GPU 1 carries the five
# resident fleet seats (gen 46 GB + embed + coder + rerank + reward = ~69.9 GB of
@@ -215,8 +215,8 @@ services:
- homepage.group=AI - Inference
- homepage.name=M.O.G.-SEC 27B (pen-test)
- homepage.icon=mdi-shield-lock
- homepage.description=Uncensored security model, Qwen3.8-27B NVFP4+MTP, 262K — the `mog-sec` seat (ana-ml2 GPU 0)
- homepage.href=http://10.250.50.54:${MOG_PORT:-8019}/docs
- homepage.description=Uncensored security model, Qwen3.8-27B NVFP4+MTP, 262K — the `mog-sec` seat (fv-ml1 GPU 0)
- homepage.href=http://10.251.50.54:${MOG_PORT:-8019}/docs
networks:
tnet:
stacks/qwen-image-bench/compose.yaml
+3 -3
View File
@@ -1,4 +1,4 @@
# qwen-image-bench — flukethoughts/Qwen-Image-Bench-NVFP4 on ana-ml2 GPU 1,
# qwen-image-bench — flukethoughts/Qwen-Image-Bench-NVFP4 on fv-ml1 GPU 1,
# REPLACING qwen3.6-35b-a3b (operator 2026-06-19). Qwen's text-to-image quality
# JUDGE model (vision-language, NVFP4 weights / vision tower bf16). NOT generative —
# it scores T2I outputs on 5 dims (overall quality, prompt match, aesthetic, LoRA
@@ -75,8 +75,8 @@ services:
- homepage.group=AI - Eval & Retrieval
- homepage.name=Qwen-Image-Bench (T2I judge, NVFP4)
- homepage.icon=mdi-image-check
- homepage.description=Qwen text-to-image quality judge (NVFP4, vision-intact) on ana-ml2 GPU 1
- homepage.href=http://10.250.50.54:${QIB_PORT:-8014}/docs
- homepage.description=Qwen text-to-image quality judge (NVFP4, vision-intact) on fv-ml1 GPU 1
- homepage.href=http://10.251.50.54:${QIB_PORT:-8014}/docs
networks:
tnet:
stacks/qwen3.5-122b/compose.yaml
+3 -3
View File
@@ -1,4 +1,4 @@
# qwen3.5-122b — bjk110/Qwen3.5-122B-A10B-abliterated-NVFP4 on ana-ml2 GPU 0,
# qwen3.5-122b — bjk110/Qwen3.5-122B-A10B-abliterated-NVFP4 on fv-ml1 GPU 0,
# REPLACING mistral-small-4 (operator 2026-06-19: down the heretic, serve this as
# the new general/`gen` model). Abliterated Qwen3.5 MoE (256 experts, 10B active),
# NVFP4 (compressed-tensors), HF format.
@@ -76,8 +76,8 @@ services:
- homepage.group=AI Systems
- homepage.name=Qwen3.5-122B-A10B (abliterated NVFP4)
- homepage.icon=mdi-creation
- homepage.description=Abliterated Qwen3.5 122B-A10B NVFP4, the new `gen` model (ana-ml2 GPU 0)
- homepage.href=http://10.250.50.54:${QWEN35_PORT:-8013}/docs
- homepage.description=Abliterated Qwen3.5 122B-A10B NVFP4, the new `gen` model (fv-ml1 GPU 0)
- homepage.href=http://10.251.50.54:${QWEN35_PORT:-8013}/docs
networks:
tnet:
stacks/qwen36-vl/compose.yaml
+3 -3
View File
@@ -1,4 +1,4 @@
# qwen36-vl — Qwen3.6-35B-A3B vision-language MoE (official NVFP4) on ana-ml2.
# qwen36-vl — Qwen3.6-35B-A3B vision-language MoE (official NVFP4) on fv-ml1.
#
# Replaces the qwen35-vl stack (Qwen3.5-9B) 2026-06-14. Co-located on GPU 1 with
# the granite summarizer + embed/rerank/reward trio. Serves on :8007.
@@ -119,8 +119,8 @@ services:
- homepage.group=AI Systems
- homepage.name=Qwen3.6-35B-A3B VL (NVFP4)
- homepage.icon=mdi-image-search
- homepage.description=Qwen3.6-35B-A3B vision-language MoE (NVFP4) via vLLM (ana-ml2)
- homepage.href=http://10.250.50.54:${QWEN_PORT}/docs
- homepage.description=Qwen3.6-35B-A3B vision-language MoE (NVFP4) via vLLM (fv-ml1)
- homepage.href=http://10.251.50.54:${QWEN_PORT}/docs
networks:
tnet:
stacks/qwopus3.5-122b/compose.yaml
+3 -3
View File
@@ -1,5 +1,5 @@
# qwopus3.5-122b — OpenYourMind/Qwopus3.5-122B-A10B-Kimi-K2.6-destilled-abliterated-NVFP4
# on ana-ml2 GPU 0, REPLACING the bjk110 text-only qwen3.5-122b as the `gen` model
# on fv-ml1 GPU 0, REPLACING the bjk110 text-only qwen3.5-122b as the `gen` model
# (operator 2026-06-19: "already ablated, already quanted, vision tower intact").
#
# Qwen3.5-122B-A10B MoE, Kimi-K2.6-distilled + abliterated, NVFP4 — and crucially
@@ -99,8 +99,8 @@ services:
- homepage.group=AI - Dormant
- homepage.name=Qwopus3.5-122B-A10B (abliterated NVFP4, vision)
- homepage.icon=mdi-creation
- homepage.description=Kimi-distilled abliterated Qwen3.5-122B-A10B NVFP4, vision-intact, the `gen` model (ana-ml2 GPU 0)
- homepage.href=http://10.250.50.54:${QWOPUS_PORT:-8013}/docs
- homepage.description=Kimi-distilled abliterated Qwen3.5-122B-A10B NVFP4, vision-intact, the `gen` model (fv-ml1 GPU 0)
- homepage.href=http://10.251.50.54:${QWOPUS_PORT:-8013}/docs
networks:
tnet:
stacks/scriberr/.env.example
+2 -2
View File
@@ -1,5 +1,5 @@
# Scriberr — copy to .env on the host at /opt/docker/compose/scriberr/.env
# Real .env is gitignored and lives only on ana-ml2.
# Real .env is gitignored and lives only on fv-ml1.
# ── Image ────────────────────────────────────────────────────────────────
# Built locally from Dockerfile.cuda.12.9 — see the compose header for why
@@ -11,7 +11,7 @@ SCRIBERR_PORT=8080
SCRIBERR_BIND=0.0.0.0
# CORS. Must list every origin the UI is actually reached from, or the
# browser blocks the API calls. Comma-separated, no spaces, no trailing /.
SCRIBERR_ALLOWED_ORIGINS=http://10.250.50.54:8080,http://scriberr.ana.internal:8080
SCRIBERR_ALLOWED_ORIGINS=http://10.251.50.54:8080,http://scriberr.fv.internal:8080
# ── GPU ──────────────────────────────────────────────────────────────────
# GPU0 is fully committed to the `gen` seat; GPU1 is the one with headroom.
stacks/scriberr/README.md
+7 -7
View File
@@ -1,17 +1,17 @@
# scriberr — self-hosted transcription + diarization (ana-ml2, GPU1)
# scriberr — self-hosted transcription + diarization (fv-ml1, GPU1)
Web UI for transcribing audio/video locally. WhisperX (Whisper + pyannote
speaker diarization) with NVIDIA Parakeet/Canary also selectable; SQLite for
state; optional summarisation and transcript chat against any OpenAI-compatible
endpoint.
- **Host:** `ana-ml2` (10.250.50.54) — GPU1
- **URL:** http://10.250.50.54:8080
- **Host:** `fv-ml1` (10.251.50.54) — GPU1
- **URL:** http://10.251.50.54:8080
- **Upstream:** https://github.com/rishikanthc/Scriberr
## The image is built locally, and that is not incidental
ana-ml2's RTX PRO 6000 Blackwell cards are **sm_120**. Upstream's published
fv-ml1's RTX PRO 6000 Blackwell cards are **sm_120**. Upstream's published
images do not cover that:
| image | built for | usable here |
@@ -28,7 +28,7 @@ image** — it will fail on these cards or quietly fall back to CPU.
### Rebuilding
```bash
ssh ana-ml2
ssh fv-ml1
cd /tank/scriberr/src/Scriberr
git pull
docker build -f Dockerfile.cuda.12.9 -t scriberr:local-blackwell .
@@ -41,7 +41,7 @@ Source checkout lives on `/tank`, not the root pool — see storage below.
```bash
# from this workstation
scripts/deploy-stack.sh ana-ml2 scriberr
scripts/deploy-stack.sh fv-ml1 scriberr
```
Then on the host, the usual:
@@ -54,7 +54,7 @@ docker compose up -d scriberr # target the service, not the whole stack
## Storage — deliberately on /tank
`/var/lib/docker` on ana-ml2 sits on `zroot` at ~87% used. Whisper, pyannote
`/var/lib/docker` on fv-ml1 sits on `zroot` at ~87% used. Whisper, pyannote
and NeMo weights are multi-GB and land in the `whisperx-env` volume, so both
mounts are bind-mounted onto `/tank` (4+ TB) instead of named volumes:
stacks/scriberr/compose.yaml
+5 -5
View File
@@ -7,7 +7,7 @@
# point it at the LiteLLM gateway rather than a paid API (see README).
#
# ── IMAGE: BUILT LOCALLY, ON PURPOSE ──────────────────────────────────────
# ana-ml2's RTX PRO 6000 Blackwell cards are **sm_120**. Upstream publishes
# fv-ml1's RTX PRO 6000 Blackwell cards are **sm_120**. Upstream publishes
# `scriberr-cuda` (built for sm_61…sm_89 — no sm_120 kernels) and documents a
# `scriberr-cuda-blackwell` image that **has never actually been published**
# (GHCR returns no tags for it, checked 2026-08-23). The sm_120 path upstream
@@ -15,7 +15,7 @@
# Do NOT "simplify" this to the published `scriberr-cuda` image — it will
# fail on these cards or silently fall back to CPU.
# Rebuild: see README "Rebuilding" — checkout lives at
# /tank/scriberr/src/Scriberr on ana-ml2.
# /tank/scriberr/src/Scriberr on fv-ml1.
#
# ── GPU PINNING ───────────────────────────────────────────────────────────
# Pinned to **GPU1** via explicit device_ids, per the house convention and
@@ -34,7 +34,7 @@ services:
ports:
- "${SCRIBERR_BIND:-0.0.0.0}:${SCRIBERR_PORT}:8080"
volumes:
# Bind mounts rather than named volumes: /var/lib/docker on ana-ml2
# Bind mounts rather than named volumes: /var/lib/docker on fv-ml1
# lives on zroot with limited headroom, while /tank has terabytes.
# Model weights (Whisper, pyannote, NeMo) land in whisperx-env and are
# multi-GB — they must not go anywhere near the root pool.
@@ -102,8 +102,8 @@ services:
- homepage.group=AI - Studios
- homepage.name=Scriberr
- homepage.icon=mdi-microphone-message
- homepage.description=Audio/video transcription + diarization (ana-ml2, GPU1)
- homepage.href=http://10.250.50.54:${SCRIBERR_PORT}
- homepage.description=Audio/video transcription + diarization (fv-ml1, GPU1)
- homepage.href=http://10.251.50.54:${SCRIBERR_PORT}
networks:
tnet:
stacks/selene/compose.yaml
+3 -3
View File
@@ -1,4 +1,4 @@
# selene — AtlaAI Selene 1 Mini (Llama 3.1 8B) judge/eval model on ana-ml2 GPU 1.
# selene — AtlaAI Selene 1 Mini (Llama 3.1 8B) judge/eval model on fv-ml1 GPU 1.
#
# Restores the judge that went offline when llama-swap was downed (it was the
# Q6_K GGUF `selene-1-mini-8b` in the llama-swap zoo). Re-served on vLLM at the
@@ -77,8 +77,8 @@ services:
- homepage.group=AI - Eval & Retrieval
- homepage.name=Selene 1 Mini 8B (judge, FP8)
- homepage.icon=mdi-gavel
- homepage.description=AtlaAI Selene 1 Mini Llama-3.1-8B judge (FP8) via vLLM (ana-ml2 GPU1)
- homepage.href=http://10.250.50.54:${SELENE_PORT}/docs
- homepage.description=AtlaAI Selene 1 Mini Llama-3.1-8B judge (FP8) via vLLM (fv-ml1 GPU1)
- homepage.href=http://10.251.50.54:${SELENE_PORT}/docs
networks:
tnet:
stacks/sglang/README.md
+5 -5
View File
@@ -1,4 +1,4 @@
# sglang — vLLM-vs-SGLang bench on ana-ml2
# sglang — vLLM-vs-SGLang bench on fv-ml1
Stood up to benchmark **SGLang against vLLM** on the same model + hardware, to
see whether SGLang's throughput/latency wins justify it as a serving option
@@ -26,18 +26,18 @@ avoid here.
## Run
```bash
# 1. On ana-ml2, after the eval frees a GPU: cp .env.example .env, set
# 1. On fv-ml1, after the eval frees a GPU: cp .env.example .env, set
# SGLANG_MODEL / SGLANG_QUANT to match the vLLM config under test, and
# SGLANG_GPU_ID to an EXCLUSIVE card.
scripts/deploy-stack.sh ana-ml2 sglang
scripts/deploy-stack.sh fv-ml1 sglang
# (or docker compose up -d on the host)
# 2. Bench SGLang:
python3 stacks/sglang/bench.py --url http://10.250.50.54:30000/v1 \
python3 stacks/sglang/bench.py --url http://10.251.50.54:30000/v1 \
--model granite-4.1-8b-nvfp4 --concurrency 1 10 50 100 200 --in-tokens 2048 --out-tokens 256
# 3. Stop SGLang, bring up vLLM on the SAME GPU + model, bench identically:
python3 stacks/sglang/bench.py --url http://10.250.50.54:8006/v1 \
python3 stacks/sglang/bench.py --url http://10.251.50.54:8006/v1 \
--model granite-4.1-8b-nvfp4 --concurrency 1 10 50 100 200 --in-tokens 2048 --out-tokens 256
# 4. Repeat the sweep at --in-tokens 30000 (the prefill-heavy agent-memory
stacks/vllm/compose.yaml
+13 -13
View File
@@ -1,7 +1,7 @@
# vLLM — Qwen3 Embedding + Reranker + Skywork Reward-V2 classifier.
#
# Originally created to replace the unmaintained Infinity stack (embed +
# rerank); generalized 2026-05-13 to host any vLLM-served model on ana-ml2,
# rerank); generalized 2026-05-13 to host any vLLM-served model on fv-ml1,
# starting with the Skywork-Reward-V2-Llama-3.1-8B reward classifier
# (AWQ-quantized locally, served from /tank/aimodels/llm/).
#
@@ -19,13 +19,13 @@
# All tunables live in .env — edit that, not this file.
#
# Pre-download models to avoid first-run delay:
# scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \
# scripts/elway fv-ml1 --playbook playbooks/pull-hf-repo.yaml \
# --var hf_repo=Qwen/Qwen3-Embedding-0.6B
# scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \
# scripts/elway fv-ml1 --playbook playbooks/pull-hf-repo.yaml \
# --var hf_repo=Qwen/Qwen3-Reranker-0.6B
#
# Skywork-Reward-V2-Llama-3.1-8B-AWQ is a locally-quantized model — lives at
# /tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ on ana-ml2 and is
# /tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ on fv-ml1 and is
# bind-mounted into the reward service at /local-models. Not from HF Hub.
services:
@@ -80,8 +80,8 @@ services:
- homepage.group=AI - Eval & Retrieval
- homepage.name=vLLM Embed (Qwen3)
- homepage.icon=mdi-vector-arrange-below
- homepage.description=Qwen3 Embedding via vLLM (ana-ml2)
- homepage.href=http://10.250.50.54:${EMBED_PORT}/docs
- homepage.description=Qwen3 Embedding via vLLM (fv-ml1)
- homepage.href=http://10.251.50.54:${EMBED_PORT}/docs
# THE fleet reranker. Backs the LiteLLM `reranker` alias, which is what every
# consumer should name — never the model, never a bake-off arm name.
@@ -155,8 +155,8 @@ services:
- homepage.group=AI - Eval & Retrieval
- homepage.name=vLLM Rerank (bge-v2-m3)
- homepage.icon=mdi-sort-variant
- homepage.description=BAAI bge-reranker-v2-m3 — the fleet reranker, backs the `reranker` alias (ana-ml2)
- homepage.href=http://10.250.50.54:${RERANK_PORT}/docs
- homepage.description=BAAI bge-reranker-v2-m3 — the fleet reranker, backs the `reranker` alias (fv-ml1)
- homepage.href=http://10.251.50.54:${RERANK_PORT}/docs
vllm-reward:
image: vllm/vllm-openai:${VLLM_VERSION}
@@ -213,8 +213,8 @@ services:
- homepage.group=AI - Eval & Retrieval
- homepage.name=vLLM Reward (Skywork)
- homepage.icon=mdi-scale-balance
- homepage.description=Skywork-Reward-V2 8B classifier via vLLM (ana-ml2)
- homepage.href=http://10.250.50.54:${REWARD_PORT}/docs
- homepage.description=Skywork-Reward-V2 8B classifier via vLLM (fv-ml1)
- homepage.href=http://10.251.50.54:${REWARD_PORT}/docs
# vllm-granite (ibm-granite/granite-4.1-8b-fp8, :8004) — RETIRED 2026-08-12,
# service block removed 2026-08-20. It was the fleet summarizer until the
@@ -296,8 +296,8 @@ services:
- homepage.group=AI - Inference
- homepage.name=vLLM Qwen2.5-Coder 1.5B (FIM)
- homepage.icon=mdi-code-braces
- homepage.description=Qwen2.5-Coder-1.5B FIM code-completion (ana-ml2, Zed edit-predictions)
- homepage.href=http://10.250.50.54:${CODER_PORT}/docs
- homepage.description=Qwen2.5-Coder-1.5B FIM code-completion (fv-ml1, Zed edit-predictions)
- homepage.href=http://10.251.50.54:${CODER_PORT}/docs
# vllm-lfm25 (LiquidAI/LFM2.5-2.6B, :8021) — RETIRED PERMANENTLY 2026-08-20 by
# operator directive. It was an EVAL-ONLY bake-off seat against granite-4.1-8b
@@ -305,7 +305,7 @@ services:
# pending. Its comparator is gone (granite retired from the roster 2026-08-15),
# it was deliberately never wired into any default/fallback routing chain, and
# LiteLLM spend logs showed 0 calls in the 4-day window ending 2026-08-21.
# Freed 8,772 MiB on ana-ml2 GPU1. The `lfm2.5-2.6b` gateway alias was removed
# Freed 8,772 MiB on fv-ml1 GPU1. The `lfm2.5-2.6b` gateway alias was removed
# in the same pass so the name 404s cleanly rather than erroring against a dead
# backend. Weights remain in the shared HF cache; nothing was deleted from disk.