fv-ml1: complete the cutover — rename, renumber, DNS, and the LiteLLM repoint
The box is physically at Fountain Valley, renamed, renumbered onto 10.251/16, and serving inference again. This lands the repo half of that. Host: hostname ana-ml2 -> fv-ml1, pinned to 10.251.50.54 by a dnsmasq reservation so the address the runbook, DNS and LiteLLM all assume is the address it actually has. Its headscale node is renamed too. The sweep ran from scripts/fv-ml1-rename-sweep.sh, whose allowlist is the reason this diff touches current-state files and not the record. Dated persistent-memory entries, archival-memory and incident notes still say ana-ml2 in 31 and 62 places respectively, because that is what the box was when those things happened. Rewriting them would make the history lie. LiteLLM was the load-bearing piece and needed more than the api_base sed the runbook describes. Twenty api_base entries repointed, but a grep-and-verify pass also caught a LIVE pass_through_endpoints target for the scalar-judge reward route still on the old address -- an api_base-only substitution would have left it dead. Four prose references describing current state were repointed as well; one historical note recording where a hand-test was run is deliberately left pointing at 10.250.50.54. Two facts in the server tables were wrong and are corrected here. The site is Fountain Valley, not Anaheim. And the box has FOUR RTX PRO 6000 Blackwell Max-Q, not two -- verified by nvidia-smi -L and independently by PCI enumeration of four GB202GL devices. That is 391 GB of VRAM rather than 196, which changes what fits on it. DNS: fv-ml1, fv-ml1-bmc and fv-gw added under the fv site via the piggyback approach, scriberr re-homed, and the ana-ml2 records removed. Applied to all three resolvers. The BMC record carries a warning that its 802.1q VLAN tag must stay disabled -- it shipped tagging VLAN 250 into an untagged port, which made it invisible to every network-side diagnostic and is the reason it appeared dead through several cable changes. Verified end to end: summarizer and sec both answer through the Anaheim gateway across the mesh to FV seats on different ports.
This commit is contained in:
@@ -131,7 +131,7 @@ group memberships) is in
|
||||
[`stacks/llama-swap/conf/config.yaml`](../../stacks/llama-swap/conf/config.yaml).
|
||||
Don't mirror the table here — it drifted last time and immediately
|
||||
went stale. Live HTTP listing:
|
||||
`curl http://10.250.50.54:9292/v1/models`.
|
||||
`curl http://10.251.50.54:9292/v1/models`.
|
||||
|
||||
**Model Groups**:
|
||||
|
||||
|
||||
@@ -73,5 +73,5 @@ Reverse this for a graceful host shutdown (done 2026-09-05 for the PERC swap).
|
||||
|
||||
## Not on this hypervisor
|
||||
|
||||
`ana-ml2` (10.250.50.54) is bare-metal Supermicro; `sfsrv-ana` (10.250.250.115) is the
|
||||
`fv-ml1` (10.251.50.54) is bare-metal Supermicro; `sfsrv-ana` (10.250.250.115) is the
|
||||
SureFire client's own Dell R630; `pfi-gx10` is at NH3.
|
||||
|
||||
@@ -18,9 +18,9 @@ implementation.
|
||||
|
||||
**CLEARED (green):**
|
||||
- Execute Brokkr's reranker-selection recommendations unattended.
|
||||
- Bring **down the prod reranker** at `ana-ml2:8002` (qwen3-reranker-0.6B) —
|
||||
- Bring **down the prod reranker** at `fv-ml1:8002` (qwen3-reranker-0.6B) —
|
||||
**temporarily OR permanently**.
|
||||
- Down **ONE** of the RP (roleplay) seats on ana-ml2 **temporarily** to free
|
||||
- Down **ONE** of the RP (roleplay) seats on fv-ml1 **temporarily** to free
|
||||
GPU/VRAM for testing.
|
||||
- Temporarily clear space for the smoke/bench.
|
||||
- Pull models, stand up side-port vLLM benches, run the harness — whatever the
|
||||
@@ -30,7 +30,7 @@ implementation.
|
||||
- **NO permanent deletion of anything** (no `rm`/`docker volume rm`/model-weight
|
||||
deletion/data destruction). Downing ≠ deleting.
|
||||
- **NO taking anything else offline** beyond (a) the prod reranker and (b) ONE
|
||||
ana-ml2 RP seat. (Not granite/embed/reward/coder/gen/a second RP seat/muninn/etc.)
|
||||
fv-ml1 RP seat. (Not granite/embed/reward/coder/gen/a second RP seat/muninn/etc.)
|
||||
- **NO rebooting machines.**
|
||||
|
||||
**Process:** accumulate assumptions here; operator reverses at the end.
|
||||
@@ -54,7 +54,7 @@ implementation.
|
||||
|
||||
## Current board at handoff
|
||||
|
||||
- **Prod reranker:** `ana-ml2:8002` = `vllm-rerank` (Qwen/Qwen3-Reranker-0.6B),
|
||||
- **Prod reranker:** `fv-ml1:8002` = `vllm-rerank` (Qwen/Qwen3-Reranker-0.6B),
|
||||
reverted to baseline `classifier_from_token:["no","yes"]`, healthy. Compose:
|
||||
`/opt/docker/compose/vllm/compose.yaml` (canonical mirror
|
||||
`stacks/vllm/compose.yaml`). Gateway alias `reranker`/`qwen3-reranker` →
|
||||
@@ -75,7 +75,7 @@ implementation.
|
||||
worldtree-dev signals the lever run is done).
|
||||
- **Dual query-set requirement (hard):** score bare-name anchor queries AND
|
||||
question-shaped; bar = recovering the name-lookup class.
|
||||
- **VRAM:** 4B ≈ 4–5 GB fp8, 8B ≈ 9 GB; ana-ml2 Blackwell has headroom.
|
||||
- **VRAM:** 4B ≈ 4–5 GB fp8, 8B ≈ 9 GB; fv-ml1 Blackwell has headroom.
|
||||
|
||||
---
|
||||
|
||||
@@ -84,12 +84,12 @@ implementation.
|
||||
### 2026-08-06 — A2 brought up (Brokkr thread 01KZBSTSJA…)
|
||||
|
||||
- **Backend:** `vllm-rerank-a2` — standalone `docker run` (NOT in the vllm compose
|
||||
stack), on ana-ml2 **GPU1**, host port **:8012** → container 8000. Image
|
||||
stack), on fv-ml1 **GPU1**, host port **:8012** → container 8000. Image
|
||||
`vllm/vllm-openai:latest` (=0.24.0). Args: model
|
||||
`tomaarsen/Qwen3-Reranker-0.6B-seq-cls`, `--runner pooling`, `--gpu-memory-utilization
|
||||
0.03`, `--max-model-len 8192`, `--dtype auto`, `--restart no`. Native
|
||||
`Qwen3ForSequenceClassification` — NO hf-overrides. Routes /rerank /score /classify.
|
||||
- **Gateway alias:** `reranker-a2-qwen3-seqcls` → `http://10.250.50.54:8012/v1`,
|
||||
- **Gateway alias:** `reranker-a2-qwen3-seqcls` → `http://10.251.50.54:8012/v1`,
|
||||
mode rerank. Added via LiteLLM **`/model/new`** (DB-backed, `store_model_in_db:true`)
|
||||
— **no gateway restart** (respects the "nothing else offline" line). Verified 200
|
||||
through the gateway.
|
||||
@@ -106,7 +106,7 @@ implementation.
|
||||
- D4 — did NOT down any RP seat (A2 is 0.6B / 3.5 GB; no VRAM pressure).
|
||||
|
||||
**Cleanup for A2 (run at end / on reversal):**
|
||||
- `ssh infra-ops@10.250.50.54 'sudo docker stop vllm-rerank-a2 && sudo docker rm vllm-rerank-a2'`
|
||||
- `ssh infra-ops@10.251.50.54 'sudo docker stop vllm-rerank-a2 && sudo docker rm vllm-rerank-a2'`
|
||||
- Delete gateway alias: `POST /model/delete {"id": <model_id>}` (id via `/model/info?model_name=reranker-a2-qwen3-seqcls`), infra-ops admin key. (DB-persisted, so it survives a restart — must be explicitly deleted.)
|
||||
- No weights deleted (red line); HF cache under /tank/aimodels/huggingface retains the 0.6B-seq-cls download.
|
||||
|
||||
@@ -114,11 +114,11 @@ implementation.
|
||||
|
||||
### 2026-08-06 — A3 + A4 pre-staged (Brokkr said pre-stage in parallel, hold A5)
|
||||
|
||||
- **A3** `vllm-rerank-a3` — ana-ml2 GPU1 :8013, `BAAI/bge-reranker-v2-m3`
|
||||
- **A3** `vllm-rerank-a3` — fv-ml1 GPU1 :8013, `BAAI/bge-reranker-v2-m3`
|
||||
(XLMRobertaForSequenceClassification), same run pattern, util 0.03. VRAM ≈ **2.3 GB**.
|
||||
Latency (20-doc, ~1500-char, shared GPU1): single p50 **105 ms**; 8-conc p50 214 ms, ~34 req/s.
|
||||
Gateway alias `reranker-a3-bge-v2-m3` via /model/new (200, verified).
|
||||
- **A4** `vllm-rerank-a4` — ana-ml2 GPU1 :8014, `Alibaba-NLP/gte-reranker-modernbert-base`
|
||||
- **A4** `vllm-rerank-a4` — fv-ml1 GPU1 :8014, `Alibaba-NLP/gte-reranker-modernbert-base`
|
||||
(ModernBertForSequenceClassification), util 0.02. VRAM ≈ **1.4 GB**. Latency: single
|
||||
p50 **102 ms**; 8-conc p50 153 ms, ~51 req/s. Gateway alias `reranker-a4-gte-modernbert`
|
||||
via /model/new (200, verified).
|
||||
@@ -129,7 +129,7 @@ implementation.
|
||||
If A5 (4B, ~4–5 GB) is greenlit: fits GPU1 tight or GPU0 (~9 GB free) — no RP-seat downing expected.
|
||||
|
||||
**Cleanup for A3/A4 (same pattern as A2):** `docker stop/rm vllm-rerank-a3 vllm-rerank-a4`
|
||||
on ana-ml2; `/model/delete` the two aliases (DB-persisted); weights retained in HF cache.
|
||||
on fv-ml1; `/model/delete` the two aliases (DB-persisted); weights retained in HF cache.
|
||||
|
||||
### 2026-08-06 — A2 verdict (Brokkr full grid): training-prior confirmed
|
||||
|
||||
@@ -202,7 +202,7 @@ on ana-ml2; `/model/delete` the two aliases (DB-persisted); weights retained in
|
||||
Brokkr signals the bake-off is closed (`docker stop/rm` + `/model/delete`).
|
||||
- **A4 (`vllm-rerank-a4` :8014)** + alias — KEEP for now (Brokkr's documented throughput fallback).
|
||||
- **A3 (`vllm-rerank-a3` :8013)** — now PRODUCTION (backs the `reranker` alias). Hardened
|
||||
2026-08-06: `docker update --restart unless-stopped` (survives ana-ml2 reboot, no recreate).
|
||||
2026-08-06: `docker update --restart unless-stopped` (survives fv-ml1 reboot, no recreate).
|
||||
A4 given the same. **Remaining follow-up (not urgent): promote A3 from throwaway `docker run`
|
||||
to a canonical compose service** (`stacks/vllm/`) for config-managed consistency — a recreate,
|
||||
so do it in a window since it briefly drops `reranker`.
|
||||
|
||||
Reference in New Issue
Block a user