From ce04f9dbdeafa5f5a0ed13a8d41096afc3c28415 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 11 Sep 2026 19:11:32 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20finalize=20fv-ml1=20cutover=20=E2=80=94?= =?UTF-8?q?=20confirmed=20addresses,=20piggyback=20DNS,=20exact=20commands?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/runbooks/fv-ml1-cutover.md | 30 +++++++++++++++++++++++------- 1 file changed, 23 insertions(+), 7 deletions(-) diff --git a/docs/runbooks/fv-ml1-cutover.md b/docs/runbooks/fv-ml1-cutover.md index 32b11cc..cd99671 100644 --- a/docs/runbooks/fv-ml1-cutover.md +++ b/docs/runbooks/fv-ml1-cutover.md @@ -14,15 +14,20 @@ copies over the wire. | thing | Anaheim (old) | Fountain Valley (new) | |---|---|---| | server subnet | 10.250.0.0/16 | **10.251.0.0/16** | -| box | `ana-ml2` 10.250.50.54 | **`fv-ml1` 10.251.50.54** ⚠ host octet mirrored — confirm | +| box | `ana-ml2` 10.250.50.54 | **`fv-ml1` 10.251.50.54** (confirmed) | | BMC | `ana-ml2-bmc` 10.250.250.50 | **`fv-ml1-bmc` 10.251.250.50** | -| site gateway / OPNsense | ana-gw 10.250.0.1 | **OPNsense 10.251.0.1** ⚠ confirm (op wrote 10.250.x.1) | -| VLAN gateways | — | server 10.251.50.1 · mgmt 10.251.250.1 | +| site gateway / OPNsense | ana-gw 10.250.0.1 | **OPNsense = `.1` in EVERY FV VLAN** (server 10.251.50.1, mgmt 10.251.250.1, …) — multi-homed gateway | | DNS name | ana-ml2.ana.internal | **fv-ml1.fv.internal** (full new `fv` site) | | mesh | via ana-scale | **OPNsense = tailscale subnet-router for 10.251.0.0/16** | -⚠ **OPEN:** FV resolver for the full `fv` site (AdGuard on OPNsense, or a container -on fv-ml1?). Until decided, fv-ml1 resolves via the existing 3 resolvers. +**DNS approach (decided): PIGGYBACK now, promote later.** `dns-sync` builds each +FQDN as `name.site.zone` with NO check that the site is in the `sites:` block, and +publishes every record to every resolver. So `fv-ml1` + `fv-ml1-bmc` records with +`site: fv` resolve fleet-wide from the existing ana/esh/nh3 resolvers immediately — +you get the full `fv-ml1.fv.internal` name with zero new infra. Add a real `fv` +resolver (AdGuard container on a future FV utility box, or extend dns-sync for +OPNsense Unbound) only when FV needs LOCAL resolution. Do NOT add `fv` to `sites:` +until that resolver exists — the sync would fail trying to reach it. ## Mesh: OPNsense as subnet-router (NOT ana-ml2) @@ -34,8 +39,19 @@ path the fleet lacked on 2026-09-11. Do NOT tie the mesh node to fv-ml1 itself. ## Blast radius — every reference to repoint (from `grep 10.250.50.54`) **Load-bearing (traffic breaks if missed):** -- [ ] `stacks/litellm/conf/config.yaml` — ~10 `api_base: http://10.250.50.54:{8015,8016,8018,8019}/v1` → `10.251.50.54`. **THE critical one** — every inference alias (gen/summarizer/classifier/image-judge/mog-sec/char-rp/meromero). Apply + `docker compose up -d litellm` on ana-docker AT cutover (box must be up at FV first). -- [ ] `dns/internal.yaml` — add `fv` site (needs resolver), add `fv-ml1` + `fv-ml1-bmc` records, remove/retire `ana-ml2` + `ana-ml2-bmc`. Then `scripts/dns-sync.py --dry-run` → apply. +- [ ] `stacks/litellm/conf/config.yaml` — **THE critical one** (every inference alias: gen/summarizer/classifier/image-judge/mog-sec/char-rp/meromero). At cutover, once fv-ml1 serves at FV: + ```bash + sed -i 's/10\.250\.50\.54/10.251.50.54/g' stacks/litellm/conf/config.yaml + scripts/deploy-stack.sh infra-ops@10.250.50.70 litellm --conf # push to ana-docker + ssh infra-ops@10.250.50.70 'cd /opt/docker/compose/litellm && sudo docker compose up -d litellm' + ``` + Verify: `gen`/`summarizer`/`mog-sec` route to fv-ml1. ⚠ Gateway STAYS on ana-docker → cross-site to FV (metro link, fine). +- [ ] `dns/internal.yaml` — add the two records below (piggyback; NOT a `sites:` entry), comment out the old `ana-ml2` / `ana-ml2-bmc` lines. Then `scripts/dns-sync.py --dry-run` → apply. + ```yaml + # ---- fv: Fountain Valley colo (piggyback DNS until FV has its own resolver) ---- + - {name: fv-ml1, site: fv, v4: 10.251.50.54, note: GPU inference, dual RTX PRO 6000 (ex ana-ml2, relocated 2026-09)} + - {name: fv-ml1-bmc, site: fv, v4: 10.251.250.50, note: BMC for fv-ml1} + ``` - [ ] mesh route to 10.251.0.0/16 approved in headscale (OPNsense subnet-router). - [ ] `stacks/ana-ml2-proxy/` — the Homepage Docker-API relay (10.250.50.54:2375). Rename/repoint to the FV IP or retire if Homepage reaches FV another way.