From a13ab598aeefd78133ad567ae32f9c34aa4ca0b4 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 11 Sep 2026 19:08:07 -0700 Subject: [PATCH] docs: fv-ml1 cutover runbook (ana-ml2 -> Fountain Valley colo) --- docs/runbooks/fv-ml1-cutover.md | 76 +++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 docs/runbooks/fv-ml1-cutover.md diff --git a/docs/runbooks/fv-ml1-cutover.md b/docs/runbooks/fv-ml1-cutover.md new file mode 100644 index 0000000..32b11cc --- /dev/null +++ b/docs/runbooks/fv-ml1-cutover.md @@ -0,0 +1,76 @@ +# Cutover: ana-ml2 → fv-ml1 (Anaheim → Fountain Valley colo) + +**Why:** ana-ml2's power draw (dual Blackwell PRO 6000, ~1.5 kW peak) repeatedly +trips the Anaheim rack breaker — the root cause of the 2026-08-26 and 2026-09-11 +whole-site outages. Moving it to the Fountain Valley colo on its own circuit. + +**Clean-cutover property:** the box is already down (BMC dark, no power) as of the +2026-09-11 outage, so this is a relocation, not a take-down. `/tank` is LOCAL ZFS +(no NFS from ana-nas — verified), so the data travels with the chassis; nothing +copies over the wire. + +## Address map (same shape as Anaheim) + +| thing | Anaheim (old) | Fountain Valley (new) | +|---|---|---| +| server subnet | 10.250.0.0/16 | **10.251.0.0/16** | +| box | `ana-ml2` 10.250.50.54 | **`fv-ml1` 10.251.50.54** ⚠ host octet mirrored — confirm | +| BMC | `ana-ml2-bmc` 10.250.250.50 | **`fv-ml1-bmc` 10.251.250.50** | +| site gateway / OPNsense | ana-gw 10.250.0.1 | **OPNsense 10.251.0.1** ⚠ confirm (op wrote 10.250.x.1) | +| VLAN gateways | — | server 10.251.50.1 · mgmt 10.251.250.1 | +| DNS name | ana-ml2.ana.internal | **fv-ml1.fv.internal** (full new `fv` site) | +| mesh | via ana-scale | **OPNsense = tailscale subnet-router for 10.251.0.0/16** | + +⚠ **OPEN:** FV resolver for the full `fv` site (AdGuard on OPNsense, or a container +on fv-ml1?). Until decided, fv-ml1 resolves via the existing 3 resolvers. + +## Mesh: OPNsense as subnet-router (NOT ana-ml2) + +`os-tailscale` plugin on the OPNsense, advertise `10.251.0.0/16`, approve the route +in headscale. Rationale: mirrors ana-scale/nh3-scale/esh-scale; **stays up when +fv-ml1 is powered off, so the BMC stays reachable over the mesh** — the out-of-band +path the fleet lacked on 2026-09-11. Do NOT tie the mesh node to fv-ml1 itself. + +## Blast radius — every reference to repoint (from `grep 10.250.50.54`) + +**Load-bearing (traffic breaks if missed):** +- [ ] `stacks/litellm/conf/config.yaml` — ~10 `api_base: http://10.250.50.54:{8015,8016,8018,8019}/v1` → `10.251.50.54`. **THE critical one** — every inference alias (gen/summarizer/classifier/image-judge/mog-sec/char-rp/meromero). Apply + `docker compose up -d litellm` on ana-docker AT cutover (box must be up at FV first). +- [ ] `dns/internal.yaml` — add `fv` site (needs resolver), add `fv-ml1` + `fv-ml1-bmc` records, remove/retire `ana-ml2` + `ana-ml2-bmc`. Then `scripts/dns-sync.py --dry-run` → apply. +- [ ] mesh route to 10.251.0.0/16 approved in headscale (OPNsense subnet-router). +- [ ] `stacks/ana-ml2-proxy/` — the Homepage Docker-API relay (10.250.50.54:2375). Rename/repoint to the FV IP or retire if Homepage reaches FV another way. + +**Cosmetic (no traffic break — rename sweep):** +- [ ] homepage `href`/`ping` labels: stacks/{llama-swap,gemma4-charrp,meromero-charrp,erp-seat,sglang,heretic2-charrp-reasoning}/compose.yaml, stacks/homepage/conf/services.yaml +- [ ] servers/ana-ml2/ → servers/fv-ml1/ (dir rename), ssh-target 10.250.50.54→10.251.50.54, ssh_config, remote-ssh allowlist (10.0.0.0/8 already covers 10.251 — no change) +- [ ] playbooks/*ana-ml2*, elway invocations infra-ops@10.250.50.54 → @10.251.50.54 +- [ ] stacks/beszel deploy target; docs/*, README.md, CLAUDE.md server table, disaster-recovery.md +- [ ] model `.PROVENANCE.txt` files reference ana-ml2 in prose (accuracy, not load-bearing) + +## Order of operations (cutover day) + +1. Physically install fv-ml1 at FV, on its dedicated circuit. Confirm the circuit + handles ~1.5 kW peak (the whole point). +2. OPNsense up: LAN 10.251.50.1 / mgmt 10.251.250.1, WAN, `os-tailscale` subnet-router + for 10.251.0.0/16, approve route in headscale. +3. fv-ml1 up on 10.251.50.54, BMC on 10.251.250.50. Verify BMC reachable over mesh + (out-of-band check). +4. Bring up the vLLM seats (same /tank, same compose) — verify :8015/:8019 serve locally. +5. Apply the LiteLLM api_base repoint on ana-docker → `up -d litellm` → verify + `gen`/`summarizer`/`mog-sec` route to FV. +6. DNS: internal.yaml `fv` site + records → `dns-sync.py`. +7. Rename sweep (cosmetic) — a scripted `git mv` + sed pass, one commit. +8. Beszel/Homepage repoint; verify dashboards. + +## Rollback + +The physical move is not reversible same-day, but the *config* is: keep the old +`ana-ml2` blocks in git history; if FV networking isn't ready, the seats simply +stay down (they already are) — no worse than the current state. Do NOT delete the +ana-ml2 DNS/LiteLLM entries until FV serves; comment them. + +## Cross-site note + +LiteLLM gateway stays on ana-docker (Anaheim); fv-ml1 serves cross-site. FV↔Anaheim +is metro (both Orange County, ~10 mi) so latency is a non-issue, but it's a NEW +dependency: if the FV↔Anaheim link drops, inference dark. Acceptable; revisit +moving the gateway to FV if FV becomes the inference hub.