docs: fv-ml1 cutover runbook (ana-ml2 -> Fountain Valley colo)
This commit is contained in:
@@ -0,0 +1,76 @@
|
||||
# Cutover: ana-ml2 → fv-ml1 (Anaheim → Fountain Valley colo)
|
||||
|
||||
**Why:** ana-ml2's power draw (dual Blackwell PRO 6000, ~1.5 kW peak) repeatedly
|
||||
trips the Anaheim rack breaker — the root cause of the 2026-08-26 and 2026-09-11
|
||||
whole-site outages. Moving it to the Fountain Valley colo on its own circuit.
|
||||
|
||||
**Clean-cutover property:** the box is already down (BMC dark, no power) as of the
|
||||
2026-09-11 outage, so this is a relocation, not a take-down. `/tank` is LOCAL ZFS
|
||||
(no NFS from ana-nas — verified), so the data travels with the chassis; nothing
|
||||
copies over the wire.
|
||||
|
||||
## Address map (same shape as Anaheim)
|
||||
|
||||
| thing | Anaheim (old) | Fountain Valley (new) |
|
||||
|---|---|---|
|
||||
| server subnet | 10.250.0.0/16 | **10.251.0.0/16** |
|
||||
| box | `ana-ml2` 10.250.50.54 | **`fv-ml1` 10.251.50.54** ⚠ host octet mirrored — confirm |
|
||||
| BMC | `ana-ml2-bmc` 10.250.250.50 | **`fv-ml1-bmc` 10.251.250.50** |
|
||||
| site gateway / OPNsense | ana-gw 10.250.0.1 | **OPNsense 10.251.0.1** ⚠ confirm (op wrote 10.250.x.1) |
|
||||
| VLAN gateways | — | server 10.251.50.1 · mgmt 10.251.250.1 |
|
||||
| DNS name | ana-ml2.ana.internal | **fv-ml1.fv.internal** (full new `fv` site) |
|
||||
| mesh | via ana-scale | **OPNsense = tailscale subnet-router for 10.251.0.0/16** |
|
||||
|
||||
⚠ **OPEN:** FV resolver for the full `fv` site (AdGuard on OPNsense, or a container
|
||||
on fv-ml1?). Until decided, fv-ml1 resolves via the existing 3 resolvers.
|
||||
|
||||
## Mesh: OPNsense as subnet-router (NOT ana-ml2)
|
||||
|
||||
`os-tailscale` plugin on the OPNsense, advertise `10.251.0.0/16`, approve the route
|
||||
in headscale. Rationale: mirrors ana-scale/nh3-scale/esh-scale; **stays up when
|
||||
fv-ml1 is powered off, so the BMC stays reachable over the mesh** — the out-of-band
|
||||
path the fleet lacked on 2026-09-11. Do NOT tie the mesh node to fv-ml1 itself.
|
||||
|
||||
## Blast radius — every reference to repoint (from `grep 10.250.50.54`)
|
||||
|
||||
**Load-bearing (traffic breaks if missed):**
|
||||
- [ ] `stacks/litellm/conf/config.yaml` — ~10 `api_base: http://10.250.50.54:{8015,8016,8018,8019}/v1` → `10.251.50.54`. **THE critical one** — every inference alias (gen/summarizer/classifier/image-judge/mog-sec/char-rp/meromero). Apply + `docker compose up -d litellm` on ana-docker AT cutover (box must be up at FV first).
|
||||
- [ ] `dns/internal.yaml` — add `fv` site (needs resolver), add `fv-ml1` + `fv-ml1-bmc` records, remove/retire `ana-ml2` + `ana-ml2-bmc`. Then `scripts/dns-sync.py --dry-run` → apply.
|
||||
- [ ] mesh route to 10.251.0.0/16 approved in headscale (OPNsense subnet-router).
|
||||
- [ ] `stacks/ana-ml2-proxy/` — the Homepage Docker-API relay (10.250.50.54:2375). Rename/repoint to the FV IP or retire if Homepage reaches FV another way.
|
||||
|
||||
**Cosmetic (no traffic break — rename sweep):**
|
||||
- [ ] homepage `href`/`ping` labels: stacks/{llama-swap,gemma4-charrp,meromero-charrp,erp-seat,sglang,heretic2-charrp-reasoning}/compose.yaml, stacks/homepage/conf/services.yaml
|
||||
- [ ] servers/ana-ml2/ → servers/fv-ml1/ (dir rename), ssh-target 10.250.50.54→10.251.50.54, ssh_config, remote-ssh allowlist (10.0.0.0/8 already covers 10.251 — no change)
|
||||
- [ ] playbooks/*ana-ml2*, elway invocations infra-ops@10.250.50.54 → @10.251.50.54
|
||||
- [ ] stacks/beszel deploy target; docs/*, README.md, CLAUDE.md server table, disaster-recovery.md
|
||||
- [ ] model `.PROVENANCE.txt` files reference ana-ml2 in prose (accuracy, not load-bearing)
|
||||
|
||||
## Order of operations (cutover day)
|
||||
|
||||
1. Physically install fv-ml1 at FV, on its dedicated circuit. Confirm the circuit
|
||||
handles ~1.5 kW peak (the whole point).
|
||||
2. OPNsense up: LAN 10.251.50.1 / mgmt 10.251.250.1, WAN, `os-tailscale` subnet-router
|
||||
for 10.251.0.0/16, approve route in headscale.
|
||||
3. fv-ml1 up on 10.251.50.54, BMC on 10.251.250.50. Verify BMC reachable over mesh
|
||||
(out-of-band check).
|
||||
4. Bring up the vLLM seats (same /tank, same compose) — verify :8015/:8019 serve locally.
|
||||
5. Apply the LiteLLM api_base repoint on ana-docker → `up -d litellm` → verify
|
||||
`gen`/`summarizer`/`mog-sec` route to FV.
|
||||
6. DNS: internal.yaml `fv` site + records → `dns-sync.py`.
|
||||
7. Rename sweep (cosmetic) — a scripted `git mv` + sed pass, one commit.
|
||||
8. Beszel/Homepage repoint; verify dashboards.
|
||||
|
||||
## Rollback
|
||||
|
||||
The physical move is not reversible same-day, but the *config* is: keep the old
|
||||
`ana-ml2` blocks in git history; if FV networking isn't ready, the seats simply
|
||||
stay down (they already are) — no worse than the current state. Do NOT delete the
|
||||
ana-ml2 DNS/LiteLLM entries until FV serves; comment them.
|
||||
|
||||
## Cross-site note
|
||||
|
||||
LiteLLM gateway stays on ana-docker (Anaheim); fv-ml1 serves cross-site. FV↔Anaheim
|
||||
is metro (both Orange County, ~10 mi) so latency is a non-issue, but it's a NEW
|
||||
dependency: if the FV↔Anaheim link drops, inference dark. Acceptable; revisit
|
||||
moving the gateway to FV if FV becomes the inference hub.
|
||||
Reference in New Issue
Block a user