93 lines
5.7 KiB
Markdown
93 lines
5.7 KiB
Markdown
# Cutover: ana-ml2 → fv-ml1 (Anaheim → Fountain Valley colo)
|
|
|
|
**Why:** ana-ml2's power draw (dual Blackwell PRO 6000, ~1.5 kW peak) repeatedly
|
|
trips the Anaheim rack breaker — the root cause of the 2026-08-26 and 2026-09-11
|
|
whole-site outages. Moving it to the Fountain Valley colo on its own circuit.
|
|
|
|
**Clean-cutover property:** the box is already down (BMC dark, no power) as of the
|
|
2026-09-11 outage, so this is a relocation, not a take-down. `/tank` is LOCAL ZFS
|
|
(no NFS from ana-nas — verified), so the data travels with the chassis; nothing
|
|
copies over the wire.
|
|
|
|
## Address map (same shape as Anaheim)
|
|
|
|
| thing | Anaheim (old) | Fountain Valley (new) |
|
|
|---|---|---|
|
|
| server subnet | 10.250.0.0/16 | **10.251.0.0/16** |
|
|
| box | `ana-ml2` 10.250.50.54 | **`fv-ml1` 10.251.50.54** (confirmed) |
|
|
| BMC | `ana-ml2-bmc` 10.250.250.50 | **`fv-ml1-bmc` 10.251.250.50** |
|
|
| site gateway / OPNsense | ana-gw 10.250.0.1 | **OPNsense = `.1` in EVERY FV VLAN** (server 10.251.50.1, mgmt 10.251.250.1, …) — multi-homed gateway |
|
|
| DNS name | ana-ml2.ana.internal | **fv-ml1.fv.internal** (full new `fv` site) |
|
|
| mesh | via ana-scale | **OPNsense = tailscale subnet-router for 10.251.0.0/16** |
|
|
|
|
**DNS approach (decided): PIGGYBACK now, promote later.** `dns-sync` builds each
|
|
FQDN as `name.site.zone` with NO check that the site is in the `sites:` block, and
|
|
publishes every record to every resolver. So `fv-ml1` + `fv-ml1-bmc` records with
|
|
`site: fv` resolve fleet-wide from the existing ana/esh/nh3 resolvers immediately —
|
|
you get the full `fv-ml1.fv.internal` name with zero new infra. Add a real `fv`
|
|
resolver (AdGuard container on a future FV utility box, or extend dns-sync for
|
|
OPNsense Unbound) only when FV needs LOCAL resolution. Do NOT add `fv` to `sites:`
|
|
until that resolver exists — the sync would fail trying to reach it.
|
|
|
|
## Mesh: OPNsense as subnet-router (NOT ana-ml2)
|
|
|
|
`os-tailscale` plugin on the OPNsense, advertise `10.251.0.0/16`, approve the route
|
|
in headscale. Rationale: mirrors ana-scale/nh3-scale/esh-scale; **stays up when
|
|
fv-ml1 is powered off, so the BMC stays reachable over the mesh** — the out-of-band
|
|
path the fleet lacked on 2026-09-11. Do NOT tie the mesh node to fv-ml1 itself.
|
|
|
|
## Blast radius — every reference to repoint (from `grep 10.250.50.54`)
|
|
|
|
**Load-bearing (traffic breaks if missed):**
|
|
- [ ] `stacks/litellm/conf/config.yaml` — **THE critical one** (every inference alias: gen/summarizer/classifier/image-judge/mog-sec/char-rp/meromero). At cutover, once fv-ml1 serves at FV:
|
|
```bash
|
|
sed -i 's/10\.250\.50\.54/10.251.50.54/g' stacks/litellm/conf/config.yaml
|
|
scripts/deploy-stack.sh infra-ops@10.250.50.70 litellm --conf # push to ana-docker
|
|
ssh infra-ops@10.250.50.70 'cd /opt/docker/compose/litellm && sudo docker compose up -d litellm'
|
|
```
|
|
Verify: `gen`/`summarizer`/`mog-sec` route to fv-ml1. ⚠ Gateway STAYS on ana-docker → cross-site to FV (metro link, fine).
|
|
- [ ] `dns/internal.yaml` — add the two records below (piggyback; NOT a `sites:` entry), comment out the old `ana-ml2` / `ana-ml2-bmc` lines. Then `scripts/dns-sync.py --dry-run` → apply.
|
|
```yaml
|
|
# ---- fv: Fountain Valley colo (piggyback DNS until FV has its own resolver) ----
|
|
- {name: fv-ml1, site: fv, v4: 10.251.50.54, note: GPU inference, dual RTX PRO 6000 (ex ana-ml2, relocated 2026-09)}
|
|
- {name: fv-ml1-bmc, site: fv, v4: 10.251.250.50, note: BMC for fv-ml1}
|
|
```
|
|
- [ ] mesh route to 10.251.0.0/16 approved in headscale (OPNsense subnet-router).
|
|
- [ ] `stacks/ana-ml2-proxy/` — the Homepage Docker-API relay (10.250.50.54:2375). Rename/repoint to the FV IP or retire if Homepage reaches FV another way.
|
|
|
|
**Cosmetic (no traffic break — rename sweep):**
|
|
- [ ] homepage `href`/`ping` labels: stacks/{llama-swap,gemma4-charrp,meromero-charrp,erp-seat,sglang,heretic2-charrp-reasoning}/compose.yaml, stacks/homepage/conf/services.yaml
|
|
- [ ] servers/ana-ml2/ → servers/fv-ml1/ (dir rename), ssh-target 10.250.50.54→10.251.50.54, ssh_config, remote-ssh allowlist (10.0.0.0/8 already covers 10.251 — no change)
|
|
- [ ] playbooks/*ana-ml2*, elway invocations infra-ops@10.250.50.54 → @10.251.50.54
|
|
- [ ] stacks/beszel deploy target; docs/*, README.md, CLAUDE.md server table, disaster-recovery.md
|
|
- [ ] model `.PROVENANCE.txt` files reference ana-ml2 in prose (accuracy, not load-bearing)
|
|
|
|
## Order of operations (cutover day)
|
|
|
|
1. Physically install fv-ml1 at FV, on its dedicated circuit. Confirm the circuit
|
|
handles ~1.5 kW peak (the whole point).
|
|
2. OPNsense up: LAN 10.251.50.1 / mgmt 10.251.250.1, WAN, `os-tailscale` subnet-router
|
|
for 10.251.0.0/16, approve route in headscale.
|
|
3. fv-ml1 up on 10.251.50.54, BMC on 10.251.250.50. Verify BMC reachable over mesh
|
|
(out-of-band check).
|
|
4. Bring up the vLLM seats (same /tank, same compose) — verify :8015/:8019 serve locally.
|
|
5. Apply the LiteLLM api_base repoint on ana-docker → `up -d litellm` → verify
|
|
`gen`/`summarizer`/`mog-sec` route to FV.
|
|
6. DNS: internal.yaml `fv` site + records → `dns-sync.py`.
|
|
7. Rename sweep (cosmetic) — a scripted `git mv` + sed pass, one commit.
|
|
8. Beszel/Homepage repoint; verify dashboards.
|
|
|
|
## Rollback
|
|
|
|
The physical move is not reversible same-day, but the *config* is: keep the old
|
|
`ana-ml2` blocks in git history; if FV networking isn't ready, the seats simply
|
|
stay down (they already are) — no worse than the current state. Do NOT delete the
|
|
ana-ml2 DNS/LiteLLM entries until FV serves; comment them.
|
|
|
|
## Cross-site note
|
|
|
|
LiteLLM gateway stays on ana-docker (Anaheim); fv-ml1 serves cross-site. FV↔Anaheim
|
|
is metro (both Orange County, ~10 mi) so latency is a non-issue, but it's a NEW
|
|
dependency: if the FV↔Anaheim link drops, inference dark. Acceptable; revisit
|
|
moving the gateway to FV if FV becomes the inference hub.
|