Files

T

vh 91bda3c480 fv-ml1: complete the cutover — rename, renumber, DNS, and the LiteLLM repoint

The box is physically at Fountain Valley, renamed, renumbered onto 10.251/16,
and serving inference again. This lands the repo half of that.

Host: hostname ana-ml2 -> fv-ml1, pinned to 10.251.50.54 by a dnsmasq
reservation so the address the runbook, DNS and LiteLLM all assume is the
address it actually has. Its headscale node is renamed too.

The sweep ran from scripts/fv-ml1-rename-sweep.sh, whose allowlist is the
reason this diff touches current-state files and not the record. Dated
persistent-memory entries, archival-memory and incident notes still say
ana-ml2 in 31 and 62 places respectively, because that is what the box was
when those things happened. Rewriting them would make the history lie.

LiteLLM was the load-bearing piece and needed more than the api_base sed the
runbook describes. Twenty api_base entries repointed, but a grep-and-verify
pass also caught a LIVE pass_through_endpoints target for the scalar-judge
reward route still on the old address -- an api_base-only substitution would
have left it dead. Four prose references describing current state were
repointed as well; one historical note recording where a hand-test was run
is deliberately left pointing at 10.250.50.54.

Two facts in the server tables were wrong and are corrected here. The site is
Fountain Valley, not Anaheim. And the box has FOUR RTX PRO 6000 Blackwell
Max-Q, not two -- verified by nvidia-smi -L and independently by PCI
enumeration of four GB202GL devices. That is 391 GB of VRAM rather than 196,
which changes what fits on it.

DNS: fv-ml1, fv-ml1-bmc and fv-gw added under the fv site via the piggyback
approach, scriberr re-homed, and the ana-ml2 records removed. Applied to all
three resolvers. The BMC record carries a warning that its 802.1q VLAN tag
must stay disabled -- it shipped tagging VLAN 250 into an untagged port,
which made it invisible to every network-side diagnostic and is the reason
it appeared dead through several cable changes.

Verified end to end: summarizer and sec both answer through the Anaheim
gateway across the mesh to FV seats on different ports.

2026-09-12 22:00:50 -07:00

5.7 KiB

Raw Permalink Blame History

Cutover: ana-ml2 → fv-ml1 (Anaheim → Fountain Valley colo)

Why: ana-ml2's power draw (dual Blackwell PRO 6000, ~1.5 kW peak) repeatedly trips the Anaheim rack breaker — the root cause of the 2026-08-26 and 2026-09-11 whole-site outages. Moving it to the Fountain Valley colo on its own circuit.

Clean-cutover property: the box is already down (BMC dark, no power) as of the 2026-09-11 outage, so this is a relocation, not a take-down. /tank is LOCAL ZFS (no NFS from ana-nas — verified), so the data travels with the chassis; nothing copies over the wire.

Address map (same shape as Anaheim)

thing	Anaheim (old)	Fountain Valley (new)
server subnet	10.250.0.0/16	10.251.0.0/16
box	`ana-ml2` 10.250.50.54	`fv-ml1` 10.251.50.54 (confirmed)
BMC	`ana-ml2-bmc` 10.250.250.50	`fv-ml1-bmc` 10.251.250.50
site gateway / OPNsense	ana-gw 10.250.0.1	OPNsense = `.1` in EVERY FV VLAN (server 10.251.50.1, mgmt 10.251.250.1, …) — multi-homed gateway
DNS name	ana-ml2.ana.internal	fv-ml1.fv.internal (full new `fv` site)
mesh	via ana-scale	OPNsense = tailscale subnet-router for 10.251.0.0/16

DNS approach (decided): PIGGYBACK now, promote later. dns-sync builds each FQDN as name.site.zone with NO check that the site is in the sites: block, and publishes every record to every resolver. So fv-ml1 + fv-ml1-bmc records with site: fv resolve fleet-wide from the existing ana/esh/nh3 resolvers immediately — you get the full fv-ml1.fv.internal name with zero new infra. Add a real fv resolver (AdGuard container on a future FV utility box, or extend dns-sync for OPNsense Unbound) only when FV needs LOCAL resolution. Do NOT add fv to sites: until that resolver exists — the sync would fail trying to reach it.

Mesh: OPNsense as subnet-router (NOT ana-ml2)

os-tailscale plugin on the OPNsense, advertise 10.251.0.0/16, approve the route in headscale. Rationale: mirrors ana-scale/nh3-scale/esh-scale; stays up when fv-ml1 is powered off, so the BMC stays reachable over the mesh — the out-of-band path the fleet lacked on 2026-09-11. Do NOT tie the mesh node to fv-ml1 itself.

Blast radius — every reference to repoint (from `grep 10.250.50.54`)

Load-bearing (traffic breaks if missed):

stacks/litellm/conf/config.yaml — THE critical one (every inference alias: gen/summarizer/classifier/image-judge/mog-sec/char-rp/meromero). At cutover, once fv-ml1 serves at FV:
```
sed -i 's/10\.250\.50\.54/10.251.50.54/g' stacks/litellm/conf/config.yaml
scripts/deploy-stack.sh infra-ops@10.250.50.70 litellm --conf   # push to ana-docker
ssh infra-ops@10.250.50.70 'cd /opt/docker/compose/litellm && sudo docker compose up -d litellm'
```
Verify: gen/summarizer/mog-sec route to fv-ml1. ⚠ Gateway STAYS on ana-docker → cross-site to FV (metro link, fine).

dns/internal.yaml — add the two records below (piggyback; NOT a sites: entry), comment out the old ana-ml2 / ana-ml2-bmc lines. Then scripts/dns-sync.py --dry-run → apply.

# ---- fv: Fountain Valley colo (piggyback DNS until FV has its own resolver) ----
- {name: fv-ml1,     site: fv, v4: 10.251.50.54,  note: GPU inference, 4x RTX PRO 6000 Blackwell Max-Q (ex ana-ml2, relocated 2026-09)}
- {name: fv-ml1-bmc, site: fv, v4: 10.251.250.50, note: BMC for fv-ml1}

mesh route to 10.251.0.0/16 approved in headscale (OPNsense subnet-router).
stacks/ana-ml2-proxy/ — the Homepage Docker-API relay (10.250.50.54:2375). Rename/repoint to the FV IP or retire if Homepage reaches FV another way.

Cosmetic (no traffic break — rename sweep):

homepage href/ping labels: stacks/{llama-swap,gemma4-charrp,meromero-charrp,erp-seat,sglang,heretic2-charrp-reasoning}/compose.yaml, stacks/homepage/conf/services.yaml
servers/ana-ml2/ → servers/fv-ml1/ (dir rename), ssh-target 10.250.50.54→10.251.50.54, ssh_config, remote-ssh allowlist (10.0.0.0/8 already covers 10.251 — no change)
playbooks/ana-ml2, elway invocations infra-ops@10.250.50.54 → @10.251.50.54
stacks/beszel deploy target; docs/*, README.md, CLAUDE.md server table, disaster-recovery.md
model .PROVENANCE.txt files reference ana-ml2 in prose (accuracy, not load-bearing)

Order of operations (cutover day)

Physically install fv-ml1 at FV, on its dedicated circuit. Confirm the circuit handles ~1.5 kW peak (the whole point).
OPNsense up: LAN 10.251.50.1 / mgmt 10.251.250.1, WAN, os-tailscale subnet-router for 10.251.0.0/16, approve route in headscale.
fv-ml1 up on 10.251.50.54, BMC on 10.251.250.50. Verify BMC reachable over mesh (out-of-band check).
Bring up the vLLM seats (same /tank, same compose) — verify :8015/:8019 serve locally.
Apply the LiteLLM api_base repoint on ana-docker → up -d litellm → verify gen/summarizer/mog-sec route to FV.
DNS: internal.yaml fv site + records → dns-sync.py.
Rename sweep (cosmetic) — a scripted git mv + sed pass, one commit.
Beszel/Homepage repoint; verify dashboards.

Rollback

The physical move is not reversible same-day, but the config is: keep the old ana-ml2 blocks in git history; if FV networking isn't ready, the seats simply stay down (they already are) — no worse than the current state. Do NOT delete the ana-ml2 DNS/LiteLLM entries until FV serves; comment them.

Cross-site note

LiteLLM gateway stays on ana-docker (Anaheim); fv-ml1 serves cross-site. FV↔Anaheim is metro (both Orange County, ~10 mi) so latency is a non-issue, but it's a NEW dependency: if the FV↔Anaheim link drops, inference dark. Acceptable; revisit moving the gateway to FV if FV becomes the inference hub.

5.7 KiB Raw Permalink Blame History