# `[2026-09-12]` FV cutover EXECUTED — the box is at Fountain Valley, renamed, renumbered, serving Operator drove to the new Fountain Valley colo and racked the OPNsense gateway + the ex-ana-ml2 GPU box. Everything below is verified by observation, not by config reading. ## Final state fv.phasefinal.com 172.83.89.66 public A record, DNS-only (Cloudflare) vb-gateway 10.251.50.1 OPNsense 26.7.3_11, mesh node 100.64.0.8 fv-ml1 10.251.50.54 mesh node 100.64.0.7, DHCP reservation fv-ml1-bmc 10.251.250.50 own interface bge1, outside the bridge seats vllm-gen/:8015, mog-sec/:8019, erp-seat, gemma4-charrp, scriberr `tank` 11.6T ONLINE with 6.14T allocated — the data travelled with the chassis as planned. Inference verified end-to-end: `summarizer` and `sec` both answer through the Anaheim LiteLLM gateway, across the mesh, to FV seats on different ports. ## ⭐⭐ FOUR GPUs, not two — every record was wrong `nvidia-smi -L` counts **4× RTX PRO 6000 Blackwell Max-Q @ 97,887 MiB** (C3/C4/E3/E4), independently confirmed by PCI enumeration of four GB202GL devices (the fifth VGA device is the ASPEED BMC framebuffer). That is **391 GB VRAM, not 196**. CLAUDE.md, README and the cutover runbook all said "dual". Corrected. ⚠ This materially changes model sizing and seat placement — worth revisiting whether seats split across irv-ml1/gx10 can consolidate. ⚠ Also means the FV circuit was likely specced against half the real draw; operator says it is on its own circuit and fine for now. ## ⭐⭐ The BMC was tagging 802.1q VLAN 250 — invisible from the network side The BMC never ARPed on any port through an hour of cable-swapping. Its addressing was **correct the whole time** (static 10.251.250.50/24, gw .1, MAC 7c:c2:55:60:fe:8a). `ipmitool lan print 1` from inside the chassis showed `802.1q VLAN ID : 250` — it was tagging every frame into an untagged access port, so the switch silently discarded them. **No cable position could ever have fixed this**, and no network-side diagnostic can see it — ARP, DHCP leases, port counters and firewall logs all just show silence. Only an in-chassis `ipmitool lan print` reveals it. Fixed with `ipmitool lan set 1 vlan id off`, which resets the IP to 0.0.0.0 (normal Supermicro behaviour), then re-set the static. Now: 0.37 ms, ports 443/80/623/5900 open, ATEN web UI and Redfish both HTTP 200. ## Routing: what actually had to change (not what we assumed) The scale nodes needed **nothing** — all three already had `RouteAll: true` and learned `10.251.0.0/16` automatically the moment headscale approved the route. What was missing: 1. **`--accept-routes` on vb-gateway.** It had no return path to the fleet, so the first UDM static route appeared to do nothing — asymmetric drop, not a missing forward route. 2. **Three gateway statics**, `10.251.0.0/16` → local scale node: NH3 UDM, ESH UDM, and the Anaheim FortiGate (config backed up to flash as `pre-fv-route-20260912` first). All three sites now reach FV by real IP; Anaheim's pre-existing routing verified intact afterwards. ## ⚠ Known gap — fv-ml1 cannot initiate to fleet LAN IPs fv-ml1 reaches mesh IPs (100.64.x) and the internet, and the fleet reaches fv-ml1 fine. But fv-ml1 → 10.100.x / 10.250.x / 10.0.x all FAIL. Packets are forwarded correctly (`pass on tailscale0 10.251.50.54 -> 10.100.50.40`), so the gap is on the return side at the far gateways. **Not currently biting**: DNS goes via tailscale MagicDNS (100.100.100.100) and inference is inbound, so nothing depends on it yet. Will bite anything that needs fv-ml1 to pull from a fleet LAN host (NAS, registries). UNRESOLVED. ## ⚠ Removed a stale if-up hook `/etc/network/if-up.d/mesh-routes` on the box still injected Anaheim mesh return routes via ana-scale 10.250.50.45 on `enp97s0f0np0.50`. Inert at FV only because that NIC is down — a landmine if it ever comes up. Removed; backup at `/root/mesh-routes.retired-20260912`. ## Credentials, all vaulted and read-back verified `fv-gateway/opnsense-api-key`, `fv-gateway/opnsense-api-secret`, `fv-gateway/infra-ops-password`, `fv-gateway/root-password`, plus `headscale/preauth-fv-{router,client}-7d-20260912` (expire 2026-09-19). ⚠ Plaintext copies still on disk: `/tmp/opn.pw` (nh3-dev), `/tmp/io.pw` + `/tmp/key.io` (fv-ml1) — redundant now, should be deleted. ## ⚠⚠ OPNsense API: session cookies are NOT enough, and this cost hours Every write API (reboot, service restart, tunables) returns **403** with only a session cookie. The CSRF token lives in a `