vh
3e727dbeb5
Captures the Fountain Valley cutover as executed rather than planned: fv-ml1 live on 10.251/16 and serving, BMC recovered after finding it was tagging 802.1q VLAN 250 into an untagged port, and the corrected four-GPU / 391 GB count that invalidates prior sizing assumptions. Also records two things that cost real time and would cost it again: OPNsense write APIs need an X-CSRFToken scraped from a script block, not a hidden form input, and a 403 that coincides with the box going down looks exactly like a successful reboot. Auto-archived 8 settled entries (Recent decisions: 8). The index stays over the 300-line cap at 385 because the guards hold -- nearly every remaining dated entry is under 14 days old or carries open deferred work, and an over-cap file that keeps live decisions beats a scannable one that lost any.
5.3 KiB
[2026-09-12] FV cutover EXECUTED — the box is at Fountain Valley, renamed, renumbered, serving
Operator drove to the new Fountain Valley colo and racked the OPNsense gateway + the ex-ana-ml2 GPU box. Everything below is verified by observation, not by config reading.
Final state
fv.phasefinal.com 172.83.89.66 public A record, DNS-only (Cloudflare)
vb-gateway 10.251.50.1 OPNsense 26.7.3_11, mesh node 100.64.0.8
fv-ml1 10.251.50.54 mesh node 100.64.0.7, DHCP reservation
fv-ml1-bmc 10.251.250.50 own interface bge1, outside the bridge
seats vllm-gen/:8015, mog-sec/:8019, erp-seat, gemma4-charrp, scriberr
tank 11.6T ONLINE with 6.14T allocated — the data travelled with the chassis as planned.
Inference verified end-to-end: summarizer and sec both answer through the Anaheim
LiteLLM gateway, across the mesh, to FV seats on different ports.
⭐⭐ FOUR GPUs, not two — every record was wrong
nvidia-smi -L counts 4× RTX PRO 6000 Blackwell Max-Q @ 97,887 MiB (C3/C4/E3/E4),
independently confirmed by PCI enumeration of four GB202GL devices (the fifth VGA device
is the ASPEED BMC framebuffer). That is 391 GB VRAM, not 196. CLAUDE.md, README and
the cutover runbook all said "dual". Corrected. ⚠ This materially changes model sizing and
seat placement — worth revisiting whether seats split across irv-ml1/gx10 can consolidate.
⚠ Also means the FV circuit was likely specced against half the real draw; operator says
it is on its own circuit and fine for now.
⭐⭐ The BMC was tagging 802.1q VLAN 250 — invisible from the network side
The BMC never ARPed on any port through an hour of cable-swapping. Its addressing was
correct the whole time (static 10.251.250.50/24, gw .1, MAC 7c:c2:55:60:fe:8a).
ipmitool lan print 1 from inside the chassis showed 802.1q VLAN ID : 250 — it was
tagging every frame into an untagged access port, so the switch silently discarded them.
No cable position could ever have fixed this, and no network-side diagnostic can see
it — ARP, DHCP leases, port counters and firewall logs all just show silence. Only an
in-chassis ipmitool lan print reveals it. Fixed with ipmitool lan set 1 vlan id off,
which resets the IP to 0.0.0.0 (normal Supermicro behaviour), then re-set the static.
Now: 0.37 ms, ports 443/80/623/5900 open, ATEN web UI and Redfish both HTTP 200.
Routing: what actually had to change (not what we assumed)
The scale nodes needed nothing — all three already had RouteAll: true and learned
10.251.0.0/16 automatically the moment headscale approved the route. What was missing:
--accept-routeson vb-gateway. It had no return path to the fleet, so the first UDM static route appeared to do nothing — asymmetric drop, not a missing forward route.- Three gateway statics,
10.251.0.0/16→ local scale node: NH3 UDM, ESH UDM, and the Anaheim FortiGate (config backed up to flash aspre-fv-route-20260912first).
All three sites now reach FV by real IP; Anaheim's pre-existing routing verified intact afterwards.
⚠ Known gap — fv-ml1 cannot initiate to fleet LAN IPs
fv-ml1 reaches mesh IPs (100.64.x) and the internet, and the fleet reaches fv-ml1 fine.
But fv-ml1 → 10.100.x / 10.250.x / 10.0.x all FAIL. Packets are forwarded correctly
(pass on tailscale0 10.251.50.54 -> 10.100.50.40), so the gap is on the return side at
the far gateways. Not currently biting: DNS goes via tailscale MagicDNS
(100.100.100.100) and inference is inbound, so nothing depends on it yet. Will bite
anything that needs fv-ml1 to pull from a fleet LAN host (NAS, registries). UNRESOLVED.
⚠ Removed a stale if-up hook
/etc/network/if-up.d/mesh-routes on the box still injected Anaheim mesh return routes
via ana-scale 10.250.50.45 on enp97s0f0np0.50. Inert at FV only because that NIC is
down — a landmine if it ever comes up. Removed; backup at /root/mesh-routes.retired-20260912.
Credentials, all vaulted and read-back verified
fv-gateway/opnsense-api-key, fv-gateway/opnsense-api-secret,
fv-gateway/infra-ops-password, fv-gateway/root-password, plus
headscale/preauth-fv-{router,client}-7d-20260912 (expire 2026-09-19).
⚠ Plaintext copies still on disk: /tmp/opn.pw (nh3-dev), /tmp/io.pw + /tmp/key.io
(fv-ml1) — redundant now, should be deleted.
⚠⚠ OPNsense API: session cookies are NOT enough, and this cost hours
Every write API (reboot, service restart, tunables) returns 403 with only a session
cookie. The CSRF token lives in a <script> block as
xhr.setRequestHeader("X-CSRFToken", "…") — not a hidden form input, so scraping for
<input type=hidden> finds nothing. With the header supplied, writes return 200.
Config restores via the legacy diag_backup.php form always worked (it carries a token).
⚠⚠ Because the reboot API 403'd silently, I repeatedly reported reboots as mine when the operator was power-cycling the box by hand. A 403 that coincides with the box going down looks exactly like success. Verify a reboot by observing the box go down AND the change take effect — never by the API's response.
Runbook: docs/runbooks/fv-ml1-cutover.md, docs/runbooks/fv-onsite-playbook.md.
Commits 91bda3c (rename/renumber/DNS/LiteLLM), 17586ab (proxy retired),
87d7555 + 48e34e1 (playbook bodies + remaining live targets).