vh
3e727dbeb5
Captures the Fountain Valley cutover as executed rather than planned: fv-ml1 live on 10.251/16 and serving, BMC recovered after finding it was tagging 802.1q VLAN 250 into an untagged port, and the corrected four-GPU / 391 GB count that invalidates prior sizing assumptions. Also records two things that cost real time and would cost it again: OPNsense write APIs need an X-CSRFToken scraped from a script block, not a hidden form input, and a 403 that coincides with the box going down looks exactly like a successful reboot. Auto-archived 8 settled entries (Recent decisions: 8). The index stays over the 300-line cap at 385 because the guards hold -- nearly every remaining dated entry is under 14 days old or carries open deferred work, and an over-cap file that keeps live decisions beats a scannable one that lost any.
93 lines
5.3 KiB
Markdown
93 lines
5.3 KiB
Markdown
# `[2026-09-12]` FV cutover EXECUTED — the box is at Fountain Valley, renamed, renumbered, serving
|
||
|
||
Operator drove to the new Fountain Valley colo and racked the OPNsense gateway + the
|
||
ex-ana-ml2 GPU box. Everything below is verified by observation, not by config reading.
|
||
|
||
## Final state
|
||
|
||
fv.phasefinal.com 172.83.89.66 public A record, DNS-only (Cloudflare)
|
||
vb-gateway 10.251.50.1 OPNsense 26.7.3_11, mesh node 100.64.0.8
|
||
fv-ml1 10.251.50.54 mesh node 100.64.0.7, DHCP reservation
|
||
fv-ml1-bmc 10.251.250.50 own interface bge1, outside the bridge
|
||
seats vllm-gen/:8015, mog-sec/:8019, erp-seat, gemma4-charrp, scriberr
|
||
|
||
`tank` 11.6T ONLINE with 6.14T allocated — the data travelled with the chassis as planned.
|
||
Inference verified end-to-end: `summarizer` and `sec` both answer through the Anaheim
|
||
LiteLLM gateway, across the mesh, to FV seats on different ports.
|
||
|
||
## ⭐⭐ FOUR GPUs, not two — every record was wrong
|
||
|
||
`nvidia-smi -L` counts **4× RTX PRO 6000 Blackwell Max-Q @ 97,887 MiB** (C3/C4/E3/E4),
|
||
independently confirmed by PCI enumeration of four GB202GL devices (the fifth VGA device
|
||
is the ASPEED BMC framebuffer). That is **391 GB VRAM, not 196**. CLAUDE.md, README and
|
||
the cutover runbook all said "dual". Corrected. ⚠ This materially changes model sizing and
|
||
seat placement — worth revisiting whether seats split across irv-ml1/gx10 can consolidate.
|
||
⚠ Also means the FV circuit was likely specced against half the real draw; operator says
|
||
it is on its own circuit and fine for now.
|
||
|
||
## ⭐⭐ The BMC was tagging 802.1q VLAN 250 — invisible from the network side
|
||
|
||
The BMC never ARPed on any port through an hour of cable-swapping. Its addressing was
|
||
**correct the whole time** (static 10.251.250.50/24, gw .1, MAC 7c:c2:55:60:fe:8a).
|
||
`ipmitool lan print 1` from inside the chassis showed `802.1q VLAN ID : 250` — it was
|
||
tagging every frame into an untagged access port, so the switch silently discarded them.
|
||
|
||
**No cable position could ever have fixed this**, and no network-side diagnostic can see
|
||
it — ARP, DHCP leases, port counters and firewall logs all just show silence. Only an
|
||
in-chassis `ipmitool lan print` reveals it. Fixed with `ipmitool lan set 1 vlan id off`,
|
||
which resets the IP to 0.0.0.0 (normal Supermicro behaviour), then re-set the static.
|
||
Now: 0.37 ms, ports 443/80/623/5900 open, ATEN web UI and Redfish both HTTP 200.
|
||
|
||
## Routing: what actually had to change (not what we assumed)
|
||
|
||
The scale nodes needed **nothing** — all three already had `RouteAll: true` and learned
|
||
`10.251.0.0/16` automatically the moment headscale approved the route. What was missing:
|
||
|
||
1. **`--accept-routes` on vb-gateway.** It had no return path to the fleet, so the first
|
||
UDM static route appeared to do nothing — asymmetric drop, not a missing forward route.
|
||
2. **Three gateway statics**, `10.251.0.0/16` → local scale node: NH3 UDM, ESH UDM, and
|
||
the Anaheim FortiGate (config backed up to flash as `pre-fv-route-20260912` first).
|
||
|
||
All three sites now reach FV by real IP; Anaheim's pre-existing routing verified intact
|
||
afterwards.
|
||
|
||
## ⚠ Known gap — fv-ml1 cannot initiate to fleet LAN IPs
|
||
|
||
fv-ml1 reaches mesh IPs (100.64.x) and the internet, and the fleet reaches fv-ml1 fine.
|
||
But fv-ml1 → 10.100.x / 10.250.x / 10.0.x all FAIL. Packets are forwarded correctly
|
||
(`pass on tailscale0 10.251.50.54 -> 10.100.50.40`), so the gap is on the return side at
|
||
the far gateways. **Not currently biting**: DNS goes via tailscale MagicDNS
|
||
(100.100.100.100) and inference is inbound, so nothing depends on it yet. Will bite
|
||
anything that needs fv-ml1 to pull from a fleet LAN host (NAS, registries). UNRESOLVED.
|
||
|
||
## ⚠ Removed a stale if-up hook
|
||
|
||
`/etc/network/if-up.d/mesh-routes` on the box still injected Anaheim mesh return routes
|
||
via ana-scale 10.250.50.45 on `enp97s0f0np0.50`. Inert at FV only because that NIC is
|
||
down — a landmine if it ever comes up. Removed; backup at `/root/mesh-routes.retired-20260912`.
|
||
|
||
## Credentials, all vaulted and read-back verified
|
||
|
||
`fv-gateway/opnsense-api-key`, `fv-gateway/opnsense-api-secret`,
|
||
`fv-gateway/infra-ops-password`, `fv-gateway/root-password`, plus
|
||
`headscale/preauth-fv-{router,client}-7d-20260912` (expire 2026-09-19).
|
||
⚠ Plaintext copies still on disk: `/tmp/opn.pw` (nh3-dev), `/tmp/io.pw` + `/tmp/key.io`
|
||
(fv-ml1) — redundant now, should be deleted.
|
||
|
||
## ⚠⚠ OPNsense API: session cookies are NOT enough, and this cost hours
|
||
|
||
Every write API (reboot, service restart, tunables) returns **403** with only a session
|
||
cookie. The CSRF token lives in a `<script>` block as
|
||
`xhr.setRequestHeader("X-CSRFToken", "…")` — **not** a hidden form input, so scraping for
|
||
`<input type=hidden>` finds nothing. With the header supplied, writes return 200.
|
||
Config *restores* via the legacy `diag_backup.php` form always worked (it carries a token).
|
||
|
||
⚠⚠ **Because the reboot API 403'd silently, I repeatedly reported reboots as mine when
|
||
the operator was power-cycling the box by hand.** A 403 that coincides with the box going
|
||
down looks exactly like success. Verify a reboot by observing the box go down AND the
|
||
change take effect — never by the API's response.
|
||
|
||
Runbook: `docs/runbooks/fv-ml1-cutover.md`, `docs/runbooks/fv-onsite-playbook.md`.
|
||
Commits `91bda3c` (rename/renumber/DNS/LiteLLM), `17586ab` (proxy retired),
|
||
`87d7555` + `48e34e1` (playbook bodies + remaining live targets).
|