# FV colo build — on-site playbook (OPNsense + fv-ml1) **Prepared 2026-09-12.** Companion to [`fv-ml1-cutover.md`](fv-ml1-cutover.md), which owns the address map, blast radius and LiteLLM repoint. This file owns the *on-site hour*: minimal surface, and not locking ourselves out. ⚠ **PRINT THIS.** The one scenario it exists for is "the network isn't working", which is exactly when a phone that needs the network is useless. --- ## THE ONE RULE > **Do not leave the colo until you can reach both boxes from off the colo network, > and you have tested the lockout bypass while standing next to the console.** --- ## Read this first — what is and isn't redundant fv-ml1's default route is `10.251.50.1`, which **is** OPNsense. Its tailscale node needs egress through OPNsense to reach the control plane. Therefore: > **OPNsense down → fv-ml1 is also unreachable. By construction. There is no > software fix for this.** Running tailscale on fv-ml1 is still worth one command, but be exact about what it buys: | | | |---|---| | **Covers — the common lockout** | Bad admin firewall rule, WebGUI bound to the wrong interface, lost password, broken UI — anything where the box still **routes** but you cannot **manage** it. Egress still works, fv-ml1 stays on the mesh, and infra-ops reaches OPNsense from inside at `10.251.50.1`. | | **Does NOT cover** | OPNsense down, rebooting, panicked, or with broken routing/NAT. Nothing behind it is reachable. | ⚠ **OPNsense is a hard single point of failure for the FV site.** Plan around that rather than pretending otherwise. (An earlier draft of this plan called fv-ml1 a second independent path — it is not, and the operator caught it. Corrected here.) --- ## Address plan — flat on day one, deliberately ``` OPNsense LAN 10.251.50.1/24 fv-ml1 10.251.50.54 fv-ml1 BMC 10.251.50.50 WAN static public v4 from the colo Route advertised 10.251.0.0/16 Headscale https://headscale.phasefinal.com (public, verified 200) Expected mesh OPNsense 100.64.0.7 / fv-ml1 100.64.0.8 ``` ⚠ **Deliberate deviation** from `fv-ml1-cutover.md`, which puts the BMC on a mgmt VLAN at `10.251.250.50`. A mgmt VLAN needs OPNsense VLAN config plus switch tagging — real time and real lockout risk during the scarcest hour — for something that can be added remotely in ten minutes. **Flat today, split later.** --- ## On-site: nine things. That is the whole job. ### OPNsense 1. **Assign interfaces — USE AUTO-DETECT** (plug the cable when prompted). ⚠ Do **not** assume port order. OPNsense assigned LAN to the wrong NIC on the lab VM on 2026-09-11; identical physical ports make it worse. 2. **WAN:** static IP / mask / gateway from the colo. 3. **DNS: `1.1.1.1` and `9.9.9.9`.** ⚠ **NOT the fleet AdGuard (10.100.50.40).** It is unreachable until the mesh is up, and the mesh needs DNS to fetch the plugin. That deadlock eats an hour. 4. **LAN:** `10.251.50.1/24`. Flat. No VLANs today. 5. **Install `os-tailscale`**, then join: ``` tailscale up --login-server https://headscale.phasefinal.com \ --authkey --advertise-routes=10.251.0.0/16 ``` ### fv-ml1 6. Power on. Confirm it boots and `/tank` imports. 7. Static `10.251.50.54`, gw `10.251.50.1`, DNS `1.1.1.1`. 8. Join as a **plain node, no route advertisement**: ``` tailscale up --login-server https://headscale.phasefinal.com --authkey ``` ### BMC 9. Static `10.251.50.50` on the flat LAN. --- ## The verify gate — do not skip From **off** the colo network (phone on LTE, and infra-ops from nh3-dev): - [ ] OPNsense reachable at `100.64.0.7` - [ ] fv-ml1 reachable at `100.64.0.8` - [ ] BMC reachable at `10.251.50.50` through the approved route - [ ] **Lockout bypass test, console in arm's reach:** add a floating rule denying the mesh to the OPNsense WebGUI → confirm `10.251.50.1` is still reachable **via fv-ml1** → **remove the rule.** ⚠ That last item is the only test that proves the bypass works *before* it is needed. **Rebooting OPNsense proves nothing** — it predictably drops the whole site for 60–90 s and then everything returns. --- ## Traps, in the order they bite - **DNS chicken-and-egg.** Public resolvers until the mesh is up. - **Interface assignment is not what you expect.** Auto-detect. - **Never change WAN settings and the management path in the same step.** One at a time, verify between. - ⭐ **If the static handoff is delayed, ASK FOR DHCP.** Tailscale needs working internet, not the final address. This single question can save the trip. - Nothing points at internal DNS or internal IPs until the mesh route is approved. --- ## Before leaving the house - [ ] Preauth keys minted, **7-day reusable**, vaulted under `headscale/` ⚠ the 2026-09-06 keys expired 2026-09-08 — a dead key at the rack is a wasted trip - [ ] This page **printed** - [ ] MacBook (`vhlk-mba26`, headscale node 5) back online as a mesh client — an independent client path, and lets the operator verify without infra-ops - [ ] Colo **remote hands**: phone number, procedure, SLA - [ ] **Physical labels on both boxes** so remote hands can act on a phone call - [ ] infra-ops reachable while on site --- ## What infra-ops does remotely, live - Approve the `10.251.0.0/16` route in headscale - Push SSH keys to both boxes, confirm key-only login - Run the reachability checks and the bypass test, report pass/fail - Add `fv-ml1` + BMC records to `dns/internal.yaml` and sync **Deferred to remote on purpose** (none needs hands on site): mgmt VLAN split, firewall policy, LiteLLM `api_base` repoint (10.250.50.54 → 10.251.50.54), vLLM seats, ana-ml2 → fv-ml1 rename sweep, Beszel, Homepage. --- ## After leaving Every remote change to OPNsense gets the **commit-confirmed** pattern: snapshot config → apply → scheduled auto-revert that fires in N minutes unless cancelled. That turns "I locked us out" from a drive to Fountain Valley into a two-minute wait. Worth pricing: a ~$150 **LTE out-of-band router with its own mesh node** is the only thing that gives true independence from OPNsense. **Ask the colo first** — many include an OOB/management network with the rack.