esh-pfi-infrastructure/docs/runbooks/fv-onsite-playbook.md

FV colo build — on-site playbook (OPNsense + fv-ml1)

Prepared 2026-09-12. Companion to fv-ml1-cutover.md, which owns the address map, blast radius and LiteLLM repoint. This file owns the on-site hour: minimal surface, and not locking ourselves out.

⚠ PRINT THIS. The one scenario it exists for is "the network isn't working", which is exactly when a phone that needs the network is useless.

---

THE ONE RULE

Do not leave the colo until you can reach both boxes from off the colo network, and you have tested the lockout bypass while standing next to the console.

---

Read this first — what is and isn't redundant

fv-ml1's default route is 10.251.50.1, which is OPNsense. Its tailscale node needs egress through OPNsense to reach the control plane. Therefore:

OPNsense down → fv-ml1 is also unreachable. By construction. There is no software fix for this.

Running tailscale on fv-ml1 is still worth one command, but be exact about what it buys:

Covers — the common lockout	Bad admin firewall rule, WebGUI bound to the wrong interface, lost password, broken UI — anything where the box still routes but you cannot manage it. Egress still works, fv-ml1 stays on the mesh, and infra-ops reaches OPNsense from inside at 10.251.50.1.
Does NOT cover	OPNsense down, rebooting, panicked, or with broken routing/NAT. Nothing behind it is reachable.

⚠ OPNsense is a hard single point of failure for the FV site. Plan around that rather than pretending otherwise. (An earlier draft of this plan called fv-ml1 a second independent path — it is not, and the operator caught it. Corrected here.)

---

Address plan — flat on day one, deliberately

OPNsense LAN      10.251.50.1/24
fv-ml1            10.251.50.54
fv-ml1 BMC        10.251.50.50
WAN               static public v4 from the colo
Route advertised  10.251.0.0/16
Headscale         https://headscale.phasefinal.com     (public, verified 200)
Expected mesh     OPNsense 100.64.0.7 / fv-ml1 100.64.0.8

⚠ Deliberate deviation from fv-ml1-cutover.md, which puts the BMC on a mgmt VLAN at 10.251.250.50. A mgmt VLAN needs OPNsense VLAN config plus switch tagging — real time and real lockout risk during the scarcest hour — for something that can be added remotely in ten minutes. Flat today, split later.

---

On-site: nine things. That is the whole job.

OPNsense

1. Assign interfaces — USE AUTO-DETECT (plug the cable when prompted). ⚠ Do not assume port order. OPNsense assigned LAN to the wrong NIC on the lab VM on 2026-09-11; identical physical ports make it worse.
2. WAN: static IP / mask / gateway from the colo.
3. DNS: 1.1.1.1 and 9.9.9.9. ⚠ NOT the fleet AdGuard (10.100.50.40). It is unreachable until the mesh is up, and the mesh needs DNS to fetch the plugin. That deadlock eats an hour.
4. LAN: 10.251.50.1/24. Flat. No VLANs today.
5. Install os-tailscale, then join:

   tailscale up --login-server https://headscale.phasefinal.com \
     --authkey <KEY FROM VAULT> --advertise-routes=10.251.0.0/16
   

fv-ml1

6. Power on. Confirm it boots and /tank imports.
7. Static 10.251.50.54, gw 10.251.50.1, DNS 1.1.1.1.
8. Join as a plain node, no route advertisement:

   tailscale up --login-server https://headscale.phasefinal.com --authkey <KEY FROM VAULT>
   

BMC

9. Static 10.251.50.50 on the flat LAN.

---

The verify gate — do not skip

From off the colo network (phone on LTE, and infra-ops from nh3-dev):

• OPNsense reachable at 100.64.0.7
• fv-ml1 reachable at 100.64.0.8
• BMC reachable at 10.251.50.50 through the approved route
• Lockout bypass test, console in arm's reach: add a floating rule denying the mesh to the OPNsense WebGUI → confirm 10.251.50.1 is still reachable via fv-ml1 → remove the rule.

⚠ That last item is the only test that proves the bypass works before it is needed. Rebooting OPNsense proves nothing — it predictably drops the whole site for 60–90 s and then everything returns.

---

Traps, in the order they bite

• DNS chicken-and-egg. Public resolvers until the mesh is up.
• Interface assignment is not what you expect. Auto-detect.
• Never change WAN settings and the management path in the same step. One at a time, verify between.
• ⭐ If the static handoff is delayed, ASK FOR DHCP. Tailscale needs working internet, not the final address. This single question can save the trip.
• Nothing points at internal DNS or internal IPs until the mesh route is approved.

---

Before leaving the house

• Preauth keys minted, 7-day reusable, vaulted under headscale/ ⚠ the 2026-09-06 keys expired 2026-09-08 — a dead key at the rack is a wasted trip
• This page printed
• MacBook (vhlk-mba26, headscale node 5) back online as a mesh client — an independent client path, and lets the operator verify without infra-ops
• Colo remote hands: phone number, procedure, SLA
• Physical labels on both boxes so remote hands can act on a phone call
• infra-ops reachable while on site

---

What infra-ops does remotely, live

• Approve the 10.251.0.0/16 route in headscale
• Push SSH keys to both boxes, confirm key-only login
• Run the reachability checks and the bypass test, report pass/fail
• Add fv-ml1 + BMC records to dns/internal.yaml and sync

Deferred to remote on purpose (none needs hands on site): mgmt VLAN split, firewall policy, LiteLLM api_base repoint (10.250.50.54 → 10.251.50.54), vLLM seats, ana-ml2 → fv-ml1 rename sweep, Beszel, Homepage.

---

After leaving

Every remote change to OPNsense gets the commit-confirmed pattern: snapshot config → apply → scheduled auto-revert that fires in N minutes unless cancelled. That turns "I locked us out" from a drive to Fountain Valley into a two-minute wait.

Worth pricing: a ~$150 LTE out-of-band router with its own mesh node is the only thing that gives true independence from OPNsense. Ask the colo first — many include an OOB/management network with the rack.