diff --git a/docs/runbooks/fv-onsite-playbook.md b/docs/runbooks/fv-onsite-playbook.md new file mode 100644 index 0000000..fc56631 --- /dev/null +++ b/docs/runbooks/fv-onsite-playbook.md @@ -0,0 +1,155 @@ +# FV colo build — on-site playbook (OPNsense + fv-ml1) + +**Prepared 2026-09-12.** Companion to [`fv-ml1-cutover.md`](fv-ml1-cutover.md), which +owns the address map, blast radius and LiteLLM repoint. This file owns the *on-site +hour*: minimal surface, and not locking ourselves out. + +⚠ **PRINT THIS.** The one scenario it exists for is "the network isn't working", +which is exactly when a phone that needs the network is useless. + +--- + +## THE ONE RULE + +> **Do not leave the colo until you can reach both boxes from off the colo network, +> and you have tested the lockout bypass while standing next to the console.** + +--- + +## Read this first — what is and isn't redundant + +fv-ml1's default route is `10.251.50.1`, which **is** OPNsense. Its tailscale node +needs egress through OPNsense to reach the control plane. Therefore: + +> **OPNsense down → fv-ml1 is also unreachable. By construction. There is no +> software fix for this.** + +Running tailscale on fv-ml1 is still worth one command, but be exact about what it buys: + +| | | +|---|---| +| **Covers — the common lockout** | Bad admin firewall rule, WebGUI bound to the wrong interface, lost password, broken UI — anything where the box still **routes** but you cannot **manage** it. Egress still works, fv-ml1 stays on the mesh, and infra-ops reaches OPNsense from inside at `10.251.50.1`. | +| **Does NOT cover** | OPNsense down, rebooting, panicked, or with broken routing/NAT. Nothing behind it is reachable. | + +⚠ **OPNsense is a hard single point of failure for the FV site.** Plan around that +rather than pretending otherwise. (An earlier draft of this plan called fv-ml1 a +second independent path — it is not, and the operator caught it. Corrected here.) + +--- + +## Address plan — flat on day one, deliberately + +``` +OPNsense LAN 10.251.50.1/24 +fv-ml1 10.251.50.54 +fv-ml1 BMC 10.251.50.50 +WAN static public v4 from the colo +Route advertised 10.251.0.0/16 +Headscale https://headscale.phasefinal.com (public, verified 200) +Expected mesh OPNsense 100.64.0.7 / fv-ml1 100.64.0.8 +``` + +⚠ **Deliberate deviation** from `fv-ml1-cutover.md`, which puts the BMC on a mgmt +VLAN at `10.251.250.50`. A mgmt VLAN needs OPNsense VLAN config plus switch tagging — +real time and real lockout risk during the scarcest hour — for something that can be +added remotely in ten minutes. **Flat today, split later.** + +--- + +## On-site: nine things. That is the whole job. + +### OPNsense + +1. **Assign interfaces — USE AUTO-DETECT** (plug the cable when prompted). + ⚠ Do **not** assume port order. OPNsense assigned LAN to the wrong NIC on the + lab VM on 2026-09-11; identical physical ports make it worse. +2. **WAN:** static IP / mask / gateway from the colo. +3. **DNS: `1.1.1.1` and `9.9.9.9`.** + ⚠ **NOT the fleet AdGuard (10.100.50.40).** It is unreachable until the mesh is + up, and the mesh needs DNS to fetch the plugin. That deadlock eats an hour. +4. **LAN:** `10.251.50.1/24`. Flat. No VLANs today. +5. **Install `os-tailscale`**, then join: + ``` + tailscale up --login-server https://headscale.phasefinal.com \ + --authkey --advertise-routes=10.251.0.0/16 + ``` + +### fv-ml1 + +6. Power on. Confirm it boots and `/tank` imports. +7. Static `10.251.50.54`, gw `10.251.50.1`, DNS `1.1.1.1`. +8. Join as a **plain node, no route advertisement**: + ``` + tailscale up --login-server https://headscale.phasefinal.com --authkey + ``` + +### BMC + +9. Static `10.251.50.50` on the flat LAN. + +--- + +## The verify gate — do not skip + +From **off** the colo network (phone on LTE, and infra-ops from nh3-dev): + +- [ ] OPNsense reachable at `100.64.0.7` +- [ ] fv-ml1 reachable at `100.64.0.8` +- [ ] BMC reachable at `10.251.50.50` through the approved route +- [ ] **Lockout bypass test, console in arm's reach:** add a floating rule denying + the mesh to the OPNsense WebGUI → confirm `10.251.50.1` is still reachable + **via fv-ml1** → **remove the rule.** + +⚠ That last item is the only test that proves the bypass works *before* it is needed. +**Rebooting OPNsense proves nothing** — it predictably drops the whole site for +60–90 s and then everything returns. + +--- + +## Traps, in the order they bite + +- **DNS chicken-and-egg.** Public resolvers until the mesh is up. +- **Interface assignment is not what you expect.** Auto-detect. +- **Never change WAN settings and the management path in the same step.** One at a + time, verify between. +- ⭐ **If the static handoff is delayed, ASK FOR DHCP.** Tailscale needs working + internet, not the final address. This single question can save the trip. +- Nothing points at internal DNS or internal IPs until the mesh route is approved. + +--- + +## Before leaving the house + +- [ ] Preauth keys minted, **7-day reusable**, vaulted under `headscale/` + ⚠ the 2026-09-06 keys expired 2026-09-08 — a dead key at the rack is a wasted trip +- [ ] This page **printed** +- [ ] MacBook (`vhlk-mba26`, headscale node 5) back online as a mesh client — an + independent client path, and lets the operator verify without infra-ops +- [ ] Colo **remote hands**: phone number, procedure, SLA +- [ ] **Physical labels on both boxes** so remote hands can act on a phone call +- [ ] infra-ops reachable while on site + +--- + +## What infra-ops does remotely, live + +- Approve the `10.251.0.0/16` route in headscale +- Push SSH keys to both boxes, confirm key-only login +- Run the reachability checks and the bypass test, report pass/fail +- Add `fv-ml1` + BMC records to `dns/internal.yaml` and sync + +**Deferred to remote on purpose** (none needs hands on site): mgmt VLAN split, +firewall policy, LiteLLM `api_base` repoint (10.250.50.54 → 10.251.50.54), vLLM +seats, ana-ml2 → fv-ml1 rename sweep, Beszel, Homepage. + +--- + +## After leaving + +Every remote change to OPNsense gets the **commit-confirmed** pattern: snapshot +config → apply → scheduled auto-revert that fires in N minutes unless cancelled. +That turns "I locked us out" from a drive to Fountain Valley into a two-minute wait. + +Worth pricing: a ~$150 **LTE out-of-band router with its own mesh node** is the only +thing that gives true independence from OPNsense. **Ask the colo first** — many +include an OOB/management network with the rack.