vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(fv): on-site playbook for the colo build, and the redundancy it does not have

Browse Source 
fv-ml1-cutover.md covers what to change; this covers the on-site hour, where the
binding constraints are limited time and not locking ourselves out of a site an
hour away.

The load-bearing correction is that fv-ml1 running tailscale is NOT a second
independent path. Its default route is the OPNsense LAN address, so its node
needs egress through OPNsense to reach the control plane; OPNsense down means
fv-ml1 is unreachable by construction. An earlier version of this plan claimed
otherwise and proposed proving it by rebooting the firewall, which would have
failed on site with the operator standing at the rack. He caught it.

What the fv-ml1 node does buy is narrower and still worth one command: it covers
control-plane lockout -- a bad admin rule, a WebGUI on the wrong interface, a
lost password -- where the box still routes but cannot be managed. It does not
cover data-plane failure. Those are different failure classes and conflating
them produced the wrong test. The gate is now a deny-admin-from-mesh test run
while the console is in arm's reach, which exercises the bypass that actually
exists.

Deliberate deviation from the cutover runbook: flat /24 with the BMC at
10.251.50.50 rather than a mgmt VLAN at 10.251.250.50. The VLAN needs OPNsense
config plus switch tagging during the scarcest hour, for something addable
remotely in ten minutes.

Also records the DNS chicken-and-egg (the mesh needs public resolvers because
the fleet AdGuard is unreachable until the mesh is up) and the interface
auto-assignment trap measured on the lab VM the same night.
This commit is contained in:
Vuong Hoang
2026-09-12 07:54:52 -07:00
parent 2dd459d2e5
commit fde7834635
1 changed files with 155 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/runbooks/fv-onsite-playbook.md
+155
View File
@@ -0,0 +1,155 @@
# FV colo build — on-site playbook (OPNsense + fv-ml1)
**Prepared 2026-09-12.** Companion to [`fv-ml1-cutover.md`](fv-ml1-cutover.md), which
owns the address map, blast radius and LiteLLM repoint. This file owns the *on-site
hour*: minimal surface, and not locking ourselves out.
⚠ **PRINT THIS.** The one scenario it exists for is "the network isn't working",
which is exactly when a phone that needs the network is useless.
---
## THE ONE RULE
> **Do not leave the colo until you can reach both boxes from off the colo network,
> and you have tested the lockout bypass while standing next to the console.**
---
## Read this first — what is and isn't redundant
fv-ml1's default route is `10.251.50.1`, which **is** OPNsense. Its tailscale node
needs egress through OPNsense to reach the control plane. Therefore:
> **OPNsense down → fv-ml1 is also unreachable. By construction. There is no
> software fix for this.**
Running tailscale on fv-ml1 is still worth one command, but be exact about what it buys:
| | |
|---|---|
| **Covers — the common lockout** | Bad admin firewall rule, WebGUI bound to the wrong interface, lost password, broken UI — anything where the box still **routes** but you cannot **manage** it. Egress still works, fv-ml1 stays on the mesh, and infra-ops reaches OPNsense from inside at `10.251.50.1`. |
| **Does NOT cover** | OPNsense down, rebooting, panicked, or with broken routing/NAT. Nothing behind it is reachable. |
⚠ **OPNsense is a hard single point of failure for the FV site.** Plan around that
rather than pretending otherwise. (An earlier draft of this plan called fv-ml1 a
second independent path — it is not, and the operator caught it. Corrected here.)
---
## Address plan — flat on day one, deliberately
```
OPNsense LAN 10.251.50.1/24
fv-ml1 10.251.50.54
fv-ml1 BMC 10.251.50.50
WAN static public v4 from the colo
Route advertised 10.251.0.0/16
Headscale https://headscale.phasefinal.com (public, verified 200)
Expected mesh OPNsense 100.64.0.7 / fv-ml1 100.64.0.8
```
⚠ **Deliberate deviation** from `fv-ml1-cutover.md`, which puts the BMC on a mgmt
VLAN at `10.251.250.50`. A mgmt VLAN needs OPNsense VLAN config plus switch tagging —
real time and real lockout risk during the scarcest hour — for something that can be
added remotely in ten minutes. **Flat today, split later.**
---
## On-site: nine things. That is the whole job.
### OPNsense
1. **Assign interfaces — USE AUTO-DETECT** (plug the cable when prompted).
⚠ Do **not** assume port order. OPNsense assigned LAN to the wrong NIC on the
lab VM on 2026-09-11; identical physical ports make it worse.
2. **WAN:** static IP / mask / gateway from the colo.
3. **DNS: `1.1.1.1` and `9.9.9.9`.**
⚠ **NOT the fleet AdGuard (10.100.50.40).** It is unreachable until the mesh is
up, and the mesh needs DNS to fetch the plugin. That deadlock eats an hour.
4. **LAN:** `10.251.50.1/24`. Flat. No VLANs today.
5. **Install `os-tailscale`**, then join:
```
tailscale up --login-server https://headscale.phasefinal.com \
--authkey <KEY FROM VAULT> --advertise-routes=10.251.0.0/16
```
### fv-ml1
6. Power on. Confirm it boots and `/tank` imports.
7. Static `10.251.50.54`, gw `10.251.50.1`, DNS `1.1.1.1`.
8. Join as a **plain node, no route advertisement**:
```
tailscale up --login-server https://headscale.phasefinal.com --authkey <KEY FROM VAULT>
```
### BMC
9. Static `10.251.50.50` on the flat LAN.
---
## The verify gate — do not skip
From **off** the colo network (phone on LTE, and infra-ops from nh3-dev):
- [ ] OPNsense reachable at `100.64.0.7`
- [ ] fv-ml1 reachable at `100.64.0.8`
- [ ] BMC reachable at `10.251.50.50` through the approved route
- [ ] **Lockout bypass test, console in arm's reach:** add a floating rule denying
the mesh to the OPNsense WebGUI → confirm `10.251.50.1` is still reachable
**via fv-ml1** → **remove the rule.**
⚠ That last item is the only test that proves the bypass works *before* it is needed.
**Rebooting OPNsense proves nothing** — it predictably drops the whole site for
60–90 s and then everything returns.
---
## Traps, in the order they bite
- **DNS chicken-and-egg.** Public resolvers until the mesh is up.
- **Interface assignment is not what you expect.** Auto-detect.
- **Never change WAN settings and the management path in the same step.** One at a
time, verify between.
- ⭐ **If the static handoff is delayed, ASK FOR DHCP.** Tailscale needs working
internet, not the final address. This single question can save the trip.
- Nothing points at internal DNS or internal IPs until the mesh route is approved.
---
## Before leaving the house
- [ ] Preauth keys minted, **7-day reusable**, vaulted under `headscale/`
⚠ the 2026-09-06 keys expired 2026-09-08 — a dead key at the rack is a wasted trip
- [ ] This page **printed**
- [ ] MacBook (`vhlk-mba26`, headscale node 5) back online as a mesh client — an
independent client path, and lets the operator verify without infra-ops
- [ ] Colo **remote hands**: phone number, procedure, SLA
- [ ] **Physical labels on both boxes** so remote hands can act on a phone call
- [ ] infra-ops reachable while on site
---
## What infra-ops does remotely, live
- Approve the `10.251.0.0/16` route in headscale
- Push SSH keys to both boxes, confirm key-only login
- Run the reachability checks and the bypass test, report pass/fail
- Add `fv-ml1` + BMC records to `dns/internal.yaml` and sync
**Deferred to remote on purpose** (none needs hands on site): mgmt VLAN split,
firewall policy, LiteLLM `api_base` repoint (10.250.50.54 → 10.251.50.54), vLLM
seats, ana-ml2 → fv-ml1 rename sweep, Beszel, Homepage.
---
## After leaving
Every remote change to OPNsense gets the **commit-confirmed** pattern: snapshot
config → apply → scheduled auto-revert that fires in N minutes unless cancelled.
That turns "I locked us out" from a drive to Fountain Valley into a two-minute wait.
Worth pricing: a ~$150 **LTE out-of-band router with its own mesh node** is the only
thing that gives true independence from OPNsense. **Ask the colo first** — many
include an OOB/management network with the rack.