feat(fv): invert the watchdog to break-glass; fv-ml1 off the mesh
Operator's design, and a better one. Once the FV SNAT rules landed, fv-ml1's mesh membership was redundant for routing and its only remaining value was as a second way in. Keeping it enrolled bought a standing second door; joining on demand buys the same recovery path without one. normal tailscaled stopped + disabled; fleet reached via the gateway SNAT fault nh3-dev / nh3-docker unreachable while the WAN is up action start tailscaled + tailscale up -> reachable at its 100.64.x address fv-ml1 is now off the mesh and its node record deleted. Verified it still reaches NH3, ESH, Anaheim, Irvine and the internet on the SNAT path alone, then the break-glass fired on cue (counted 1..4, joined at 5 as 100.64.0.10), answered ping and ssh from nh3-dev, and was closed again cleanly. No auto-leave, deliberately: once open the door stays open until a human runs systemctl disable --now tailscaled. A watchdog that re-closes on recovery flaps, and a flapping recovery path is down exactly when someone finally looks. It also skips entirely when already on the mesh, which is what makes it idempotent after firing. The question exposed a hole worth more than the redesign. The stored rejoin key was one of the 2026-09-12 FV cutover keys, expiring 2026-09-19 -- a break-glass credential that dies in four days and fails silently at the only moment it matters. Replaced with a dedicated 1-year reusable key (headscale ID 8, expires 2027-09-15), vaulted as fv-ml1/headscale-breakglass-key, root:600 on the host. That also closes the standing self-join risk rather than trading it: the two stale reusable keys (IDs 5, 6) are expired, so the mesh now has exactly one live reusable key -- purpose-built, on a host we control -- instead of two orphans nobody owned. Rejoin uses --accept-routes=false and the reason is in the script: on 2026-09-14 tailscale up --accept-routes on this box accepted its OWN subnet from the gateway and black-holed it. That happened with a human watching; here it runs unattended, during an incident, on a box already in trouble.
This commit is contained in:
+10
-10
@@ -127,9 +127,9 @@ on the FV gateway, all `src=10.251.50.0/24`, dst `10.100.0.0/16` / `10.0.0.0/16`
|
||||
Anaheim only. Commits `fa04f45`, `0ab9da5`.
|
||||
|
||||
### Safety net at FV
|
||||
- **Dead man's switch on fv-ml1** — `fv-mesh-watchdog` timer, 1/min, two independent
|
||||
anchors, restores known-good Tailscale after 5 consecutive mesh-only failures, no-ops
|
||||
when the WAN anchor is also down. Proven by positive control. `8c8559b`.
|
||||
- **Break-glass mesh path on fv-ml1** — the box is now OFF the mesh; `fv-mesh-watchdog`
|
||||
(1/min) JOINS it when nh3-dev/nh3-docker go unreachable while the WAN is up. Verified
|
||||
end to end off-mesh: joined as `100.64.0.10`, answered ping+ssh, closed cleanly.
|
||||
- **FV firewall config in the nightly restic run** via ana-docker's pre-backup hook
|
||||
(`80d982d`); `infra-ops` now has an SSH key + `/bin/sh` on the gateway, so the
|
||||
allowlisted WAN path gives a shell rather than API-only.
|
||||
@@ -142,12 +142,12 @@ from the mesh at its LAN address (`9dbd829`); ESPHome pinned/relocated/rotated
|
||||
(`d1769ed` ff); `kb` KB-search tool shipped (`68fa80f`).
|
||||
|
||||
### Open loose ends (none blocking)
|
||||
- **nh3-dev is still a bare mesh member.** Operator authorized retiring both it and
|
||||
fv-ml1 from the mesh; fv-ml1 turned out **load-bearing** (removing it cut FV off from
|
||||
NH3/ESH) and was restored as `100.64.0.9`. nh3-dev was never touched. Now that the
|
||||
SNAT fix exists the original rationale may hold again — **unexecuted, operator's call.**
|
||||
- **Two reusable headscale pre-auth keys** (IDs 5, 6) live until **2026-09-19** — the one
|
||||
credential shape that lets an unknown device self-join. Vaulted, unused, FV cutover done.
|
||||
- **nh3-dev is still a bare mesh member** — retirement authorized, never executed.
|
||||
**Operator's call.** (fv-ml1 IS now retired: off-mesh, node record deleted, reaching the
|
||||
fleet on the gateway's SNAT alone, with break-glass rejoin as its safety net.)
|
||||
- ✅ **Stale reusable pre-auth keys retired** (IDs 5, 6 expired). The mesh now has exactly
|
||||
one live reusable key: the dedicated fv-ml1 break-glass key, ID 8, expires 2027-09-15,
|
||||
vaulted `fv-ml1/headscale-breakglass-key`.
|
||||
- **FV WAN admin is plain HTTP**, no TLS. Allowlisted to ESH/ANA/NH3 egress — ⚠ pinned to
|
||||
**egress IPs**, so a WAN change at any site silently removes the fallback.
|
||||
- **Legacy `/32` ANA NAT rule** redundant and invisible to `source_nat/search_rule`;
|
||||
@@ -161,7 +161,7 @@ from the mesh at its LAN address (`9dbd829`); ESPHome pinned/relocated/rotated
|
||||
|
||||
- `[2026-09-15]` **FV cross-site routing fixed — one OPNsense outbound-NAT rule had been scoped to Anaheim only.** fv-ml1 now reaches NH3/ESH/IRV/ANA/mesh/internet; four rules, all `src=10.251.50.0/24`. The diagnostic signature is the valuable part: every layer looks correct and the discriminator is that *every other site pair works*. → `persistent-memory.d/2026-09-15-fv-cross-site-snat.md`
|
||||
|
||||
- `[2026-09-15]` **Dead man's switch on fv-ml1** — mesh watchdog, two independent anchors, deliberately no-ops when the uplink itself is down. Proven by positive control, not assumed. → `persistent-memory.d/2026-09-15-fv-mesh-watchdog.md`
|
||||
- `[2026-09-15]` **Break-glass mesh path on fv-ml1** — inverted from a restore-watchdog on the operator's suggestion: the box is OFF the mesh and the watchdog JOINS it on fleet loss. Exposed a rejoin key expiring in 4 days; replaced with a dedicated 1-year key and the two stale reusable keys retired. → `persistent-memory.d/2026-09-15-fv-mesh-watchdog.md`
|
||||
|
||||
- `[2026-09-15]` **Fleet identity/group/path conventions pinned + docker trees → `root:docker 2775` setgid on 5 hosts.** `svc-*` in 800-849, infra-ops 850, docker 851, `vh` for new hosts with no retro-renames; `0777` cleared; `linus` deleted; `llmuser` de-privileged. → `persistent-memory.d/2026-09-15-fleet-identity-conventions.md`
|
||||
|
||||
|
||||
Reference in New Issue
Block a user