feat(mesh): retire nh3-dev from the mesh and revert the masquerade it required

nh3-dev sits on the NH3 LAN and reaches every site through its own default
gateway; RouteAll was already false, so it never used the tunnel for routing.
Membership bought a 100.64.0.4 address nothing referenced -- grep across the
repo and ~/development found only docs and memory hits.

It also cost something concrete. A host running Tailscale installs
-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP, and because the fleet's
subnet routers preserve source rather than masquerading RFC1918, a mesh client's
packet reached nh3-dev's ens18 still sourced 100.64.x and was dropped silently.
That is why nh3-dev.nh3.internal failed from the mesh while every NH3 host that
does not run Tailscale worked, and it needed a -d 10.100.10.50/32 -j MASQUERADE
exception on nh3-scale to paper over. Retiring the membership removed the
anti-spoof rule, so the exception went with it -- mesh-exit-masq.sh is back to
the two rules it had before yesterday.

Verified after: nh3-dev reachable at 10.100.10.50 from ESH, Anaheim, FV, Irvine
and NH3, and reaching all four sites plus the internet itself. fv-ml1
unaffected.

The mesh is now six nodes and every one has a job: three site routers,
vb-gateway, irv-ml1 (Irvine's own router, no separate scale node), and the
operator's MacBook Air. Nothing is enrolled just in case.
This commit is contained in:
vh
2026-09-15 01:05:23 -07:00
parent 959a743256
commit a65cdf65d9
3 changed files with 17 additions and 44 deletions
+5 -5
View File
@@ -137,14 +137,14 @@ Anaheim only. Commits `fa04f45`, `0ab9da5`.
### Fleet hygiene landed ### Fleet hygiene landed
`root:docker 2775` setgid deploy trees on all 5 hosts + `0777` cleared (`826a63b`); `root:docker 2775` setgid deploy trees on all 5 hosts + `0777` cleared (`826a63b`);
conventions pinned in `docs/pfi/fleet-conventions.md` with a read-only audit playbook conventions pinned in `docs/pfi/fleet-conventions.md` with a read-only audit playbook
(`abef67a`); `linus` deleted and `llmuser` de-privileged (`ce7b07f`); nh3-dev reachable (`abef67a`); `linus` deleted and `llmuser` de-privileged (`ce7b07f`); nh3-dev retired from the mesh and its masquerade
from the mesh at its LAN address (`9dbd829`); ESPHome pinned/relocated/rotated exception reverted with it (`9dbd829` superseded); ESPHome pinned/relocated/rotated
(`d1769ed` ff); `kb` KB-search tool shipped (`68fa80f`). (`d1769ed` ff); `kb` KB-search tool shipped (`68fa80f`).
### Open loose ends (none blocking) ### Open loose ends (none blocking)
- **nh3-dev is still a bare mesh member** — retirement authorized, never executed. - ✅ **Mesh membership retired for both nh3-dev and fv-ml1** (2026-09-15). Six nodes remain:
**Operator's call.** (fv-ml1 IS now retired: off-mesh, node record deleted, reaching the the three site routers, vb-gateway, irv-ml1 (Irvine's own router) and the MacBook Air.
fleet on the gateway's SNAT alone, with break-glass rejoin as its safety net.) Every one of those has a job; nothing is enrolled "just in case" any more.
- ✅ **Stale reusable pre-auth keys retired** (IDs 5, 6 expired). The mesh now has exactly - ✅ **Stale reusable pre-auth keys retired** (IDs 5, 6 expired). The mesh now has exactly
one live reusable key: the dedicated fv-ml1 break-glass key, ID 8, expires 2027-09-15, one live reusable key: the dedicated fv-ml1 break-glass key, ID 8, expires 2027-09-15,
vaulted `fv-ml1/headscale-breakglass-key`. vaulted `fv-ml1/headscale-breakglass-key`.
+12 -18
View File
@@ -64,24 +64,18 @@ local Bash already executes here — no SSH-to-self needed for non-privileged wo
(it existed only to satisfy the v1 CLI, which is now gone). Normalising the (it existed only to satisfy the v1 CLI, which is now gone). Normalising the
parent directory is **unresolved — operator's call**; `/opt/docker` itself is a separate three-way split (`755` root, `777` parent directory is **unresolved — operator's call**; `/opt/docker` itself is a separate three-way split (`755` root, `777`
root on two hosts, `755` lkraven). root on two hosts, `755` lkraven).
- **Reachable from the mesh at its LAN address — via a masquerade exception on - **NOT a mesh node** (retired 2026-09-15). nh3-dev sits on the NH3 LAN and reaches
nh3-scale, not via anything on this box.** nh3-dev runs Tailscale, so it every site through its own default gateway — `RouteAll: false` meant it never used
carries the anti-spoof rule `-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j the tunnel for routing anyway, so membership bought only a `100.64.0.4` address
DROP`. The fleet's subnet routers run `NoSNAT: true` with RFC1918 exempted nothing referenced. ⭐ It also *cost* something: a host running Tailscale installs
from masquerade (deliberate source preservation, a departure from Tailscale's `-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP`, and because the fleet's
`--snat-subnet-routes=true` default), so a mesh client's packet arrived on subnet routers preserve source rather than masquerading RFC1918, a mesh client's
`ens18` still sourced `100.64.x` and was dropped there — silently. Every NH3 packet reached `ens18` still sourced `100.64.x` and was silently dropped — which is
host that does *not* run Tailscale was unaffected, which is what made it look why `nh3-dev.nh3.internal` failed from the mesh while every non-Tailscale NH3 host
like a DNS or routing fault. Fixed 2026-09-14 by one rule in worked. That needed a `-d 10.100.10.50/32 -j MASQUERADE` exception on nh3-scale;
`/usr/local/sbin/mesh-exit-masq.sh` on **nh3-scale (CT 107, nh3-pve)**: retiring the membership removed the anti-spoof rule and the exception with it.
`-d 10.100.10.50/32 -j MASQUERADE`, placed above the RFC1918 RETURNs. Copy Verified after: reachable at `10.100.10.50` from ESH, Anaheim, FV, Irvine and NH3,
kept at `servers/nh3-pve/mesh-exit-masq.sh`. and reaching all four sites plus the internet itself.
⚠ **Do not instead advertise `10.100.10.50/32` from nh3-dev.** Tried the same
day: it black-holed nh3-dev from ESH, Anaheim, FV and Irvine while leaving its
own LAN and the internet up. `ip rule` here puts `lookup 52` at priority 5270,
ahead of `main` at 32766, and becoming a subnet router let table 52 capture
cross-site traffic this node has no accepted route for (`RouteAll: false`).
A one-host check against its own LAN passes cleanly — test all four sites.
- **ttyd fleet driver-seat** — web/iPad seat into the zellij `Claude` session (ttyd - **ttyd fleet driver-seat** — web/iPad seat into the zellij `Claude` session (ttyd
behind Caddy; OSC52 clipboard shim). User systemd services under `~/.config`. behind Caddy; OSC52 clipboard shim). User systemd services under `~/.config`.
- **mead-hall** — Bifrost tool-provider sidecar (`:5173`), CI-deployed from - **mead-hall** — Bifrost tool-provider sidecar (`:5173`), CI-deployed from
-21
View File
@@ -2,27 +2,6 @@
# Masquerade mesh clients' INTERNET-bound (exit-node) traffic only; preserve site-to-site source. # Masquerade mesh clients' INTERNET-bound (exit-node) traffic only; preserve site-to-site source.
iptables -t nat -F MESH-EXIT 2>/dev/null || iptables -t nat -N MESH-EXIT iptables -t nat -F MESH-EXIT 2>/dev/null || iptables -t nat -N MESH-EXIT
# ── Mesh-member hosts on this routed subnet — MUST come before the RFC1918 RETURNs ──
# A host that runs Tailscale itself installs an anti-spoof rule:
# -A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP
# With source preservation, a mesh client's packet reaches that host's ETHERNET
# interface still carrying its 100.64.x source, and is dropped there — silently,
# before anything can answer. Hosts that do NOT run Tailscale are unaffected,
# which is why every other NH3 address worked and only this one did not.
# Masquerading just these destinations makes them behave like every other host
# while leaving source preservation absolute for the rest of the subnet.
# (2026-09-14. Tailscale's own default is --snat-subnet-routes=true, i.e. SNAT
# everything; this file is the deliberate departure from that, so the exception
# belongs here rather than as a reason to abandon the design.)
#
# ⚠ Do NOT "fix" this instead by advertising the host's /32 from the host
# itself. That was tried on nh3-dev 2026-09-14 and black-holed it from ESH,
# Anaheim, FV and Irvine — `ip rule` there puts `lookup 52` at priority 5270,
# ahead of main at 32766, and becoming a subnet router let table 52 capture
# cross-site traffic the node had no accepted route for. Its own LAN and the
# internet kept working, so a narrow check looks clean. Fix it at the router.
iptables -t nat -A MESH-EXIT -d 10.100.10.50/32 -j MASQUERADE # nh3-dev
iptables -t nat -A MESH-EXIT -d 10.0.0.0/8 -j RETURN iptables -t nat -A MESH-EXIT -d 10.0.0.0/8 -j RETURN
iptables -t nat -A MESH-EXIT -d 172.16.0.0/12 -j RETURN iptables -t nat -A MESH-EXIT -d 172.16.0.0/12 -j RETURN
iptables -t nat -A MESH-EXIT -d 192.168.0.0/16 -j RETURN iptables -t nat -A MESH-EXIT -d 192.168.0.0/16 -j RETURN