feat(mesh): retire nh3-dev from the mesh and revert the masquerade it required

nh3-dev sits on the NH3 LAN and reaches every site through its own default
gateway; RouteAll was already false, so it never used the tunnel for routing.
Membership bought a 100.64.0.4 address nothing referenced -- grep across the
repo and ~/development found only docs and memory hits.

It also cost something concrete. A host running Tailscale installs
-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP, and because the fleet's
subnet routers preserve source rather than masquerading RFC1918, a mesh client's
packet reached nh3-dev's ens18 still sourced 100.64.x and was dropped silently.
That is why nh3-dev.nh3.internal failed from the mesh while every NH3 host that
does not run Tailscale worked, and it needed a -d 10.100.10.50/32 -j MASQUERADE
exception on nh3-scale to paper over. Retiring the membership removed the
anti-spoof rule, so the exception went with it -- mesh-exit-masq.sh is back to
the two rules it had before yesterday.

Verified after: nh3-dev reachable at 10.100.10.50 from ESH, Anaheim, FV, Irvine
and NH3, and reaching all four sites plus the internet itself. fv-ml1
unaffected.

The mesh is now six nodes and every one has a job: three site routers,
vb-gateway, irv-ml1 (Irvine's own router, no separate scale node), and the
operator's MacBook Air. Nothing is enrolled just in case.
This commit is contained in:
2026-09-15 01:05:23 -07:00
parent 959a743256
commit a65cdf65d9
3 changed files with 17 additions and 44 deletions
+5 -5
View File
@@ -137,14 +137,14 @@ Anaheim only. Commits `fa04f45`, `0ab9da5`.
### Fleet hygiene landed
`root:docker 2775` setgid deploy trees on all 5 hosts + `0777` cleared (`826a63b`);
conventions pinned in `docs/pfi/fleet-conventions.md` with a read-only audit playbook
(`abef67a`); `linus` deleted and `llmuser` de-privileged (`ce7b07f`); nh3-dev reachable
from the mesh at its LAN address (`9dbd829`); ESPHome pinned/relocated/rotated
(`abef67a`); `linus` deleted and `llmuser` de-privileged (`ce7b07f`); nh3-dev retired from the mesh and its masquerade
exception reverted with it (`9dbd829` superseded); ESPHome pinned/relocated/rotated
(`d1769ed` ff); `kb` KB-search tool shipped (`68fa80f`).
### Open loose ends (none blocking)
- **nh3-dev is still a bare mesh member** — retirement authorized, never executed.
**Operator's call.** (fv-ml1 IS now retired: off-mesh, node record deleted, reaching the
fleet on the gateway's SNAT alone, with break-glass rejoin as its safety net.)
- **Mesh membership retired for both nh3-dev and fv-ml1** (2026-09-15). Six nodes remain:
the three site routers, vb-gateway, irv-ml1 (Irvine's own router) and the MacBook Air.
Every one of those has a job; nothing is enrolled "just in case" any more.
-**Stale reusable pre-auth keys retired** (IDs 5, 6 expired). The mesh now has exactly
one live reusable key: the dedicated fv-ml1 break-glass key, ID 8, expires 2027-09-15,
vaulted `fv-ml1/headscale-breakglass-key`.
+12 -18
View File
@@ -64,24 +64,18 @@ local Bash already executes here — no SSH-to-self needed for non-privileged wo
(it existed only to satisfy the v1 CLI, which is now gone). Normalising the
parent directory is **unresolved — operator's call**; `/opt/docker` itself is a separate three-way split (`755` root, `777`
root on two hosts, `755` lkraven).
- **Reachable from the mesh at its LAN address — via a masquerade exception on
nh3-scale, not via anything on this box.** nh3-dev runs Tailscale, so it
carries the anti-spoof rule `-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j
DROP`. The fleet's subnet routers run `NoSNAT: true` with RFC1918 exempted
from masquerade (deliberate source preservation, a departure from Tailscale's
`--snat-subnet-routes=true` default), so a mesh client's packet arrived on
`ens18` still sourced `100.64.x` and was dropped there — silently. Every NH3
host that does *not* run Tailscale was unaffected, which is what made it look
like a DNS or routing fault. Fixed 2026-09-14 by one rule in
`/usr/local/sbin/mesh-exit-masq.sh` on **nh3-scale (CT 107, nh3-pve)**:
`-d 10.100.10.50/32 -j MASQUERADE`, placed above the RFC1918 RETURNs. Copy
kept at `servers/nh3-pve/mesh-exit-masq.sh`.
**Do not instead advertise `10.100.10.50/32` from nh3-dev.** Tried the same
day: it black-holed nh3-dev from ESH, Anaheim, FV and Irvine while leaving its
own LAN and the internet up. `ip rule` here puts `lookup 52` at priority 5270,
ahead of `main` at 32766, and becoming a subnet router let table 52 capture
cross-site traffic this node has no accepted route for (`RouteAll: false`).
A one-host check against its own LAN passes cleanly — test all four sites.
- **NOT a mesh node** (retired 2026-09-15). nh3-dev sits on the NH3 LAN and reaches
every site through its own default gateway — `RouteAll: false` meant it never used
the tunnel for routing anyway, so membership bought only a `100.64.0.4` address
nothing referenced. ⭐ It also *cost* something: a host running Tailscale installs
`-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP`, and because the fleet's
subnet routers preserve source rather than masquerading RFC1918, a mesh client's
packet reached `ens18` still sourced `100.64.x` and was silently dropped — which is
why `nh3-dev.nh3.internal` failed from the mesh while every non-Tailscale NH3 host
worked. That needed a `-d 10.100.10.50/32 -j MASQUERADE` exception on nh3-scale;
retiring the membership removed the anti-spoof rule and the exception with it.
Verified after: reachable at `10.100.10.50` from ESH, Anaheim, FV, Irvine and NH3,
and reaching all four sites plus the internet itself.
- **ttyd fleet driver-seat** — web/iPad seat into the zellij `Claude` session (ttyd
behind Caddy; OSC52 clipboard shim). User systemd services under `~/.config`.
- **mead-hall** — Bifrost tool-provider sidecar (`:5173`), CI-deployed from
-21
View File
@@ -2,27 +2,6 @@
# Masquerade mesh clients' INTERNET-bound (exit-node) traffic only; preserve site-to-site source.
iptables -t nat -F MESH-EXIT 2>/dev/null || iptables -t nat -N MESH-EXIT
# ── Mesh-member hosts on this routed subnet — MUST come before the RFC1918 RETURNs ──
# A host that runs Tailscale itself installs an anti-spoof rule:
# -A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP
# With source preservation, a mesh client's packet reaches that host's ETHERNET
# interface still carrying its 100.64.x source, and is dropped there — silently,
# before anything can answer. Hosts that do NOT run Tailscale are unaffected,
# which is why every other NH3 address worked and only this one did not.
# Masquerading just these destinations makes them behave like every other host
# while leaving source preservation absolute for the rest of the subnet.
# (2026-09-14. Tailscale's own default is --snat-subnet-routes=true, i.e. SNAT
# everything; this file is the deliberate departure from that, so the exception
# belongs here rather than as a reason to abandon the design.)
#
# ⚠ Do NOT "fix" this instead by advertising the host's /32 from the host
# itself. That was tried on nh3-dev 2026-09-14 and black-holed it from ESH,
# Anaheim, FV and Irvine — `ip rule` there puts `lookup 52` at priority 5270,
# ahead of main at 32766, and becoming a subnet router let table 52 capture
# cross-site traffic the node had no accepted route for. Its own LAN and the
# internet kept working, so a narrow check looks clean. Fix it at the router.
iptables -t nat -A MESH-EXIT -d 10.100.10.50/32 -j MASQUERADE # nh3-dev
iptables -t nat -A MESH-EXIT -d 10.0.0.0/8 -j RETURN
iptables -t nat -A MESH-EXIT -d 172.16.0.0/12 -j RETURN
iptables -t nat -A MESH-EXIT -d 192.168.0.0/16 -j RETURN