diff --git a/persistent-memory.md b/persistent-memory.md index 33a57a2..2f72662 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -137,14 +137,14 @@ Anaheim only. Commits `fa04f45`, `0ab9da5`. ### Fleet hygiene landed `root:docker 2775` setgid deploy trees on all 5 hosts + `0777` cleared (`826a63b`); conventions pinned in `docs/pfi/fleet-conventions.md` with a read-only audit playbook -(`abef67a`); `linus` deleted and `llmuser` de-privileged (`ce7b07f`); nh3-dev reachable -from the mesh at its LAN address (`9dbd829`); ESPHome pinned/relocated/rotated +(`abef67a`); `linus` deleted and `llmuser` de-privileged (`ce7b07f`); nh3-dev retired from the mesh and its masquerade +exception reverted with it (`9dbd829` superseded); ESPHome pinned/relocated/rotated (`d1769ed` ff); `kb` KB-search tool shipped (`68fa80f`). ### Open loose ends (none blocking) -- **nh3-dev is still a bare mesh member** — retirement authorized, never executed. - **Operator's call.** (fv-ml1 IS now retired: off-mesh, node record deleted, reaching the - fleet on the gateway's SNAT alone, with break-glass rejoin as its safety net.) +- ✅ **Mesh membership retired for both nh3-dev and fv-ml1** (2026-09-15). Six nodes remain: + the three site routers, vb-gateway, irv-ml1 (Irvine's own router) and the MacBook Air. + Every one of those has a job; nothing is enrolled "just in case" any more. - ✅ **Stale reusable pre-auth keys retired** (IDs 5, 6 expired). The mesh now has exactly one live reusable key: the dedicated fv-ml1 break-glass key, ID 8, expires 2027-09-15, vaulted `fv-ml1/headscale-breakglass-key`. diff --git a/servers/nh3-dev/README.md b/servers/nh3-dev/README.md index ed04ce7..6f8d36d 100644 --- a/servers/nh3-dev/README.md +++ b/servers/nh3-dev/README.md @@ -64,24 +64,18 @@ local Bash already executes here — no SSH-to-self needed for non-privileged wo (it existed only to satisfy the v1 CLI, which is now gone). Normalising the parent directory is **unresolved — operator's call**; `/opt/docker` itself is a separate three-way split (`755` root, `777` root on two hosts, `755` lkraven). -- **Reachable from the mesh at its LAN address — via a masquerade exception on - nh3-scale, not via anything on this box.** nh3-dev runs Tailscale, so it - carries the anti-spoof rule `-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j - DROP`. The fleet's subnet routers run `NoSNAT: true` with RFC1918 exempted - from masquerade (deliberate source preservation, a departure from Tailscale's - `--snat-subnet-routes=true` default), so a mesh client's packet arrived on - `ens18` still sourced `100.64.x` and was dropped there — silently. Every NH3 - host that does *not* run Tailscale was unaffected, which is what made it look - like a DNS or routing fault. Fixed 2026-09-14 by one rule in - `/usr/local/sbin/mesh-exit-masq.sh` on **nh3-scale (CT 107, nh3-pve)**: - `-d 10.100.10.50/32 -j MASQUERADE`, placed above the RFC1918 RETURNs. Copy - kept at `servers/nh3-pve/mesh-exit-masq.sh`. - ⚠ **Do not instead advertise `10.100.10.50/32` from nh3-dev.** Tried the same - day: it black-holed nh3-dev from ESH, Anaheim, FV and Irvine while leaving its - own LAN and the internet up. `ip rule` here puts `lookup 52` at priority 5270, - ahead of `main` at 32766, and becoming a subnet router let table 52 capture - cross-site traffic this node has no accepted route for (`RouteAll: false`). - A one-host check against its own LAN passes cleanly — test all four sites. +- **NOT a mesh node** (retired 2026-09-15). nh3-dev sits on the NH3 LAN and reaches + every site through its own default gateway — `RouteAll: false` meant it never used + the tunnel for routing anyway, so membership bought only a `100.64.0.4` address + nothing referenced. ⭐ It also *cost* something: a host running Tailscale installs + `-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP`, and because the fleet's + subnet routers preserve source rather than masquerading RFC1918, a mesh client's + packet reached `ens18` still sourced `100.64.x` and was silently dropped — which is + why `nh3-dev.nh3.internal` failed from the mesh while every non-Tailscale NH3 host + worked. That needed a `-d 10.100.10.50/32 -j MASQUERADE` exception on nh3-scale; + retiring the membership removed the anti-spoof rule and the exception with it. + Verified after: reachable at `10.100.10.50` from ESH, Anaheim, FV, Irvine and NH3, + and reaching all four sites plus the internet itself. - **ttyd fleet driver-seat** — web/iPad seat into the zellij `Claude` session (ttyd behind Caddy; OSC52 clipboard shim). User systemd services under `~/.config`. - **mead-hall** — Bifrost tool-provider sidecar (`:5173`), CI-deployed from diff --git a/servers/nh3-pve/mesh-exit-masq.sh b/servers/nh3-pve/mesh-exit-masq.sh index 41fd027..c9b34d5 100644 --- a/servers/nh3-pve/mesh-exit-masq.sh +++ b/servers/nh3-pve/mesh-exit-masq.sh @@ -2,27 +2,6 @@ # Masquerade mesh clients' INTERNET-bound (exit-node) traffic only; preserve site-to-site source. iptables -t nat -F MESH-EXIT 2>/dev/null || iptables -t nat -N MESH-EXIT -# ── Mesh-member hosts on this routed subnet — MUST come before the RFC1918 RETURNs ── -# A host that runs Tailscale itself installs an anti-spoof rule: -# -A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP -# With source preservation, a mesh client's packet reaches that host's ETHERNET -# interface still carrying its 100.64.x source, and is dropped there — silently, -# before anything can answer. Hosts that do NOT run Tailscale are unaffected, -# which is why every other NH3 address worked and only this one did not. -# Masquerading just these destinations makes them behave like every other host -# while leaving source preservation absolute for the rest of the subnet. -# (2026-09-14. Tailscale's own default is --snat-subnet-routes=true, i.e. SNAT -# everything; this file is the deliberate departure from that, so the exception -# belongs here rather than as a reason to abandon the design.) -# -# ⚠ Do NOT "fix" this instead by advertising the host's /32 from the host -# itself. That was tried on nh3-dev 2026-09-14 and black-holed it from ESH, -# Anaheim, FV and Irvine — `ip rule` there puts `lookup 52` at priority 5270, -# ahead of main at 32766, and becoming a subnet router let table 52 capture -# cross-site traffic the node had no accepted route for. Its own LAN and the -# internet kept working, so a narrow check looks clean. Fix it at the router. -iptables -t nat -A MESH-EXIT -d 10.100.10.50/32 -j MASQUERADE # nh3-dev - iptables -t nat -A MESH-EXIT -d 10.0.0.0/8 -j RETURN iptables -t nat -A MESH-EXIT -d 172.16.0.0/12 -j RETURN iptables -t nat -A MESH-EXIT -d 192.168.0.0/16 -j RETURN