fix(mesh): make nh3-dev reachable at its LAN address from the mesh

One rule on nh3-scale (CT 107): -d 10.100.10.50/32 -j MASQUERADE, above the
RFC1918 RETURNs in /usr/local/sbin/mesh-exit-masq.sh, so it survives a reboot
rather than living only in the running ruleset.

Cause. A host that runs Tailscale installs -A ts-input -s 100.64.0.0/10
! -i tailscale0 -j DROP. The fleet's subnet routers run NoSNAT: true with
RFC1918 explicitly exempted from masquerade — deliberate source preservation,
and a departure from Tailscale's own --snat-subnet-routes=true default — so a
mesh client's packet reached nh3-dev's ens18 still sourced 100.64.x and died
at the anti-spoof rule. Every NH3 host that does not run Tailscale was
unaffected, which is why this read as a DNS or routing fault rather than a
policy one. Masquerading just this destination makes it behave like every
other host and leaves source preservation absolute elsewhere.

Verified before and after against 13 targets from nh3-dev and 9 from the
MacBook Air, and again after restarting the service so the chain was rebuilt
from the script rather than from the manual insert. nh3-dev.nh3.internal now
resolves and connects from the mesh, ssh and the Booth port included, with no
script changes anywhere.

Records the failed approach prominently, because it is the attractive one:
advertising 10.100.10.50/32 from nh3-dev itself black-holed it from ESH,
Anaheim, FV and Irvine. ip rule there puts lookup 52 at priority 5270 ahead of
main at 32766, and becoming a subnet router let table 52 capture cross-site
traffic the node has no accepted route for. Its own LAN and the internet kept
working throughout, so a single-host check confirms a break it cannot see.
This commit is contained in:
2026-09-14 23:18:32 -07:00
parent e64193171b
commit 9dbd829b9d
2 changed files with 50 additions and 0 deletions
+18
View File
@@ -64,6 +64,24 @@ local Bash already executes here — no SSH-to-self needed for non-privileged wo
(it existed only to satisfy the v1 CLI, which is now gone). Normalising the
parent directory is **unresolved — operator's call**; `/opt/docker` itself is a separate three-way split (`755` root, `777`
root on two hosts, `755` lkraven).
- **Reachable from the mesh at its LAN address — via a masquerade exception on
nh3-scale, not via anything on this box.** nh3-dev runs Tailscale, so it
carries the anti-spoof rule `-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j
DROP`. The fleet's subnet routers run `NoSNAT: true` with RFC1918 exempted
from masquerade (deliberate source preservation, a departure from Tailscale's
`--snat-subnet-routes=true` default), so a mesh client's packet arrived on
`ens18` still sourced `100.64.x` and was dropped there — silently. Every NH3
host that does *not* run Tailscale was unaffected, which is what made it look
like a DNS or routing fault. Fixed 2026-09-14 by one rule in
`/usr/local/sbin/mesh-exit-masq.sh` on **nh3-scale (CT 107, nh3-pve)**:
`-d 10.100.10.50/32 -j MASQUERADE`, placed above the RFC1918 RETURNs. Copy
kept at `servers/nh3-pve/mesh-exit-masq.sh`.
**Do not instead advertise `10.100.10.50/32` from nh3-dev.** Tried the same
day: it black-holed nh3-dev from ESH, Anaheim, FV and Irvine while leaving its
own LAN and the internet up. `ip rule` here puts `lookup 52` at priority 5270,
ahead of `main` at 32766, and becoming a subnet router let table 52 capture
cross-site traffic this node has no accepted route for (`RouteAll: false`).
A one-host check against its own LAN passes cleanly — test all four sites.
- **ttyd fleet driver-seat** — web/iPad seat into the zellij `Claude` session (ttyd
behind Caddy; OSC52 clipboard shim). User systemd services under `~/.config`.
- **mead-hall** — Bifrost tool-provider sidecar (`:5173`), CI-deployed from
+32
View File
@@ -0,0 +1,32 @@
#!/bin/sh
# Masquerade mesh clients' INTERNET-bound (exit-node) traffic only; preserve site-to-site source.
iptables -t nat -F MESH-EXIT 2>/dev/null || iptables -t nat -N MESH-EXIT
# ── Mesh-member hosts on this routed subnet — MUST come before the RFC1918 RETURNs ──
# A host that runs Tailscale itself installs an anti-spoof rule:
# -A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP
# With source preservation, a mesh client's packet reaches that host's ETHERNET
# interface still carrying its 100.64.x source, and is dropped there — silently,
# before anything can answer. Hosts that do NOT run Tailscale are unaffected,
# which is why every other NH3 address worked and only this one did not.
# Masquerading just these destinations makes them behave like every other host
# while leaving source preservation absolute for the rest of the subnet.
# (2026-09-14. Tailscale's own default is --snat-subnet-routes=true, i.e. SNAT
# everything; this file is the deliberate departure from that, so the exception
# belongs here rather than as a reason to abandon the design.)
#
# ⚠ Do NOT "fix" this instead by advertising the host's /32 from the host
# itself. That was tried on nh3-dev 2026-09-14 and black-holed it from ESH,
# Anaheim, FV and Irvine — `ip rule` there puts `lookup 52` at priority 5270,
# ahead of main at 32766, and becoming a subnet router let table 52 capture
# cross-site traffic the node had no accepted route for. Its own LAN and the
# internet kept working, so a narrow check looks clean. Fix it at the router.
iptables -t nat -A MESH-EXIT -d 10.100.10.50/32 -j MASQUERADE # nh3-dev
iptables -t nat -A MESH-EXIT -d 10.0.0.0/8 -j RETURN
iptables -t nat -A MESH-EXIT -d 172.16.0.0/12 -j RETURN
iptables -t nat -A MESH-EXIT -d 192.168.0.0/16 -j RETURN
iptables -t nat -A MESH-EXIT -d 100.64.0.0/10 -j RETURN
iptables -t nat -A MESH-EXIT -j MASQUERADE
iptables -t nat -C POSTROUTING -s 100.64.0.0/10 -o eth0 -j MESH-EXIT 2>/dev/null \
|| iptables -t nat -A POSTROUTING -s 100.64.0.0/10 -o eth0 -j MESH-EXIT