fix(mesh): make nh3-dev reachable at its LAN address from the mesh

One rule on nh3-scale (CT 107): -d 10.100.10.50/32 -j MASQUERADE, above the
RFC1918 RETURNs in /usr/local/sbin/mesh-exit-masq.sh, so it survives a reboot
rather than living only in the running ruleset.

Cause. A host that runs Tailscale installs -A ts-input -s 100.64.0.0/10
! -i tailscale0 -j DROP. The fleet's subnet routers run NoSNAT: true with
RFC1918 explicitly exempted from masquerade — deliberate source preservation,
and a departure from Tailscale's own --snat-subnet-routes=true default — so a
mesh client's packet reached nh3-dev's ens18 still sourced 100.64.x and died
at the anti-spoof rule. Every NH3 host that does not run Tailscale was
unaffected, which is why this read as a DNS or routing fault rather than a
policy one. Masquerading just this destination makes it behave like every
other host and leaves source preservation absolute elsewhere.

Verified before and after against 13 targets from nh3-dev and 9 from the
MacBook Air, and again after restarting the service so the chain was rebuilt
from the script rather than from the manual insert. nh3-dev.nh3.internal now
resolves and connects from the mesh, ssh and the Booth port included, with no
script changes anywhere.

Records the failed approach prominently, because it is the attractive one:
advertising 10.100.10.50/32 from nh3-dev itself black-holed it from ESH,
Anaheim, FV and Irvine. ip rule there puts lookup 52 at priority 5270 ahead of
main at 32766, and becoming a subnet router let table 52 capture cross-site
traffic the node has no accepted route for. Its own LAN and the internet kept
working throughout, so a single-host check confirms a break it cannot see.
This commit is contained in:
vh
2026-09-14 23:18:32 -07:00
parent e64193171b
commit 9dbd829b9d
2 changed files with 50 additions and 0 deletions
+18
View File
@@ -64,6 +64,24 @@ local Bash already executes here — no SSH-to-self needed for non-privileged wo
(it existed only to satisfy the v1 CLI, which is now gone). Normalising the (it existed only to satisfy the v1 CLI, which is now gone). Normalising the
parent directory is **unresolved — operator's call**; `/opt/docker` itself is a separate three-way split (`755` root, `777` parent directory is **unresolved — operator's call**; `/opt/docker` itself is a separate three-way split (`755` root, `777`
root on two hosts, `755` lkraven). root on two hosts, `755` lkraven).
- **Reachable from the mesh at its LAN address — via a masquerade exception on
nh3-scale, not via anything on this box.** nh3-dev runs Tailscale, so it
carries the anti-spoof rule `-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j
DROP`. The fleet's subnet routers run `NoSNAT: true` with RFC1918 exempted
from masquerade (deliberate source preservation, a departure from Tailscale's
`--snat-subnet-routes=true` default), so a mesh client's packet arrived on
`ens18` still sourced `100.64.x` and was dropped there — silently. Every NH3
host that does *not* run Tailscale was unaffected, which is what made it look
like a DNS or routing fault. Fixed 2026-09-14 by one rule in
`/usr/local/sbin/mesh-exit-masq.sh` on **nh3-scale (CT 107, nh3-pve)**:
`-d 10.100.10.50/32 -j MASQUERADE`, placed above the RFC1918 RETURNs. Copy
kept at `servers/nh3-pve/mesh-exit-masq.sh`.
⚠ **Do not instead advertise `10.100.10.50/32` from nh3-dev.** Tried the same
day: it black-holed nh3-dev from ESH, Anaheim, FV and Irvine while leaving its
own LAN and the internet up. `ip rule` here puts `lookup 52` at priority 5270,
ahead of `main` at 32766, and becoming a subnet router let table 52 capture
cross-site traffic this node has no accepted route for (`RouteAll: false`).
A one-host check against its own LAN passes cleanly — test all four sites.
- **ttyd fleet driver-seat** — web/iPad seat into the zellij `Claude` session (ttyd - **ttyd fleet driver-seat** — web/iPad seat into the zellij `Claude` session (ttyd
behind Caddy; OSC52 clipboard shim). User systemd services under `~/.config`. behind Caddy; OSC52 clipboard shim). User systemd services under `~/.config`.
- **mead-hall** — Bifrost tool-provider sidecar (`:5173`), CI-deployed from - **mead-hall** — Bifrost tool-provider sidecar (`:5173`), CI-deployed from
+32
View File
@@ -0,0 +1,32 @@
#!/bin/sh
# Masquerade mesh clients' INTERNET-bound (exit-node) traffic only; preserve site-to-site source.
iptables -t nat -F MESH-EXIT 2>/dev/null || iptables -t nat -N MESH-EXIT
# ── Mesh-member hosts on this routed subnet — MUST come before the RFC1918 RETURNs ──
# A host that runs Tailscale itself installs an anti-spoof rule:
# -A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP
# With source preservation, a mesh client's packet reaches that host's ETHERNET
# interface still carrying its 100.64.x source, and is dropped there — silently,
# before anything can answer. Hosts that do NOT run Tailscale are unaffected,
# which is why every other NH3 address worked and only this one did not.
# Masquerading just these destinations makes them behave like every other host
# while leaving source preservation absolute for the rest of the subnet.
# (2026-09-14. Tailscale's own default is --snat-subnet-routes=true, i.e. SNAT
# everything; this file is the deliberate departure from that, so the exception
# belongs here rather than as a reason to abandon the design.)
#
# ⚠ Do NOT "fix" this instead by advertising the host's /32 from the host
# itself. That was tried on nh3-dev 2026-09-14 and black-holed it from ESH,
# Anaheim, FV and Irvine — `ip rule` there puts `lookup 52` at priority 5270,
# ahead of main at 32766, and becoming a subnet router let table 52 capture
# cross-site traffic the node had no accepted route for. Its own LAN and the
# internet kept working, so a narrow check looks clean. Fix it at the router.
iptables -t nat -A MESH-EXIT -d 10.100.10.50/32 -j MASQUERADE # nh3-dev
iptables -t nat -A MESH-EXIT -d 10.0.0.0/8 -j RETURN
iptables -t nat -A MESH-EXIT -d 172.16.0.0/12 -j RETURN
iptables -t nat -A MESH-EXIT -d 192.168.0.0/16 -j RETURN
iptables -t nat -A MESH-EXIT -d 100.64.0.0/10 -j RETURN
iptables -t nat -A MESH-EXIT -j MASQUERADE
iptables -t nat -C POSTROUTING -s 100.64.0.0/10 -o eth0 -j MESH-EXIT 2>/dev/null \
|| iptables -t nat -A POSTROUTING -s 100.64.0.0/10 -o eth0 -j MESH-EXIT