diff --git a/docs/pfi/headscale-mesh-plan.md b/docs/pfi/headscale-mesh-plan.md index f080d52..7bb1fa3 100644 --- a/docs/pfi/headscale-mesh-plan.md +++ b/docs/pfi/headscale-mesh-plan.md @@ -402,3 +402,47 @@ irv-ml1). Fleet-wide reach added via gateway static routes `10.6.110.0/24 → lo (NH3 UDM→nh3-scale, ESH UDM→esh-scale, FortiGate route 22→ana-scale). Verified: ana-docker→10.6.110.1 and esh-docker-vm→10.6.110.50 both OK over the mesh. So irv-ml1 both REACHES the fleet /16s (accept-routes) and EXPOSES its Irvine LAN to the fleet (advertise-routes) — a full mesh subnet router. + +## 2026-09-18 — NH3↔Anaheim was relayed, not direct; fixed at the Anaheim edge + +For months every NH3→Anaheim flow crossed Tailscale's LA DERP relay rather +than a direct path. `tailscale ping` from nh3-scale reported `direct +connection not established` after 8 attempts, all pongs via `DERP(lax)` at +373–522 ms. DERP is a throttled fallback, not a data plane, so this cost both +a fixed round-trip penalty and a hard bandwidth ceiling on everything +crossing that hop — LiteLLM, Beszel, task-board, vor, the Henge. + +**Cause.** ana-scale advertised `38.120.12.42:41641`, but the Anaheim NAT was +not preserving the port (netcheck mapped it to `:60798`) and `PortMapping` was +empty — no UPnP/NAT-PMP — so inbound hole-punching always failed. +`MappingVariesByDestIP: false`, so not symmetric NAT; simply no reachable +inbound endpoint. ESH↔Anaheim and NH3↔ESH were already direct; only this pair +failed. + +**Fix.** Four additive objects on ana-gw (10.250.0.1) giving ana-scale a +stable inbound endpoint. The box runs `central-nat enable`, so a policy's +`dstaddr` is the real internal address rather than the VIP — the existing +`wg-to-ana-wg` VIP+policy pair is the house template. + +| object | value | +|---|---| +| `firewall address` | `ana-scale-ip` → 10.250.50.45/32, iface `servers` | +| `firewall service custom` | `Tailscale-41641` → `udp-portrange 41641` | +| `firewall vip` | `tailscale-to-ana-scale` → 38.120.12.42:41641 udp → 10.250.50.45:41641, extintf wan1 | +| `firewall policy` id 75 | wan1→servers, all→`ana-scale-ip`, `Tailscale-41641`, accept | + +**Measured before → after:** + +| path | before | after | +|---|---|---| +| `tailscale ping` nh3-scale→ana-scale | 373–522 ms via DERP(lax) | **6 ms direct** | +| STT via the Anaheim LiteLLM gateway (96 kB clip) | 1.399–1.449 s | **0.237–0.270 s** | +| Beszel HTTP, nh3-dev→ana-docker | 0.94–1.29 s | **0.014–0.016 s** | + +⚠ **This edge has no out-of-band access.** Back up before touching it — +`ssh infra-ops@10.250.0.1 'show' > file` captures the full non-default config +without needing a tftp server — and make additive changes only. Credentials +are vaulted at `fortigate/ana-gw-infra-ops-password`. + +⚠ **Still outstanding: irv-ml1 (100.64.0.6) remains `relay "lax"`.** Same class +of problem at a different site; this change does not address it.