vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(mesh): record the ana-gw port-forward that ended the NH3↔ANA relay

Browse Source 
Every NH3→Anaheim flow had been crossing Tailscale's LA DERP relay rather than
a direct path, for long enough to have carried 78 GB tx on the NH3 side alone.
DERP is a throttled fallback, so this imposed both a fixed round-trip penalty
and a bandwidth ceiling on LiteLLM, Beszel, task-board, vor and the Henge
alike. It surfaced as a voice-loop latency report from tts-dev, not as a
network alarm, because nothing monitors whether a mesh path is direct.

ana-scale advertised 38.120.12.42:41641 while the Anaheim NAT mapped it to
:60798 with no port-mapping protocol available, so inbound hole-punching always
failed. Four additive objects on ana-gw give it a stable inbound endpoint.

  tailscale ping nh3-scale->ana-scale   373-522 ms via DERP  ->  6 ms direct
  STT via the ANA gateway, 96 kB clip   1.399-1.449 s        ->  0.237-0.270 s
  Beszel HTTP nh3-dev->ana-docker       0.94-1.29 s          ->  0.014-0.016 s

Documents the house template that matters for this box: it runs central-nat,
so a policy dstaddr is the real internal address and not the VIP. Also records
that the pre-change config was captured with `show` to a local file rather
than a tftp job, since this edge has no out-of-band access and a backup is
mandatory before touching it.

irv-ml1 remains relayed and is called out as outstanding.
This commit is contained in:
Vuong Hoang
2026-09-18 14:16:36 -07:00
parent 5b20b02fb9
commit 5a9fad8240
1 changed files with 44 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/pfi/headscale-mesh-plan.md
+44
View File
@@ -402,3 +402,47 @@ irv-ml1). Fleet-wide reach added via gateway static routes `10.6.110.0/24 → lo
(NH3 UDM→nh3-scale, ESH UDM→esh-scale, FortiGate route 22→ana-scale). Verified: ana-docker→10.6.110.1
and esh-docker-vm→10.6.110.50 both OK over the mesh. So irv-ml1 both REACHES the fleet /16s
(accept-routes) and EXPOSES its Irvine LAN to the fleet (advertise-routes) — a full mesh subnet router.
## 2026-09-18 — NH3↔Anaheim was relayed, not direct; fixed at the Anaheim edge
For months every NH3→Anaheim flow crossed Tailscale's LA DERP relay rather
than a direct path. `tailscale ping` from nh3-scale reported `direct
connection not established` after 8 attempts, all pongs via `DERP(lax)` at
373–522 ms. DERP is a throttled fallback, not a data plane, so this cost both
a fixed round-trip penalty and a hard bandwidth ceiling on everything
crossing that hop — LiteLLM, Beszel, task-board, vor, the Henge.
**Cause.** ana-scale advertised `38.120.12.42:41641`, but the Anaheim NAT was
not preserving the port (netcheck mapped it to `:60798`) and `PortMapping` was
empty — no UPnP/NAT-PMP — so inbound hole-punching always failed.
`MappingVariesByDestIP: false`, so not symmetric NAT; simply no reachable
inbound endpoint. ESH↔Anaheim and NH3↔ESH were already direct; only this pair
failed.
**Fix.** Four additive objects on ana-gw (10.250.0.1) giving ana-scale a
stable inbound endpoint. The box runs `central-nat enable`, so a policy's
`dstaddr` is the real internal address rather than the VIP — the existing
`wg-to-ana-wg` VIP+policy pair is the house template.
| object | value |
|---|---|
| `firewall address` | `ana-scale-ip` → 10.250.50.45/32, iface `servers` |
| `firewall service custom` | `Tailscale-41641` → `udp-portrange 41641` |
| `firewall vip` | `tailscale-to-ana-scale` → 38.120.12.42:41641 udp → 10.250.50.45:41641, extintf wan1 |
| `firewall policy` id 75 | wan1→servers, all→`ana-scale-ip`, `Tailscale-41641`, accept |
**Measured before → after:**
| path | before | after |
|---|---|---|
| `tailscale ping` nh3-scale→ana-scale | 373–522 ms via DERP(lax) | **6 ms direct** |
| STT via the Anaheim LiteLLM gateway (96 kB clip) | 1.399–1.449 s | **0.237–0.270 s** |
| Beszel HTTP, nh3-dev→ana-docker | 0.94–1.29 s | **0.014–0.016 s** |
⚠ **This edge has no out-of-band access.** Back up before touching it —
`ssh infra-ops@10.250.0.1 'show' > file` captures the full non-default config
without needing a tftp server — and make additive changes only. Credentials
are vaulted at `fortigate/ana-gw-infra-ops-password`.
⚠ **Still outstanding: irv-ml1 (100.64.0.6) remains `relay "lax"`.** Same class
of problem at a different site; this change does not address it.