# [2026-09-18] NH3↔Anaheim was DERP-relayed, not direct — a UDP 41641 port-forward on ana-gw fixed it

Every NH3→Anaheim flow had been crossing Tailscale's LA DERP relay rather than a
direct path, long enough to have carried **78 GB tx on the NH3 side alone**. DERP
is a deliberately throttled fallback — a reachability mechanism, not a data plane —
so this imposed both a fixed round-trip penalty and a bandwidth ceiling on LiteLLM,
Beszel, task-board, vor and the Henge alike.

## How it surfaced, and how it did NOT

tts-dev reported ext-stt transcriptions 4–18× slower through the LiteLLM gateway
than straight at the Parakeet host, with a ~1.9 s fixed floor plus a per-byte cost.
They had already cleared GPU contention, the ASR itself and output quality with
real controls.

**No monitor caught it.** Beszel had every host green. Nothing on this fleet checks
whether a mesh path is direct.

## The diagnosis, and two wrong causes published on the way

Their hypothesis was a dead first target in a LiteLLM fallback list. Checked: `ext-stt`
has exactly ONE deployment, no fallback, nothing to retry against. Langfuse looked like
a great candidate for a fixed floor — env vars still set, container gone — but it was
retired from the callback list 2026-06-20 and nothing calls it per-request.

The decisive arm was one neither of us had run: **measure from ana-docker itself**, so
both arms share the identical onward path to FV.

    ana-docker -> fv-ml1 DIRECT        0.289 / 0.218 / 0.273 / 0.220 s
    ana-docker -> gateway -> fv-ml1    0.463 / 0.211 / 0.222 / 0.222 s

The gateway adds ~0. tts-dev measured 2618 ms for that clip; from the gateway's own
host it is 220 ms. Their two arms differed in TWO things — whether LiteLLM was in the
path AND which destination the packets crossed the network to reach — and the entire
delta was attributed to the variable of interest.

    nh3-dev -> fv-ml1  direct        0.269 / 0.219 / 0.216 s
    nh3-dev -> ANA gateway           1.449 / 1.399 / 1.404 s

    nh3-scale$ tailscale ping 100.64.0.3
      pong via DERP(lax) in 507ms, 518, 521, 522, 373, 423, 464, 493
      direct connection not established

## Why hole-punching failed

ana-scale advertised `38.120.12.42:41641`, but its netcheck mapped to `:60798` — the
Anaheim NAT was not preserving the port — and `PortMapping` was empty, so no UPnP or
NAT-PMP was establishing one. `MappingVariesByDestIP: false`, so not symmetric NAT;
simply no reachable inbound endpoint. ESH↔Anaheim and NH3↔ESH were already direct;
only this pair failed.

## The fix — four ADDITIVE objects on ana-gw (10.250.0.1)

⚠ **This box runs `central-nat enable`, so a policy's `dstaddr` is the REAL internal
address and not the VIP.** The existing `wg-to-ana-wg` VIP+policy pair is the house
template; writing what a FortiGate normally wants would have failed.

| object | value |
|---|---|
| `firewall address` | `ana-scale-ip` → 10.250.50.45/32, iface `servers` |
| `firewall service custom` | `Tailscale-41641` → `udp-portrange 41641` |
| `firewall vip` | `tailscale-to-ana-scale` → 38.120.12.42:41641 udp → 10.250.50.45:41641, extintf wan1 |
| `firewall policy` id 75 | wan1→servers, all→`ana-scale-ip`, `Tailscale-41641`, accept |

## Measured before → after

| path | before | after |
|---|---|---|
| `tailscale ping` nh3-scale→ana-scale | 373–522 ms via DERP(lax) | **6 ms direct** |
| STT via the ANA gateway (96 kB clip) | 1.399–1.449 s | **0.237–0.270 s** |
| Beszel HTTP nh3-dev→ana-docker | 0.94–1.29 s | **0.014–0.016 s** |

tts-dev confirmed independently from talk's own vantage: 2618 → 249 ms and 4119 →
548 ms, converged with their direct arm.

## Operational notes for next time

⚠ **This edge has NO out-of-band access.** Back up first — `ssh infra-ops@10.250.0.1
'show' > file` captures the full non-default config (11,320 lines) without needing a
tftp server; kept at `~/backups/ana-gw-config-BEFORE-<stamp>.txt`. Credentials vaulted
at `fortigate/ana-gw-infra-ops-password`, reached with
`sshpass -e ssh -o PubkeyAuthentication=no`. **Additive objects only** — never edit an
existing policy or interface over a 327 ms link you cannot recover.

⚠ **irv-ml1 (100.64.0.6) remains `relay "lax"`.** Same class, different site, not fixed
by this. Measured 24-25 ms on HTTP from nh3-dev, so it is not costing what Anaheim was.
I warned tts-dev their TTS stack was paying for it; they measured and I was wrong.

Documented in `docs/pfi/headscale-mesh-plan.md`; commit `5a9fad8`. See
[[2026-09-18-fleet-dns-ring-and-ratelimit]] — the ANA AdGuard only became a viable
cross-site DNS backup because of this fix.