memory: snapshot — three silent fleet faults found and fixed in one afternoon
An infrastructure day with no training work, and the through-line is that every fault was invisible to monitoring. NH3↔Anaheim had been crossing a throttled DERP relay rather than a direct path for long enough to carry 78 GB; `.internal` DNS was failing roughly one lookup in ten from two independent causes; SearXNG had exactly one working general web engine. Nothing alarmed on any of it. All three surfaced because tts-dev had a 1545 ms voice-loop budget and chose to measure rather than adapt around the problem. Also landed: althing v3.6.3, which makes hyphenated search work for the first time on a fleet whose hostnames are nearly all hyphenated; FleetTools, a capability index autoloaded by Claude, Codex and Grok from one symlinked file; Miranda's Hermes plugin moved from a copy to a repo symlink; Worldtree's env.sh secrets vaulted. Six detail files. The in-flight section is rewritten and shrinks 142 lines to 64 — it opens on the one thing this session did NOT verify, lv-mccarthy's run outcome, which was left untouched and must not be assumed good. Two foot-guns recorded, both mine: the ESH egress experiment reverted on a diagnosis the rollback itself falsified, and `!ENV` in searxng settings, which has no constructor in that build and crash-looped the container ten times. No archival this run. 165 of 169 dated entries are under the 14-day guard and the remaining four all carry open deferred pointers, so the index stays over the soft cap at 480 lines — an over-cap file that keeps live decisions beats a scannable one that lost one.
This commit is contained in:
@@ -0,0 +1,91 @@
|
||||
# [2026-09-18] NH3↔Anaheim was DERP-relayed, not direct — a UDP 41641 port-forward on ana-gw fixed it
|
||||
|
||||
Every NH3→Anaheim flow had been crossing Tailscale's LA DERP relay rather than a
|
||||
direct path, long enough to have carried **78 GB tx on the NH3 side alone**. DERP
|
||||
is a deliberately throttled fallback — a reachability mechanism, not a data plane —
|
||||
so this imposed both a fixed round-trip penalty and a bandwidth ceiling on LiteLLM,
|
||||
Beszel, task-board, vor and the Henge alike.
|
||||
|
||||
## How it surfaced, and how it did NOT
|
||||
|
||||
tts-dev reported ext-stt transcriptions 4–18× slower through the LiteLLM gateway
|
||||
than straight at the Parakeet host, with a ~1.9 s fixed floor plus a per-byte cost.
|
||||
They had already cleared GPU contention, the ASR itself and output quality with
|
||||
real controls.
|
||||
|
||||
**No monitor caught it.** Beszel had every host green. Nothing on this fleet checks
|
||||
whether a mesh path is direct.
|
||||
|
||||
## The diagnosis, and two wrong causes published on the way
|
||||
|
||||
Their hypothesis was a dead first target in a LiteLLM fallback list. Checked: `ext-stt`
|
||||
has exactly ONE deployment, no fallback, nothing to retry against. Langfuse looked like
|
||||
a great candidate for a fixed floor — env vars still set, container gone — but it was
|
||||
retired from the callback list 2026-06-20 and nothing calls it per-request.
|
||||
|
||||
The decisive arm was one neither of us had run: **measure from ana-docker itself**, so
|
||||
both arms share the identical onward path to FV.
|
||||
|
||||
ana-docker -> fv-ml1 DIRECT 0.289 / 0.218 / 0.273 / 0.220 s
|
||||
ana-docker -> gateway -> fv-ml1 0.463 / 0.211 / 0.222 / 0.222 s
|
||||
|
||||
The gateway adds ~0. tts-dev measured 2618 ms for that clip; from the gateway's own
|
||||
host it is 220 ms. Their two arms differed in TWO things — whether LiteLLM was in the
|
||||
path AND which destination the packets crossed the network to reach — and the entire
|
||||
delta was attributed to the variable of interest.
|
||||
|
||||
nh3-dev -> fv-ml1 direct 0.269 / 0.219 / 0.216 s
|
||||
nh3-dev -> ANA gateway 1.449 / 1.399 / 1.404 s
|
||||
|
||||
nh3-scale$ tailscale ping 100.64.0.3
|
||||
pong via DERP(lax) in 507ms, 518, 521, 522, 373, 423, 464, 493
|
||||
direct connection not established
|
||||
|
||||
## Why hole-punching failed
|
||||
|
||||
ana-scale advertised `38.120.12.42:41641`, but its netcheck mapped to `:60798` — the
|
||||
Anaheim NAT was not preserving the port — and `PortMapping` was empty, so no UPnP or
|
||||
NAT-PMP was establishing one. `MappingVariesByDestIP: false`, so not symmetric NAT;
|
||||
simply no reachable inbound endpoint. ESH↔Anaheim and NH3↔ESH were already direct;
|
||||
only this pair failed.
|
||||
|
||||
## The fix — four ADDITIVE objects on ana-gw (10.250.0.1)
|
||||
|
||||
⚠ **This box runs `central-nat enable`, so a policy's `dstaddr` is the REAL internal
|
||||
address and not the VIP.** The existing `wg-to-ana-wg` VIP+policy pair is the house
|
||||
template; writing what a FortiGate normally wants would have failed.
|
||||
|
||||
| object | value |
|
||||
|---|---|
|
||||
| `firewall address` | `ana-scale-ip` → 10.250.50.45/32, iface `servers` |
|
||||
| `firewall service custom` | `Tailscale-41641` → `udp-portrange 41641` |
|
||||
| `firewall vip` | `tailscale-to-ana-scale` → 38.120.12.42:41641 udp → 10.250.50.45:41641, extintf wan1 |
|
||||
| `firewall policy` id 75 | wan1→servers, all→`ana-scale-ip`, `Tailscale-41641`, accept |
|
||||
|
||||
## Measured before → after
|
||||
|
||||
| path | before | after |
|
||||
|---|---|---|
|
||||
| `tailscale ping` nh3-scale→ana-scale | 373–522 ms via DERP(lax) | **6 ms direct** |
|
||||
| STT via the ANA gateway (96 kB clip) | 1.399–1.449 s | **0.237–0.270 s** |
|
||||
| Beszel HTTP nh3-dev→ana-docker | 0.94–1.29 s | **0.014–0.016 s** |
|
||||
|
||||
tts-dev confirmed independently from talk's own vantage: 2618 → 249 ms and 4119 →
|
||||
548 ms, converged with their direct arm.
|
||||
|
||||
## Operational notes for next time
|
||||
|
||||
⚠ **This edge has NO out-of-band access.** Back up first — `ssh infra-ops@10.250.0.1
|
||||
'show' > file` captures the full non-default config (11,320 lines) without needing a
|
||||
tftp server; kept at `~/backups/ana-gw-config-BEFORE-<stamp>.txt`. Credentials vaulted
|
||||
at `fortigate/ana-gw-infra-ops-password`, reached with
|
||||
`sshpass -e ssh -o PubkeyAuthentication=no`. **Additive objects only** — never edit an
|
||||
existing policy or interface over a 327 ms link you cannot recover.
|
||||
|
||||
⚠ **irv-ml1 (100.64.0.6) remains `relay "lax"`.** Same class, different site, not fixed
|
||||
by this. Measured 24-25 ms on HTTP from nh3-dev, so it is not costing what Anaheim was.
|
||||
I warned tts-dev their TTS stack was paying for it; they measured and I was wrong.
|
||||
|
||||
Documented in `docs/pfi/headscale-mesh-plan.md`; commit `5a9fad8`. See
|
||||
[[2026-09-18-fleet-dns-ring-and-ratelimit]] — the ANA AdGuard only became a viable
|
||||
cross-site DNS backup because of this fix.
|
||||
Reference in New Issue
Block a user