vh
148a5a34da
An infrastructure day with no training work, and the through-line is that every fault was invisible to monitoring. NH3↔Anaheim had been crossing a throttled DERP relay rather than a direct path for long enough to carry 78 GB; `.internal` DNS was failing roughly one lookup in ten from two independent causes; SearXNG had exactly one working general web engine. Nothing alarmed on any of it. All three surfaced because tts-dev had a 1545 ms voice-loop budget and chose to measure rather than adapt around the problem. Also landed: althing v3.6.3, which makes hyphenated search work for the first time on a fleet whose hostnames are nearly all hyphenated; FleetTools, a capability index autoloaded by Claude, Codex and Grok from one symlinked file; Miranda's Hermes plugin moved from a copy to a repo symlink; Worldtree's env.sh secrets vaulted. Six detail files. The in-flight section is rewritten and shrinks 142 lines to 64 — it opens on the one thing this session did NOT verify, lv-mccarthy's run outcome, which was left untouched and must not be assumed good. Two foot-guns recorded, both mine: the ESH egress experiment reverted on a diagnosis the rollback itself falsified, and `!ENV` in searxng settings, which has no constructor in that build and crash-looped the container ten times. No archival this run. 165 of 169 dated entries are under the 14-day guard and the remaining four all carry open deferred pointers, so the index stays over the soft cap at 480 lines — an over-cap file that keeps live decisions beats a scannable one that lost one.
4.6 KiB
[2026-09-18] NH3↔Anaheim was DERP-relayed, not direct — a UDP 41641 port-forward on ana-gw fixed it
Every NH3→Anaheim flow had been crossing Tailscale's LA DERP relay rather than a direct path, long enough to have carried 78 GB tx on the NH3 side alone. DERP is a deliberately throttled fallback — a reachability mechanism, not a data plane — so this imposed both a fixed round-trip penalty and a bandwidth ceiling on LiteLLM, Beszel, task-board, vor and the Henge alike.
How it surfaced, and how it did NOT
tts-dev reported ext-stt transcriptions 4–18× slower through the LiteLLM gateway than straight at the Parakeet host, with a ~1.9 s fixed floor plus a per-byte cost. They had already cleared GPU contention, the ASR itself and output quality with real controls.
No monitor caught it. Beszel had every host green. Nothing on this fleet checks whether a mesh path is direct.
The diagnosis, and two wrong causes published on the way
Their hypothesis was a dead first target in a LiteLLM fallback list. Checked: ext-stt
has exactly ONE deployment, no fallback, nothing to retry against. Langfuse looked like
a great candidate for a fixed floor — env vars still set, container gone — but it was
retired from the callback list 2026-06-20 and nothing calls it per-request.
The decisive arm was one neither of us had run: measure from ana-docker itself, so both arms share the identical onward path to FV.
ana-docker -> fv-ml1 DIRECT 0.289 / 0.218 / 0.273 / 0.220 s
ana-docker -> gateway -> fv-ml1 0.463 / 0.211 / 0.222 / 0.222 s
The gateway adds ~0. tts-dev measured 2618 ms for that clip; from the gateway's own host it is 220 ms. Their two arms differed in TWO things — whether LiteLLM was in the path AND which destination the packets crossed the network to reach — and the entire delta was attributed to the variable of interest.
nh3-dev -> fv-ml1 direct 0.269 / 0.219 / 0.216 s
nh3-dev -> ANA gateway 1.449 / 1.399 / 1.404 s
nh3-scale$ tailscale ping 100.64.0.3
pong via DERP(lax) in 507ms, 518, 521, 522, 373, 423, 464, 493
direct connection not established
Why hole-punching failed
ana-scale advertised 38.120.12.42:41641, but its netcheck mapped to :60798 — the
Anaheim NAT was not preserving the port — and PortMapping was empty, so no UPnP or
NAT-PMP was establishing one. MappingVariesByDestIP: false, so not symmetric NAT;
simply no reachable inbound endpoint. ESH↔Anaheim and NH3↔ESH were already direct;
only this pair failed.
The fix — four ADDITIVE objects on ana-gw (10.250.0.1)
⚠ This box runs central-nat enable, so a policy's dstaddr is the REAL internal
address and not the VIP. The existing wg-to-ana-wg VIP+policy pair is the house
template; writing what a FortiGate normally wants would have failed.
| object | value |
|---|---|
firewall address |
ana-scale-ip → 10.250.50.45/32, iface servers |
firewall service custom |
Tailscale-41641 → udp-portrange 41641 |
firewall vip |
tailscale-to-ana-scale → 38.120.12.42:41641 udp → 10.250.50.45:41641, extintf wan1 |
firewall policy id 75 |
wan1→servers, all→ana-scale-ip, Tailscale-41641, accept |
Measured before → after
| path | before | after |
|---|---|---|
tailscale ping nh3-scale→ana-scale |
373–522 ms via DERP(lax) | 6 ms direct |
| STT via the ANA gateway (96 kB clip) | 1.399–1.449 s | 0.237–0.270 s |
| Beszel HTTP nh3-dev→ana-docker | 0.94–1.29 s | 0.014–0.016 s |
tts-dev confirmed independently from talk's own vantage: 2618 → 249 ms and 4119 → 548 ms, converged with their direct arm.
Operational notes for next time
⚠ This edge has NO out-of-band access. Back up first — ssh infra-ops@10.250.0.1 'show' > file captures the full non-default config (11,320 lines) without needing a
tftp server; kept at ~/backups/ana-gw-config-BEFORE-<stamp>.txt. Credentials vaulted
at fortigate/ana-gw-infra-ops-password, reached with
sshpass -e ssh -o PubkeyAuthentication=no. Additive objects only — never edit an
existing policy or interface over a 327 ms link you cannot recover.
⚠ irv-ml1 (100.64.0.6) remains relay "lax". Same class, different site, not fixed
by this. Measured 24-25 ms on HTTP from nh3-dev, so it is not costing what Anaheim was.
I warned tts-dev their TTS stack was paying for it; they measured and I was wrong.
Documented in docs/pfi/headscale-mesh-plan.md; commit 5a9fad8. See
2026-09-18-fleet-dns-ring-and-ratelimit — the ANA AdGuard only became a viable
cross-site DNS backup because of this fix.