# [2026-09-18] `.internal` DNS was failing ~10% of lookups — two causes, both fleet-wide

Two independent faults, fixed in order. Together they were costing roughly one
`.internal` lookup in ten either a hard failure or a five-second stall, on every
DHCP client at every site.

## Fault 1 — a PUBLIC resolver was the fallback for a PRIVATE zone

DHCP handed out `10.100.50.40, 1.1.1.1` on both NH3 VLANs. When the internal
resolver missed a packet, the resolver waited out its timeout, fell through to
Cloudflare, and got an **authoritative NXDOMAIN** — so a transient miss became a
definitive "no such host" rather than a retry. Binary failure: instant, or five
seconds then `gaierror`.

**Anaheim was worse.** Its FortiGate handed clients *itself* as resolver and
forwarded to `1.1.1.1`/`1.0.0.1`, so **no ANA host could resolve `.internal` at
all** — `ana-docker`, which HOSTS the Anaheim AdGuard, had `1.1.1.1` in its own
`resolv.conf`.

⚠ **`/etc/resolv.conf` on these hosts is DHCP-managed** (dhclient on ens18). Hand
editing it survives exactly until the next lease renewal — tts-dev's original
framing was "fix nh3-dev's resolv.conf", which would have worked, been verified,
and then silently reverted. Change it at the source:

- **UniFi**: `dhcpd_dns_1` / `dhcpd_dns_2` per network object.
- **FortiGate**: `config system dhcp server`, `set dns-service specify` +
  `dns-server1` / `dns-server2`.
- Then `sudo dhclient -1 -v ens18` to pick it up **without releasing** the lease.

**Operator's ruling was broader than the proposal**: each site's backup resolver
should be *another site's* resolver. ESH already worked this way; the other two
were set, and the ring closes.

    ESH  primary 10.0.50.45    backup 10.100.50.40  (NH3)   [pre-existing]
    NH3  primary 10.100.50.40  backup 10.250.50.70  (ANA)   [changed]
    ANA  primary 10.250.50.70  backup 10.0.50.45    (ESH)   [changed]

All six members verified to answer the private zone. ⚠ The SureFire tenant DHCP
server (`sfsrv`) and the fortilink switch-management server were deliberately left
alone — tenant property.

**Result: hard failures 3/40 → 0/40. The 5 s stalls remained.**

## Fault 2 — THE ROOT CAUSE: AdGuard rate-limiting, identical on all three

    ratelimit: 20
    ratelimit_subnet_len_ipv4: 24
    ratelimit_whitelist: []

**20 queries per second shared across an entire /24 — per subnet, not per client.**
Every host on a VLAN draws from one bucket, so one busy container starves the rest,
and anything over the line is **silently dropped**, costing the client its full 5 s
resolver timeout.

Demonstrated rather than inferred — 60 concurrent queries at one resolver:

| | before | after |
|---|---|---|
| median | 5044 ms | 72 ms |
| timed out (>4 s) | **40 of 60** | **0 of 60** |

Set to `ratelimit: 0` on all three. These are LAN-only resolvers behind three
firewalls; that setting exists to blunt internet-side DNS amplification, which is
not this.

⚠ **Restarting AdGuard is required for a config change and briefly drops that site's
primary resolver — do the three ONE AT A TIME** so the cross-site ring always has a
live member. That the ring made this rollout safe is the point of having built it
first, an hour earlier.

## End-to-end, the original symptom

40 TCP connects to a service by name vs by IP; the by-IP arm is the control.

| | original | after ring only | after ratelimit 0 |
|---|---|---|---|
| by NAME p90 | 120 ms | 5017 ms | **13.5 ms** |
| by NAME >1 s | 3/40 | 5/40 | **0/40** |
| by NAME failed | 2/40 | 0/40 | **0/40** |
| by IP p90 (control) | 25 ms | — | 11.7 ms |

tts-dev confirmed from talk's own request path: name and IP **1.5 ms apart at the
median**. Resolution has stopped being a cost rather than become a smaller one.

## What this says about monitoring

Nothing on this fleet watches DNS success rate. The fleet's actual DNS monitor
today was a voice loop with a 1545 ms budget whose owner chose to measure instead
of adapting around the problem. Both faults were invisible to Beszel and Uptime
Kuma — every host was up, every service answered.

Related: [[2026-09-18-nh3-ana-derp-relay]] — the ANA AdGuard is only a sensible
cross-site backup because that fix took NH3→ANA from ~1.2 s to ~15 ms the same
afternoon. The two changes compose.