esh-pfi-infrastructure/persistent-memory.d/2026-08-19-fleet-internal-dns.md

[2026-08-19] Fleet .internal DNS — git-sourced, agent-managed, three resolvers

Operator: "with ipv6 i can't memorize the IP addresses anymore. need a way to keep track of local .internal dns names that can be agent managed and is lightweight." Built and live in one session; commit b8003c7.

Shape

dns/internal.yaml     source of truth — 38 hosts + 4 service aliases
scripts/dns-sync.py   reconciles AdGuard resolvers against it
stacks/adguard-ana/   the colo's resolver, which did not exist
dns/README.md         workflow, naming, the IPv6 caveat

Deliberately the same posture as deploy-stack.sh: the file is intent, the resolvers are derived state, you see a diff before anything changes. --dry-run / --yes / --site <s>. Verified idempotent — a second run prints nothing to do.

Naming is <host>.<site>.internal with sites ana / esh / nh3 (operator's call). .internal is ICANN-reserved for private use since 2024; .local is reserved for mDNS, which is why the pre-existing searxng.pfi.local was a standards collision that merely happened to work.

Every name is published to every resolver — the site label says where a host is, not which resolver knows about it.

The framing correction that mattered most

The ask reads as "I can't memorise v6 addresses", but the deeper problem is that v6 addresses are derived, not assigned, so they cannot reliably be written down once either. SLAAC gives EUI-64 (MAC-coupled) or privacy-extension (rotating) addresses, and UniFi has no v6 equivalent of a DHCP reservation — so a hand-maintained v6 table rots on its own.

⇒ The fix has two halves and only the second is DNS: (1) pin static v6 on server-class hosts, (2) then the name table is just a file. Surfaced to the operator before building.

Verified 2026-08-19: no fleet host has a global v6 address at all yet — ESH's /56 is live only on esh-cameras, NH3's LANs are back to ipv6_interface_type: none, the colo has none. So the v6: column ships EMPTY and correct, and the naming layer was built first rather than blocking on v6. Names established now need no renaming when addresses land.

Suggested convention when they do (awaiting operator): each server static at its site's /64 with low-order bits echoing the v4 host octet — esh-docker-vm at …::45 — so addresses are declarable and semi-memorable.

Two properties not to break

Authority is scoped to the ZONE, not the resolver. Only rewrites ending in .internal are managed. ESH's resolver turned out to carry three hand-made esteban.net rewrites (eshnas, brotherprinter, eshhome) — my first read of the config missed them, because an awk range on rewrites: matched an empty-looking block. A resolver-wide authoritative sync would have silently deleted all three on first run. Verified intact after sync.

Within .internal it IS authoritative — names added by hand in the AdGuard UI get deleted by the next sync. That is the point: one place to look.

The colo had no resolver at all

ESH and NH3 each ran AdGuard; ana-docker resolved straight against 1.1.1.1, so the colo had no way to answer for internal names. Closed with stacks/adguard-ana/.

⚠️ Its API is on 8053, not 8080 — :8080 and :3000 were already taken on that busy host. The port is therefore carried per-site in the yaml, not assumed by the script, so the odd one out cannot be forgotten.

⚠️ It ships with no blocklists, deliberately. The other two filter ads for human browsing; this one resolves for a rack of servers, where a blocklist false-positive breaks service-to-service calls at 3am for no upside.

First boot uses a seed config (conf/AdGuardHome.seed.yaml) copied into the conf volume before first start, so the container comes up configured instead of sitting in the setup wizard.

Credential — service account, not the operator's

Added a dedicated infra-ops AdGuard user to all three resolvers rather than asking for the lkraven password (per the standing migrate-off-operator- creds directive). Password vaulted at nh3-dev/adguard-infra-ops-password; lkraven untouched; pre-change configs backed up on each host as AdGuardHome.yaml.bak-preinfraops-*. Both existing resolvers kept answering across the restart.

Two landmines worth keeping:

• Go's bcrypt rejects htpasswd's $2y$ prefix. Same algorithm, different marker; golang.org/x/crypto/bcrypt accepts only $2a$/$2b$. Normalise the prefix, and self-verify the hash with htpasswd -vb BEFORE installing it on a live resolver.
• The vault appends a trailing newline on get. A password carrying a stray \n fails auth in a way that looks exactly like a wrong password. dns-sync.py strips it.

pfi.local migration — and the one that must NOT move

searxng.pfi.local → searxng.ana.internal, with the old Host() kept alongside in the Traefik rule so nothing breaks mid-migration; both return 200. Drop the fallback once the access log shows the old name unused.

matrix.pfi.local deliberately NOT migrated. A Matrix server_name is baked into every user ID, room ID and signing key, and federation identity derives from it — renaming it is not a DNS change, it is rebuilding the homeserver's identity and invalidating its history. The operator approved "migrate pfi.local" generally; this was surfaced as a deliberate exclusion rather than executed blindly.

Still open

Colo hosts still point at 1.1.1.1, so they do not yet use the new resolver — it only answers what asks it directly. Repointing a whole site's DNS is a bigger change than standing the service up, and is the operator's to schedule.

See also 2026-08-17-fleet-ipv6-mesh.