# headscale-ddns — keep `headscale.phasefinal.com` pointed at the NH3 WAN v4

A user timer on nh3-dev, every 10 minutes: read the WAN v4, PATCH the Cloudflare
A record if it moved. It is how every mesh client finds the control plane, so a
silent failure here eventually costs the mesh.

Installed at `~/.local/bin/headscale-ddns.sh` with user units
`headscale-ddns.{service,timer}`. **Tracked here since 2026-09-22** — it was
running untracked before that, so a fix to it lived on exactly one disk.

## The 2026-09-22 failure, and what it exposed

The unit failed at 15:28 (exit 1) after succeeding all afternoon. Cause was
transient — `icanhazip.com` did not answer inside its 10s cap, so `$IP` came back
empty and the regex guard refused it. No DNS impact: the record already held the
right address and the next timer run succeeded.

**What actually mattered was that the alarm carried no cause.** Both failure
paths were `|| exit 1` in silence, so the failed-START notifier fired correctly
and said only "exit status 1". An alarm you cannot act on costs the same triage
as no alarm.

Two fixes, both verified by making them fail:

- **Every exit path now says why** — a missing vault key names the key; an empty
  token is distinguished from a failed read; a dead WAN lookup says so and adds
  *"DNS left unchanged"*, which is the fact the reader needs.
- **The WAN lookup retries 3×** with announced attempts. One third-party blip
  should not page a human, and a *silent* retry would hide a degrading
  dependency.

## 2026-09-23: the Cloudflare half got the same treatment

Three of the four failures in the week to 09-23 (09-19, 09-21, 09-23) had one
signature: the **Cloudflare zone lookup returned an empty body at its 15s cap**.
The script parsed that with a bare `json.load`, crashed with tracebacks instead of
a cause, and carried empty IDs on to a PATCH against `zones//dns_records/`.
Cloudflare rejected it, so there was no DNS impact — fail-closed by accident.

Now, verified by `test_headscale_ddns.py` (curl, vault and sleep stubbed; 12
cases) and a live run:

- **Every Cloudflare call retries 3×**, announced, and only counts when the body
  says `success: true`.
- **`success: true` is not trusted as shape.** Every body goes through one
  validating extractor (`pick`): exactly one zone named `phasefinal.com`,
  exactly one A record named `headscale.phasefinal.com`, non-empty `id` and
  `content`. Two A records are refused rather than half-updated, and an empty
  `id` can no longer shift the record's content into the id slot.
- **Nothing is written without both IDs**, and the run ends on a CONFIRMATION
  that the record now reads the new address, not on an `echo` that always exits
  0.
- **Only a global unicast IPv4 is published.** Loopback, RFC1918, link-local,
  CGNAT 100.64/10 and documentation ranges are junk answers, retried and then
  refused.
- `curl -q` (must be the first argument) ignores `~/.curlrc`, so a verbose config
  there can never write the bearer token into the journal.

Cross-model bug-hunt (heid round "Talus") found the shape-trust class; all eight
findings folded.

## ⚠ The vault read is 17 of the script's 18 seconds

Measured 2026-09-22: `secret get` takes **~17s**, everything else ~1s. It runs
every 10 minutes. That is not a fault — but it bounds any retry budget here, and
it is a fleet-wide cost worth knowing: svos-dev's own alarm unit carries the same
17-second note. Any script in a timer that reads the vault pays it.

## Triage

```sh
systemctl --user status headscale-ddns.service
journalctl --user -u headscale-ddns.service -n 50 --no-pager
dig +short headscale.phasefinal.com @1.1.1.1     # the thing that actually matters
/home/lkraven/.local/bin/headscale-ddns.sh       # safe to run by hand; idempotent
```

A failed unit clears itself on the next timer run; it does not need
`reset-failed` unless you want the state gone immediately.