vh
fedd4b6d95
Three of the four failures in the week to 2026-09-23 (09-19, 09-21, 09-23) had one signature. The Cloudflare zone lookup returned an empty body at its 15s cap, a bare json.load crashed with tracebacks instead of a cause, and the script carried empty IDs on to a PATCH against zones//dns_records/. Cloudflare rejected it, so there was no DNS impact, but only by accident. - Cloudflare calls go through cf(): 3 announced tries, and a call counts only when the body says success:true. - success:true is not trusted as shape. pick() validates every body and prints only the fields asked for, or one line saying why not. It requires exactly one zone named phasefinal.com and exactly one A record named headscale.phasefinal.com, each with a non-empty id and content. Two A records are refused rather than half-updated, and an empty id can no longer shift the content into the id slot. - No write without both IDs. The run ends on a confirmation that the record now reads the new address. The previous final line was an echo whose exit status was always 0, even when the parse inside it failed. - Only a global unicast IPv4 is published (python ipaddress is_global). Loopback, RFC1918, link-local, CGNAT and documentation ranges are retried and then refused. - curl -q as the first argument ignores ~/.curlrc, so a verbose config can never log the bearer token. The vault CLI path is quoted. The empty data-array expansion is safe under set -u on bash < 4.4. Tests: services/headscale-ddns/test_headscale_ddns.py, 12 cases with curl, the vault CLI and sleep stubbed. Each failure case asserts the FATAL line's stated reason, so a run that died earlier for an unrelated cause cannot pass it. The documentation-range fixtures (203.0.113.x) were themselves rejected by the new public-IP guard: a free positive control. Live: a manual run and a unit run both printed "unchanged 70.230.226.88", Result=success. Cross-model bug-hunt (heid "Talus", Gróa + seat): all 8 findings folded.
80 lines
3.8 KiB
Markdown
80 lines
3.8 KiB
Markdown
# headscale-ddns — keep `headscale.phasefinal.com` pointed at the NH3 WAN v4
|
||
|
||
A user timer on nh3-dev, every 10 minutes: read the WAN v4, PATCH the Cloudflare
|
||
A record if it moved. It is how every mesh client finds the control plane, so a
|
||
silent failure here eventually costs the mesh.
|
||
|
||
Installed at `~/.local/bin/headscale-ddns.sh` with user units
|
||
`headscale-ddns.{service,timer}`. **Tracked here since 2026-09-22** — it was
|
||
running untracked before that, so a fix to it lived on exactly one disk.
|
||
|
||
## The 2026-09-22 failure, and what it exposed
|
||
|
||
The unit failed at 15:28 (exit 1) after succeeding all afternoon. Cause was
|
||
transient — `icanhazip.com` did not answer inside its 10s cap, so `$IP` came back
|
||
empty and the regex guard refused it. No DNS impact: the record already held the
|
||
right address and the next timer run succeeded.
|
||
|
||
**What actually mattered was that the alarm carried no cause.** Both failure
|
||
paths were `|| exit 1` in silence, so the failed-START notifier fired correctly
|
||
and said only "exit status 1". An alarm you cannot act on costs the same triage
|
||
as no alarm.
|
||
|
||
Two fixes, both verified by making them fail:
|
||
|
||
- **Every exit path now says why** — a missing vault key names the key; an empty
|
||
token is distinguished from a failed read; a dead WAN lookup says so and adds
|
||
*"DNS left unchanged"*, which is the fact the reader needs.
|
||
- **The WAN lookup retries 3×** with announced attempts. One third-party blip
|
||
should not page a human, and a *silent* retry would hide a degrading
|
||
dependency.
|
||
|
||
## 2026-09-23: the Cloudflare half got the same treatment
|
||
|
||
Three of the four failures in the week to 09-23 (09-19, 09-21, 09-23) had one
|
||
signature: the **Cloudflare zone lookup returned an empty body at its 15s cap**.
|
||
The script parsed that with a bare `json.load`, crashed with tracebacks instead of
|
||
a cause, and carried empty IDs on to a PATCH against `zones//dns_records/`.
|
||
Cloudflare rejected it, so there was no DNS impact — fail-closed by accident.
|
||
|
||
Now, verified by `test_headscale_ddns.py` (curl, vault and sleep stubbed; 12
|
||
cases) and a live run:
|
||
|
||
- **Every Cloudflare call retries 3×**, announced, and only counts when the body
|
||
says `success: true`.
|
||
- **`success: true` is not trusted as shape.** Every body goes through one
|
||
validating extractor (`pick`): exactly one zone named `phasefinal.com`,
|
||
exactly one A record named `headscale.phasefinal.com`, non-empty `id` and
|
||
`content`. Two A records are refused rather than half-updated, and an empty
|
||
`id` can no longer shift the record's content into the id slot.
|
||
- **Nothing is written without both IDs**, and the run ends on a CONFIRMATION
|
||
that the record now reads the new address, not on an `echo` that always exits
|
||
0.
|
||
- **Only a global unicast IPv4 is published.** Loopback, RFC1918, link-local,
|
||
CGNAT 100.64/10 and documentation ranges are junk answers, retried and then
|
||
refused.
|
||
- `curl -q` (must be the first argument) ignores `~/.curlrc`, so a verbose config
|
||
there can never write the bearer token into the journal.
|
||
|
||
Cross-model bug-hunt (heid round "Talus") found the shape-trust class; all eight
|
||
findings folded.
|
||
|
||
## ⚠ The vault read is 17 of the script's 18 seconds
|
||
|
||
Measured 2026-09-22: `secret get` takes **~17s**, everything else ~1s. It runs
|
||
every 10 minutes. That is not a fault — but it bounds any retry budget here, and
|
||
it is a fleet-wide cost worth knowing: svos-dev's own alarm unit carries the same
|
||
17-second note. Any script in a timer that reads the vault pays it.
|
||
|
||
## Triage
|
||
|
||
```sh
|
||
systemctl --user status headscale-ddns.service
|
||
journalctl --user -u headscale-ddns.service -n 50 --no-pager
|
||
dig +short headscale.phasefinal.com @1.1.1.1 # the thing that actually matters
|
||
/home/lkraven/.local/bin/headscale-ddns.sh # safe to run by hand; idempotent
|
||
```
|
||
|
||
A failed unit clears itself on the next timer run; it does not need
|
||
`reset-failed` unless you want the state gone immediately.
|