vh
fedd4b6d95
Three of the four failures in the week to 2026-09-23 (09-19, 09-21, 09-23) had one signature. The Cloudflare zone lookup returned an empty body at its 15s cap, a bare json.load crashed with tracebacks instead of a cause, and the script carried empty IDs on to a PATCH against zones//dns_records/. Cloudflare rejected it, so there was no DNS impact, but only by accident. - Cloudflare calls go through cf(): 3 announced tries, and a call counts only when the body says success:true. - success:true is not trusted as shape. pick() validates every body and prints only the fields asked for, or one line saying why not. It requires exactly one zone named phasefinal.com and exactly one A record named headscale.phasefinal.com, each with a non-empty id and content. Two A records are refused rather than half-updated, and an empty id can no longer shift the content into the id slot. - No write without both IDs. The run ends on a confirmation that the record now reads the new address. The previous final line was an echo whose exit status was always 0, even when the parse inside it failed. - Only a global unicast IPv4 is published (python ipaddress is_global). Loopback, RFC1918, link-local, CGNAT and documentation ranges are retried and then refused. - curl -q as the first argument ignores ~/.curlrc, so a verbose config can never log the bearer token. The vault CLI path is quoted. The empty data-array expansion is safe under set -u on bash < 4.4. Tests: services/headscale-ddns/test_headscale_ddns.py, 12 cases with curl, the vault CLI and sleep stubbed. Each failure case asserts the FATAL line's stated reason, so a run that died earlier for an unrelated cause cannot pass it. The documentation-range fixtures (203.0.113.x) were themselves rejected by the new public-IP guard: a free positive control. Live: a manual run and a unit run both printed "unchanged 70.230.226.88", Result=success. Cross-model bug-hunt (heid "Talus", Gróa + seat): all 8 findings folded.
headscale-ddns — keep headscale.phasefinal.com pointed at the NH3 WAN v4
A user timer on nh3-dev, every 10 minutes: read the WAN v4, PATCH the Cloudflare A record if it moved. It is how every mesh client finds the control plane, so a silent failure here eventually costs the mesh.
Installed at ~/.local/bin/headscale-ddns.sh with user units
headscale-ddns.{service,timer}. Tracked here since 2026-09-22 — it was
running untracked before that, so a fix to it lived on exactly one disk.
The 2026-09-22 failure, and what it exposed
The unit failed at 15:28 (exit 1) after succeeding all afternoon. Cause was
transient — icanhazip.com did not answer inside its 10s cap, so $IP came back
empty and the regex guard refused it. No DNS impact: the record already held the
right address and the next timer run succeeded.
What actually mattered was that the alarm carried no cause. Both failure
paths were || exit 1 in silence, so the failed-START notifier fired correctly
and said only "exit status 1". An alarm you cannot act on costs the same triage
as no alarm.
Two fixes, both verified by making them fail:
- Every exit path now says why — a missing vault key names the key; an empty token is distinguished from a failed read; a dead WAN lookup says so and adds "DNS left unchanged", which is the fact the reader needs.
- The WAN lookup retries 3× with announced attempts. One third-party blip should not page a human, and a silent retry would hide a degrading dependency.
2026-09-23: the Cloudflare half got the same treatment
Three of the four failures in the week to 09-23 (09-19, 09-21, 09-23) had one
signature: the Cloudflare zone lookup returned an empty body at its 15s cap.
The script parsed that with a bare json.load, crashed with tracebacks instead of
a cause, and carried empty IDs on to a PATCH against zones//dns_records/.
Cloudflare rejected it, so there was no DNS impact — fail-closed by accident.
Now, verified by test_headscale_ddns.py (curl, vault and sleep stubbed; 12
cases) and a live run:
- Every Cloudflare call retries 3×, announced, and only counts when the body
says
success: true. success: trueis not trusted as shape. Every body goes through one validating extractor (pick): exactly one zone namedphasefinal.com, exactly one A record namedheadscale.phasefinal.com, non-emptyidandcontent. Two A records are refused rather than half-updated, and an emptyidcan no longer shift the record's content into the id slot.- Nothing is written without both IDs, and the run ends on a CONFIRMATION
that the record now reads the new address, not on an
echothat always exits 0. - Only a global unicast IPv4 is published. Loopback, RFC1918, link-local, CGNAT 100.64/10 and documentation ranges are junk answers, retried and then refused.
curl -q(must be the first argument) ignores~/.curlrc, so a verbose config there can never write the bearer token into the journal.
Cross-model bug-hunt (heid round "Talus") found the shape-trust class; all eight findings folded.
⚠ The vault read is 17 of the script's 18 seconds
Measured 2026-09-22: secret get takes ~17s, everything else ~1s. It runs
every 10 minutes. That is not a fault — but it bounds any retry budget here, and
it is a fleet-wide cost worth knowing: svos-dev's own alarm unit carries the same
17-second note. Any script in a timer that reads the vault pays it.
Triage
systemctl --user status headscale-ddns.service
journalctl --user -u headscale-ddns.service -n 50 --no-pager
dig +short headscale.phasefinal.com @1.1.1.1 # the thing that actually matters
/home/lkraven/.local/bin/headscale-ddns.sh # safe to run by hand; idempotent
A failed unit clears itself on the next timer run; it does not need
reset-failed unless you want the state gone immediately.