fix(headscale-ddns): retry Cloudflare, validate every response, never write blind

Three of the four failures in the week to 2026-09-23 (09-19, 09-21, 09-23)
had one signature. The Cloudflare zone lookup returned an empty body at its
15s cap, a bare json.load crashed with tracebacks instead of a cause, and the
script carried empty IDs on to a PATCH against zones//dns_records/.
Cloudflare rejected it, so there was no DNS impact, but only by accident.

- Cloudflare calls go through cf(): 3 announced tries, and a call counts
  only when the body says success:true.
- success:true is not trusted as shape. pick() validates every body and
  prints only the fields asked for, or one line saying why not. It requires
  exactly one zone named phasefinal.com and exactly one A record named
  headscale.phasefinal.com, each with a non-empty id and content. Two A
  records are refused rather than half-updated, and an empty id can no
  longer shift the content into the id slot.
- No write without both IDs. The run ends on a confirmation that the record
  now reads the new address. The previous final line was an echo whose exit
  status was always 0, even when the parse inside it failed.
- Only a global unicast IPv4 is published (python ipaddress is_global).
  Loopback, RFC1918, link-local, CGNAT and documentation ranges are retried
  and then refused.
- curl -q as the first argument ignores ~/.curlrc, so a verbose config can
  never log the bearer token. The vault CLI path is quoted. The empty
  data-array expansion is safe under set -u on bash < 4.4.

Tests: services/headscale-ddns/test_headscale_ddns.py, 12 cases with curl,
the vault CLI and sleep stubbed. Each failure case asserts the FATAL line's
stated reason, so a run that died earlier for an unrelated cause cannot pass
it. The documentation-range fixtures (203.0.113.x) were themselves rejected
by the new public-IP guard: a free positive control.

Live: a manual run and a unit run both printed "unchanged 70.230.226.88",
Result=success. Cross-model bug-hunt (heid "Talus", Gróa + seat): all 8
findings folded.
This commit is contained in:
vh
2026-09-23 16:27:58 -07:00
parent 466f7aa4e6
commit fedd4b6d95
3 changed files with 305 additions and 9 deletions
+30
View File
@@ -29,6 +29,36 @@ Two fixes, both verified by making them fail:
should not page a human, and a *silent* retry would hide a degrading
dependency.
## 2026-09-23: the Cloudflare half got the same treatment
Three of the four failures in the week to 09-23 (09-19, 09-21, 09-23) had one
signature: the **Cloudflare zone lookup returned an empty body at its 15s cap**.
The script parsed that with a bare `json.load`, crashed with tracebacks instead of
a cause, and carried empty IDs on to a PATCH against `zones//dns_records/`.
Cloudflare rejected it, so there was no DNS impact — fail-closed by accident.
Now, verified by `test_headscale_ddns.py` (curl, vault and sleep stubbed; 12
cases) and a live run:
- **Every Cloudflare call retries 3×**, announced, and only counts when the body
says `success: true`.
- **`success: true` is not trusted as shape.** Every body goes through one
validating extractor (`pick`): exactly one zone named `phasefinal.com`,
exactly one A record named `headscale.phasefinal.com`, non-empty `id` and
`content`. Two A records are refused rather than half-updated, and an empty
`id` can no longer shift the record's content into the id slot.
- **Nothing is written without both IDs**, and the run ends on a CONFIRMATION
that the record now reads the new address, not on an `echo` that always exits
0.
- **Only a global unicast IPv4 is published.** Loopback, RFC1918, link-local,
CGNAT 100.64/10 and documentation ranges are junk answers, retried and then
refused.
- `curl -q` (must be the first argument) ignores `~/.curlrc`, so a verbose config
there can never write the bearer token into the journal.
Cross-model bug-hunt (heid round "Talus") found the shape-trust class; all eight
findings folded.
## ⚠ The vault read is 17 of the script's 18 seconds
Measured 2026-09-22: `secret get` takes **~17s**, everything else ~1s. It runs