vh
30517fd603
The failed-START notifier built earlier today had its first REAL firing at
15:28: headscale-ddns.service exited 1 after succeeding all afternoon. The
detection worked. The alarm was also useless, and that is the finding.
Both failure paths exited 1 IN SILENCE, so the message said "exit status 1" and
nothing else. An alarm you cannot act on costs the same triage as no alarm at
all -- the notifier did its job and the subject script had no diagnostics for it
to carry.
Cause was transient and harmless: icanhazip.com did not answer inside its 10s
cap, so the IP came back empty and the regex guard refused it. No DNS impact --
the record already held the right address, verified against 1.1.1.1 before
touching anything, and the next timer run succeeded. Arithmetic confirms it:
~17s vault read + 10s curl timeout = 27s against the 28s the failing run took.
Fixed, both verified by making them fail:
- every exit path names its cause; a missing vault key names the key, an
EMPTY token is distinguished from a failed read, and a dead WAN lookup adds
"DNS left unchanged" because that is the fact the reader needs
- the WAN lookup retries 3x with ANNOUNCED attempts -- one third-party blip
should not page a human, and a silent retry would hide a degrading
dependency
⚠ ALSO: this script was not tracked anywhere. A fix to the thing every mesh
client resolves through lived on exactly one disk. Script, unit and timer are
in the repo now.
Measured and recorded: the vault read is 17 of the script's 18 seconds, every
10 minutes. Not a fault, but it bounds any retry budget and it is fleet-wide --
svos-dev's alarm unit carries the same 17-second note.
16 lines
635 B
Desktop File
16 lines
635 B
Desktop File
[Unit]
|
|
Description=Update headscale.phasefinal.com A record to NH3 WAN v4
|
|
[Service]
|
|
Type=oneshot
|
|
ExecStart=/home/lkraven/.local/bin/headscale-ddns.sh
|
|
|
|
# /home/lkraven/.config/systemd/user/headscale-ddns.service.d/10-onfailure-althing.conf
|
|
# Installed as <unit>.service.d/10-onfailure-althing.conf on every fleet user
|
|
# unit. A drop-in rather than an edit to the unit file so it is visible in
|
|
# `systemctl --user cat`, reversible by deleting one file, and survives the
|
|
# unit being reinstalled by its own deploy.
|
|
#
|
|
# %n is the failing unit's full name; the template receives it as %I.
|
|
[Unit]
|
|
OnFailure=althing-notify-failure@%n.service
|