vh
e979ccb337
The daily backup-freshness check has been unable to raise an alert since the
2026-08-28 althing v3 cutover. It called althing-cli, which v3 DELETED rather
than deprecated. The check itself never stopped working: it detected three
stale backups every morning and told nobody, and the only trace was a WARN
line inside a unit that was already reporting `failed` for the stale backups
themselves. Three weeks, silent.
Four changes, because swapping the binary alone would have left it dead:
* althing-cli -> postbox.
* Add ALTHING_POST_OFFICE to the systemd user unit AND to the installer that
writes it. postbox has no default address by design and a user unit
inherits nothing from the interactive shell, so the binary swap on its own
would have failed with a different message. Fixing only the live unit
would have been undone by the next installer run; the two are now verified
to agree.
* Recipient infra-ops -> infra-hermes. This runs AS infra-ops, so the old
address mailed the alarm to itself — the mirror trap named in CLAUDE.md.
Day-to-day checks are infra-hermes's half of the split; he escalates.
* Split the exit codes. 1 now means "backups stale, someone was told";
2 means "the alert path itself failed". A broken alarm is a worse fault
than the thing it watches and must not be indistinguishable from it.
Adds --test-alert: a positive control that sends a real message through the
real path on demand. The wire was cut for three weeks precisely because
nothing ever exercised it in the healthy state, and an alarm whose success
path is never run is not known to work.
Verified: positive control delivered; missing-address and unreachable-post-
office both correctly exit 2; a real run through systemd delivered the alert
and exited 1.
43 lines
1.4 KiB
Bash
Executable File
43 lines
1.4 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# install-backup-freshness-timer.sh — install/refresh the daily backup-freshness
|
|
# alert as a systemd USER timer on nh3-dev (the only host with SSH to all backup
|
|
# stores + althing-cli). Idempotent; re-run after editing the wrapper/check.
|
|
# Requires linger (loginctl enable-linger lkraven) so it fires without a login.
|
|
set -euo pipefail
|
|
UNIT_DIR="$HOME/.config/systemd/user"
|
|
REPO=/home/lkraven/development/eshpfi-management
|
|
mkdir -p "$UNIT_DIR"
|
|
|
|
cat > "$UNIT_DIR/backup-freshness.service" <<EOF
|
|
[Unit]
|
|
Description=Fleet backup freshness check + althing alert
|
|
After=network-online.target
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
# Both are required: postbox has no default post-office address by design, and
|
|
# a systemd user unit inherits neither from the interactive shell. Omitting
|
|
# ALTHING_POST_OFFICE is why swapping althing-cli->postbox alone would have
|
|
# left this unit just as dead, with a different error message.
|
|
Environment=ALTHING_HANDLE=infra-ops
|
|
Environment=ALTHING_POST_OFFICE=http://10.100.50.40:8390
|
|
ExecStart=$REPO/scripts/backup-freshness-alert.sh
|
|
EOF
|
|
|
|
cat > "$UNIT_DIR/backup-freshness.timer" <<EOF
|
|
[Unit]
|
|
Description=Daily fleet backup freshness check (08:00)
|
|
|
|
[Timer]
|
|
OnCalendar=*-*-* 08:00:00
|
|
Persistent=true
|
|
|
|
[Install]
|
|
WantedBy=timers.target
|
|
EOF
|
|
|
|
systemctl --user daemon-reload
|
|
systemctl --user enable --now backup-freshness.timer
|
|
echo "installed. next run:"
|
|
systemctl --user list-timers backup-freshness.timer --all --no-pager
|