fix(alerts): the installer skipped failed units — the exact ones it exists for
Two defects, both found by the tool failing to cover a unit that was already
broken. talk.service has been sitting in `failed` (exit 143) while its
containers keep serving 200 on :8092 -- precisely the "unit died, the thing
keeps serving, nobody knows" case this alarm was built for -- and the installer
had not hooked it.
1. SELECTION: --state=running skipped anything not already healthy. A unit that
is down at install time never got hooked, which inverts the tool's purpose.
Now the union of enabled unit-files and all loaded units.
2. PARSING, and this is the sharper one: systemd decorates a FAILED unit with a
leading "●", so `awk '{print $1}'` returned the bullet instead of the name,
and the sanitiser reduced it to an empty string and dropped the row. The
parser silently lost exactly the rows that matter. --plain suppresses the
decoration. Same shape as every other instrument error today -- it reported
cleanly while looking at the wrong thing.
Selection now 23 units. Deliberately INCLUDES the timer-driven oneshots
(dev-backup, ha-backup, fleet-tls-cert-check, headscale-ddns,
seat-inventory-drift, brokkr-landscape-scan, soong-ci-relay): a backup or a
cert check that fails silently is the same class, and all seven were clean at
install time so they are not a known noise source. EXCLUDES dbus, gpg-agent and
dirmngr as OS plumbing, the notifier's own template and instances, and
svos-failed-alarm -- hooking an alarm with an alarm is a loop with extra steps.
This commit is contained in:
@@ -12,7 +12,22 @@ BIN="${HOME}/.local/bin"
|
||||
# dbus.service is systemd's own plumbing, not a fleet service -- excluded.
|
||||
# althing-notify-failure@ is excluded because a notifier must never report
|
||||
# itself (see the template's comment).
|
||||
SKIP_RE='^(dbus|althing-notify-failure@).*'
|
||||
# althing-notify-failure@ covers both the bare template and every instance
|
||||
# (althing-notify-failure@booth.service.service) -- a notifier must never
|
||||
# report itself. dbus is systemd's own plumbing, not a fleet service.
|
||||
# EXCLUDED, and why each:
|
||||
# dbus / gpg-agent / dirmngr -- OS plumbing, not fleet services; they cycle
|
||||
# routinely and benignly.
|
||||
# althing-notify-failure@ -- both the template and every instance. A
|
||||
# notifier must never report itself.
|
||||
# svos-failed-alarm -- an alarm; hooking an alarm with an alarm is
|
||||
# a loop with extra steps. (Being retired.)
|
||||
# DELIBERATELY INCLUDED: timer-driven oneshots (dev-backup, ha-backup,
|
||||
# fleet-tls-cert-check, headscale-ddns, seat-inventory-drift,
|
||||
# brokkr-landscape-scan, soong-ci-relay). A backup or a cert check that fails
|
||||
# silently is the same "nobody knows" class this tool exists for, and all seven
|
||||
# were clean at install time so they are not a known noise source.
|
||||
SKIP_RE='^(dbus|gpg-agent|dirmngr|althing-notify-failure@|svos-failed-alarm)'
|
||||
|
||||
run() { if [ "$DRY" = "--dry-run" ]; then echo " would: $*"; else "$@"; fi; }
|
||||
|
||||
@@ -21,8 +36,28 @@ run install -Dm755 althing-notify-failure "${BIN}/althing-notify-failure"
|
||||
run install -Dm644 althing-notify-failure@.service "${UD}/althing-notify-failure@.service"
|
||||
|
||||
echo "== hooking fleet user units =="
|
||||
mapfile -t UNITS < <(systemctl --user list-units --type=service --state=running --no-pager \
|
||||
| awk '{print $1}' | grep '\.service$' | grep -Ev "$SKIP_RE")
|
||||
# ⚠ ENABLED, not running. The first cut selected --state=running, which skips
|
||||
# exactly the units that most need the alarm: one that is already down at
|
||||
# install time never gets hooked. Found on 2026-09-22 with talk.service sitting
|
||||
# in failed state (exit 143) while its containers kept serving on :8092 -- the
|
||||
# precise failure this tool exists for, missed because the installer would not
|
||||
# look at anything that was not already healthy.
|
||||
#
|
||||
# Union of enabled unit files and currently-loaded units, so a unit that is
|
||||
# enabled-but-down and a unit that is running-but-not-enabled both get hooked.
|
||||
#
|
||||
# ⚠ --plain IS LOAD-BEARING. systemd decorates a FAILED unit with a leading
|
||||
# "●", so `awk '{print $1}'` returns the bullet and not the unit name -- and a
|
||||
# sanitiser then reduces it to an empty string and drops the row. Measured on
|
||||
# 2026-09-22: talk.service sat in failed state and was silently skipped by this
|
||||
# very installer, i.e. the parser lost precisely the unit the tool exists for.
|
||||
# --plain suppresses the decoration so every row has the name in field 1.
|
||||
mapfile -t UNITS < <( { systemctl --user list-unit-files --type=service --state=enabled --plain --no-pager \
|
||||
| awk '{print $1}'
|
||||
systemctl --user list-units --type=service --all --plain --no-pager \
|
||||
| awk '{print $1}'
|
||||
} | grep '\.service$' | sort -u \
|
||||
| grep -Ev "$SKIP_RE" )
|
||||
for u in "${UNITS[@]}"; do
|
||||
d="${UD}/${u}.d"
|
||||
run mkdir -p "$d"
|
||||
|
||||
Reference in New Issue
Block a user