From ef70b2ffa139e7864c26f473b4a0e4c6bd11e762 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 22 Sep 2026 08:52:30 -0700 Subject: [PATCH] =?UTF-8?q?fix(alerts):=20the=20installer=20skipped=20fail?= =?UTF-8?q?ed=20units=20=E2=80=94=20the=20exact=20ones=20it=20exists=20for?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two defects, both found by the tool failing to cover a unit that was already broken. talk.service has been sitting in `failed` (exit 143) while its containers keep serving 200 on :8092 -- precisely the "unit died, the thing keeps serving, nobody knows" case this alarm was built for -- and the installer had not hooked it. 1. SELECTION: --state=running skipped anything not already healthy. A unit that is down at install time never got hooked, which inverts the tool's purpose. Now the union of enabled unit-files and all loaded units. 2. PARSING, and this is the sharper one: systemd decorates a FAILED unit with a leading "●", so `awk '{print $1}'` returned the bullet instead of the name, and the sanitiser reduced it to an empty string and dropped the row. The parser silently lost exactly the rows that matter. --plain suppresses the decoration. Same shape as every other instrument error today -- it reported cleanly while looking at the wrong thing. Selection now 23 units. Deliberately INCLUDES the timer-driven oneshots (dev-backup, ha-backup, fleet-tls-cert-check, headscale-ddns, seat-inventory-drift, brokkr-landscape-scan, soong-ci-relay): a backup or a cert check that fails silently is the same class, and all seven were clean at install time so they are not a known noise source. EXCLUDES dbus, gpg-agent and dirmngr as OS plumbing, the notifier's own template and instances, and svos-failed-alarm -- hooking an alarm with an alarm is a loop with extra steps. --- services/althing-notify-failure/install.sh | 41 ++++++++++++++++++++-- 1 file changed, 38 insertions(+), 3 deletions(-) diff --git a/services/althing-notify-failure/install.sh b/services/althing-notify-failure/install.sh index 771d6fb..4e7c0f1 100755 --- a/services/althing-notify-failure/install.sh +++ b/services/althing-notify-failure/install.sh @@ -12,7 +12,22 @@ BIN="${HOME}/.local/bin" # dbus.service is systemd's own plumbing, not a fleet service -- excluded. # althing-notify-failure@ is excluded because a notifier must never report # itself (see the template's comment). -SKIP_RE='^(dbus|althing-notify-failure@).*' +# althing-notify-failure@ covers both the bare template and every instance +# (althing-notify-failure@booth.service.service) -- a notifier must never +# report itself. dbus is systemd's own plumbing, not a fleet service. +# EXCLUDED, and why each: +# dbus / gpg-agent / dirmngr -- OS plumbing, not fleet services; they cycle +# routinely and benignly. +# althing-notify-failure@ -- both the template and every instance. A +# notifier must never report itself. +# svos-failed-alarm -- an alarm; hooking an alarm with an alarm is +# a loop with extra steps. (Being retired.) +# DELIBERATELY INCLUDED: timer-driven oneshots (dev-backup, ha-backup, +# fleet-tls-cert-check, headscale-ddns, seat-inventory-drift, +# brokkr-landscape-scan, soong-ci-relay). A backup or a cert check that fails +# silently is the same "nobody knows" class this tool exists for, and all seven +# were clean at install time so they are not a known noise source. +SKIP_RE='^(dbus|gpg-agent|dirmngr|althing-notify-failure@|svos-failed-alarm)' run() { if [ "$DRY" = "--dry-run" ]; then echo " would: $*"; else "$@"; fi; } @@ -21,8 +36,28 @@ run install -Dm755 althing-notify-failure "${BIN}/althing-notify-failure" run install -Dm644 althing-notify-failure@.service "${UD}/althing-notify-failure@.service" echo "== hooking fleet user units ==" -mapfile -t UNITS < <(systemctl --user list-units --type=service --state=running --no-pager \ - | awk '{print $1}' | grep '\.service$' | grep -Ev "$SKIP_RE") +# ⚠ ENABLED, not running. The first cut selected --state=running, which skips +# exactly the units that most need the alarm: one that is already down at +# install time never gets hooked. Found on 2026-09-22 with talk.service sitting +# in failed state (exit 143) while its containers kept serving on :8092 -- the +# precise failure this tool exists for, missed because the installer would not +# look at anything that was not already healthy. +# +# Union of enabled unit files and currently-loaded units, so a unit that is +# enabled-but-down and a unit that is running-but-not-enabled both get hooked. +# +# ⚠ --plain IS LOAD-BEARING. systemd decorates a FAILED unit with a leading +# "●", so `awk '{print $1}'` returns the bullet and not the unit name -- and a +# sanitiser then reduces it to an empty string and drops the row. Measured on +# 2026-09-22: talk.service sat in failed state and was silently skipped by this +# very installer, i.e. the parser lost precisely the unit the tool exists for. +# --plain suppresses the decoration so every row has the name in field 1. +mapfile -t UNITS < <( { systemctl --user list-unit-files --type=service --state=enabled --plain --no-pager \ + | awk '{print $1}' + systemctl --user list-units --type=service --all --plain --no-pager \ + | awk '{print $1}' + } | grep '\.service$' | sort -u \ + | grep -Ev "$SKIP_RE" ) for u in "${UNITS[@]}"; do d="${UD}/${u}.d" run mkdir -p "$d"