feat(sudo-io-archive): weekly fleet archive of sudo session recordings (pack, verify, then clear); fv-ml1 HF_HOME on tank and root-pool cleanup
This commit is contained in:
@@ -0,0 +1,43 @@
|
||||
# sudo-io-archive — fleet sudo session recordings: archive, back up, clear
|
||||
|
||||
Every `/etc/sudoers.d/infra-ops` carries `Defaults:infra-ops log_output`, so sudo records each infra-ops session
|
||||
(command line, timing, stdin/stdout/stderr) under `/var/log/sudo-io/XX/YY/ZZ`, with no retention. fv-ml1 had 16 G
|
||||
and esh-docker-vm 15 G by 2026-10-04. Prime: "those audits should be archived and backed up, then cleared. That
|
||||
should be a regular thing."
|
||||
|
||||
## What runs
|
||||
|
||||
- **Weekly, Sunday 05:40 PT**: `sudo-io-archive.timer` → `sudo-io-archive.service` (nh3-dev **user** units, installed
|
||||
from this folder into `~/.config/systemd/user/`). A failure pages infra-ops via `althing-notify-failure@`.
|
||||
- **Collector** `sudo-io-archive` (nh3-dev): for each host in `hosts`, asks the host to pack every session older than
|
||||
30 days (`SUDO_IO_DAYS`), pulls the archive, checks sha256 + gzip integrity, keeps it as
|
||||
`~/archive/sudo-io/<host>/<host>-<UTC stamp>-older-than-30d.tar.gz` (+ `.sha256`, mode 600), then tells the host
|
||||
to delete exactly what it archived. One ops-log line per host.
|
||||
- **Host side** `sudo-io-archive-host.sh`, streamed over ssh into `sudo -n bash -s`, phases `pack <days>`,
|
||||
`commit <sha256>`, `status`. Works under `/var/log/sudo-io-archive/` (outbox, list, cutoff).
|
||||
|
||||
## Safety properties
|
||||
|
||||
- A session is eligible only when nothing in it changed after the cutoff, so running sessions are never touched.
|
||||
- The host deletes nothing until the collector has the archive and has matched its sha256. Removal is
|
||||
`tar --remove-files` over the same list the archive was built from (no `rm` on a variable; the list is refused if
|
||||
any entry is absolute or contains `..`). A failed run leaves the host's logs and outbox; the next run re-ships it.
|
||||
- The archives are covered by nh3-dev's backup: VM 102 is in nh3-pve's nightly vzdump to pbs-ana, mirrored to pbs-nh3.
|
||||
|
||||
## Run by hand
|
||||
|
||||
```bash
|
||||
services/sudo-io-archive/sudo-io-archive # all hosts
|
||||
services/sudo-io-archive/sudo-io-archive fv-ml1 # one host
|
||||
ssh infra-ops@<ip> 'sudo -n bash -s -- status' < services/sudo-io-archive/sudo-io-archive-host.sh
|
||||
```
|
||||
|
||||
Replay an archived session: untar it anywhere, then `sudoreplay -d <dir> <XX/YY/ZZ>` (as root).
|
||||
|
||||
## Where the volume comes from (2026-10-04)
|
||||
|
||||
Almost all of it is DATA piped through sudo, not commands: one fv-ml1 session is a 14.45 G recording of
|
||||
`sudo tar -C /tank/aimodels -cf - erp-tune-v6-nvfp4a16` (a model copy), and esh-docker-vm records a ~380 M
|
||||
`sudo rsync --server --sender … /var/lib/docker/volumes/homeassistant…` stream every night. Exempting bulk-transfer
|
||||
commands from output recording (`Defaults!/usr/bin/rsync,/usr/bin/tar !log_output` in the infra-ops sudoers) would keep
|
||||
the command line in the audit but stop recording the data; that is a policy change for Prime, not made here.
|
||||
@@ -0,0 +1,27 @@
|
||||
# name ssh target (infra-ops, NOPASSWD sudo, `Defaults:infra-ops log_output` in /etc/sudoers.d/infra-ops)
|
||||
# Hosts with no infra-ops sudo (pfi-postgres, ana-wg, nh3-nas, the SureFire hosts) record nothing and are not listed.
|
||||
pfi-pve infra-ops@10.250.250.31
|
||||
nh3-pve infra-ops@10.100.250.60
|
||||
nh3-pve-2 infra-ops@10.100.250.62
|
||||
esh-pve infra-ops@10.0.250.35
|
||||
esh-nas-pve infra-ops@10.0.50.55
|
||||
pbs-ana infra-ops@10.250.50.90
|
||||
pbs-nh3 infra-ops@10.100.50.90
|
||||
ana-docker infra-ops@10.250.50.70
|
||||
ana-nas infra-ops@10.250.50.50
|
||||
ana-filebot infra-ops@10.250.50.53
|
||||
corviduo-dev infra-ops@10.250.50.152
|
||||
pfi-pteradactyl infra-ops@10.250.50.55
|
||||
pfi-tacticalrmm infra-ops@10.250.50.57
|
||||
fv-ml1 infra-ops@10.251.50.54
|
||||
irv-ml1 infra-ops@irv-ml1.nh3.internal
|
||||
pfi-gx10 infra-ops@10.100.50.60
|
||||
nh3-dev infra-ops@10.100.10.50
|
||||
nh3-docker infra-ops@10.100.50.40
|
||||
nh3-extdev infra-ops@10.100.50.42
|
||||
nh3-ml1 infra-ops@10.100.50.80
|
||||
esh-docker-vm infra-ops@10.0.50.45
|
||||
esh-vm-db infra-ops@10.0.50.60
|
||||
esh-ml1 infra-ops@10.0.50.80
|
||||
esh-matter infra-ops@10.0.90.20
|
||||
vm-esh-nas infra-ops@10.0.50.154
|
||||
Executable
+60
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env bash
|
||||
# Collector for the fleet's sudo I/O logs. Runs on nh3-dev (weekly user timer, or by hand).
|
||||
# services/sudo-io-archive/sudo-io-archive # every host in ./hosts
|
||||
# services/sudo-io-archive/sudo-io-archive fv-ml1 pbs-ana # just these (names from ./hosts)
|
||||
# SUDO_IO_DAYS=30 (default) SUDO_IO_ARCHIVE_DIR=~/archive/sudo-io (default)
|
||||
#
|
||||
# Per host: the host packs sessions older than SUDO_IO_DAYS into one outbox archive (sudo-io-archive-host.sh pack),
|
||||
# this side pulls it, checks its sha256 and gzip integrity, keeps it under <dir>/<host>/, and only then tells the
|
||||
# host to delete exactly what it archived (… commit <sha>). A host that fails at any step keeps its logs and its
|
||||
# outbox; the next run re-ships the same outbox. Backed up because nh3-dev (VM 102 on nh3-pve) is in the nightly
|
||||
# vzdump to pbs-ana, which pbs-nh3 mirrors.
|
||||
#
|
||||
# Nothing here deletes anything locally (auto-memory feedback_no_rm_on_loop_variable): a failed pull leaves a
|
||||
# `.part` file behind, which the summary reports.
|
||||
set -uo pipefail
|
||||
umask 077 # the archives hold whatever the sessions printed; owner-only
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
days=${SUDO_IO_DAYS:-30}
|
||||
dest=${SUDO_IO_ARCHIVE_DIR:-$HOME/archive/sudo-io}
|
||||
hostscript="$here/sudo-io-archive-host.sh"
|
||||
ssh_opts=(-o BatchMode=yes -o ConnectTimeout=10 -o ServerAliveInterval=30)
|
||||
opslog="$here/../../scripts/ops-log"
|
||||
want=("$@")
|
||||
fail=0
|
||||
summary=()
|
||||
|
||||
# hosts come in on fd 3: ssh reads stdin, and on fd 0 the first pull swallowed the rest of the list (2026-10-04)
|
||||
while read -r -u 3 name target; do
|
||||
[[ -z ${name:-} || $name == \#* ]] && continue
|
||||
if [ ${#want[@]} -gt 0 ] && [[ ! " ${want[*]} " =~ " $name " ]]; then continue; fi
|
||||
r=$(ssh "${ssh_opts[@]}" "$target" "sudo -n bash -s -- pack $days" < "$hostscript" 2>&1 | grep '^RESULT' | tail -1)
|
||||
read -r _ kind sha sessions files _ <<<"${r:-RESULT ERROR}"
|
||||
case "$kind" in
|
||||
NOSUDOIO|NOTHING) summary+=("$name: ${kind,,}"); continue ;;
|
||||
PACKED) ;;
|
||||
*) summary+=("$name: PACK FAILED (${r:-no answer})"); fail=1; continue ;;
|
||||
esac
|
||||
install -d -m 700 "$dest/$name"
|
||||
f="$dest/$name/$name-$(date -u +%Y%m%dT%H%M%SZ)-older-than-${days}d.tar.gz"
|
||||
if ! ssh -n "${ssh_opts[@]}" "$target" 'cat /var/log/sudo-io-archive/outbox.tar.gz' > "$f.part"; then
|
||||
summary+=("$name: PULL FAILED, left $f.part"); fail=1; continue
|
||||
fi
|
||||
got=$(sha256sum "$f.part" | cut -d' ' -f1)
|
||||
if [ "$got" != "$sha" ] || ! gzip -t "$f.part" 2>/dev/null; then
|
||||
summary+=("$name: VERIFY FAILED (sha $got vs $sha), left $f.part"); fail=1; continue
|
||||
fi
|
||||
mv -n "$f.part" "$f"
|
||||
echo "$sha $(basename "$f")" > "$f.sha256"
|
||||
c=$(ssh "${ssh_opts[@]}" "$target" "sudo -n bash -s -- commit $sha" < "$hostscript" 2>&1 | grep '^RESULT' | tail -1)
|
||||
if [[ $c != "RESULT COMMITTED"* ]]; then
|
||||
summary+=("$name: archived to $f but COMMIT FAILED (${c:-no answer}); host keeps its logs"); fail=1; continue
|
||||
fi
|
||||
size=$(du -h "$f" | cut -f1)
|
||||
summary+=("$name: $sessions sessions / $files files -> $f ($size); host ${c#RESULT COMMITTED }")
|
||||
[ -x "$opslog" ] && "$opslog" record --host "$name" --action archive --target sudo-io \
|
||||
--detail "sudo I/O logs older than ${days}d: $sessions sessions archived to nh3-dev $f ($size, sha256 ${sha:0:12}) and cleared on the host" >/dev/null 2>&1
|
||||
done 3< "$here/hosts"
|
||||
|
||||
printf '%s\n' "${summary[@]}"
|
||||
exit $fail
|
||||
@@ -0,0 +1,91 @@
|
||||
#!/usr/bin/env bash
|
||||
# Host side of the sudo I/O log archiver. Runs as ROOT on a fleet host, fed over ssh by the collector:
|
||||
# ssh infra-ops@<host> 'sudo -n bash -s -- pack 30' < sudo-io-archive-host.sh
|
||||
# ssh infra-ops@<host> 'sudo -n bash -s -- commit <sha256>' < sudo-io-archive-host.sh
|
||||
#
|
||||
# Why this exists: every infra-ops sudoers file carries `Defaults:infra-ops log_output`, so sudo records each
|
||||
# session under /var/log/sudo-io/XX/YY/ZZ with no retention at all. fv-ml1 reached 16 G / 14,076 sessions by
|
||||
# 2026-10-04. Prime: "those audits should be archived and backed up, then cleared. That should be a regular thing."
|
||||
#
|
||||
# Safety rules this script keeps (eshpfi auto-memory feedback_no_rm_on_loop_variable):
|
||||
# - every destructive target is a LITERAL path; nothing is deleted through a shell variable;
|
||||
# - the session logs themselves are removed only by `tar --remove-files` reading the SAME list the shipped archive
|
||||
# was built from, and only after the collector has confirmed the archive's sha256 on its side;
|
||||
# - a session is eligible only when NO file in it is newer than the cutoff, so a still-running session is never
|
||||
# touched, and the list is refused if any entry is absolute or contains `..`.
|
||||
# Reading stdin is off limits here: bash -s is reading this very script from it.
|
||||
set -euo pipefail
|
||||
phase=${1:?usage: pack <days> | commit <sha256> | status}
|
||||
arg=${2:-}
|
||||
|
||||
if [ ! -d /var/log/sudo-io ]; then echo "RESULT NOSUDOIO"; exit 0; fi
|
||||
install -d -m 700 /var/log/sudo-io-archive
|
||||
|
||||
case "$phase" in
|
||||
pack)
|
||||
[[ $arg =~ ^[0-9]+$ ]] || { echo "pack needs a day count"; exit 2; }
|
||||
if [ -f /var/log/sudo-io-archive/outbox.tar.gz ]; then
|
||||
# a previous run packed but never committed: ship that same archive again rather than repacking
|
||||
chgrp infra-ops /var/log/sudo-io-archive /var/log/sudo-io-archive/outbox.tar.gz
|
||||
chmod 750 /var/log/sudo-io-archive
|
||||
chmod 640 /var/log/sudo-io-archive/outbox.tar.gz
|
||||
echo "RESULT PACKED $(cut -d' ' -f1 /var/log/sudo-io-archive/outbox.sha256) $(cat /var/log/sudo-io-archive/count) reused"
|
||||
exit 0
|
||||
fi
|
||||
touch -d "-${arg} days" /var/log/sudo-io-archive/cutoff
|
||||
: > /var/log/sudo-io-archive/list
|
||||
# session dirs are XX/YY/ZZ; include one only if nothing inside it changed after the cutoff
|
||||
while IFS= read -r -d '' rel; do
|
||||
if [ -z "$(find "/var/log/sudo-io/$rel" -newer /var/log/sudo-io-archive/cutoff -print -quit)" ]; then
|
||||
printf '%s\0' "$rel" >> /var/log/sudo-io-archive/list
|
||||
fi
|
||||
done < <(find /var/log/sudo-io -mindepth 3 -maxdepth 3 -type d -printf '%P\0')
|
||||
sessions=$(tr -cd '\0' < /var/log/sudo-io-archive/list | wc -c)
|
||||
if [ "$sessions" -eq 0 ]; then
|
||||
rm -f /var/log/sudo-io-archive/list
|
||||
rm -f /var/log/sudo-io-archive/cutoff
|
||||
echo "RESULT NOTHING"
|
||||
exit 0
|
||||
fi
|
||||
# sh -c so the session dirs land BEFORE -type (plain `xargs find -type f` appends them after the predicate)
|
||||
files=$(cd /var/log/sudo-io && xargs -0 -a /var/log/sudo-io-archive/list sh -c 'find "$@" -type f' sh | wc -l)
|
||||
# gzip, not zstd: it is on every Debian host, and sudo already gzips the stream files inside each session
|
||||
tar -C /var/log/sudo-io --null -T /var/log/sudo-io-archive/list -cf - | gzip -1 > /var/log/sudo-io-archive/outbox.tar.gz
|
||||
gzip -t /var/log/sudo-io-archive/outbox.tar.gz
|
||||
members=$(tar -tzf /var/log/sudo-io-archive/outbox.tar.gz | grep -vc '/$' || true)
|
||||
if [ "$members" -ne "$files" ]; then
|
||||
echo "archive holds $members files, the sessions hold $files: refusing"; exit 3
|
||||
fi
|
||||
echo "$sessions $files" > /var/log/sudo-io-archive/count
|
||||
sha256sum /var/log/sudo-io-archive/outbox.tar.gz > /var/log/sudo-io-archive/outbox.sha256
|
||||
# the collector reads the outbox as plain infra-ops, NOT through sudo: sudo's log_output would record the whole
|
||||
# archive as a new session, and sudo >= 1.9.14 (Debian 13) truncated the piped stream at exit (2026-10-04:
|
||||
# fv-ml1 and corviduo-dev pulls came back 52 K and 7 K short).
|
||||
chgrp infra-ops /var/log/sudo-io-archive /var/log/sudo-io-archive/outbox.tar.gz
|
||||
chmod 750 /var/log/sudo-io-archive
|
||||
chmod 640 /var/log/sudo-io-archive/outbox.tar.gz
|
||||
echo "RESULT PACKED $(cut -d' ' -f1 /var/log/sudo-io-archive/outbox.sha256) $sessions $files"
|
||||
;;
|
||||
commit)
|
||||
[[ $arg =~ ^[0-9a-f]{64}$ ]] || { echo "commit needs the collector's sha256"; exit 2; }
|
||||
[ -f /var/log/sudo-io-archive/outbox.tar.gz ] || { echo "no outbox to commit"; exit 3; }
|
||||
[ "$(cut -d' ' -f1 /var/log/sudo-io-archive/outbox.sha256)" = "$arg" ] || { echo "sha mismatch with the collector: refusing"; exit 3; }
|
||||
[ "$(sha256sum /var/log/sudo-io-archive/outbox.tar.gz | cut -d' ' -f1)" = "$arg" ] || { echo "outbox changed since pack: refusing"; exit 3; }
|
||||
if grep -z -q -E '(^/|(^|/)\.\.(/|$))' /var/log/sudo-io-archive/list; then echo "unsafe path in list: refusing"; exit 3; fi
|
||||
# removal = exactly the archived list, via tar itself (no rm on a variable). Output discarded; the archive that
|
||||
# matters is the one the collector already holds.
|
||||
# --ignore-failed-read: a retried commit (files already gone) must still finish its cleanup, not wedge forever
|
||||
tar -C /var/log/sudo-io --null -T /var/log/sudo-io-archive/list --remove-files --ignore-failed-read -cf - | cat > /dev/null
|
||||
find /var/log/sudo-io -mindepth 1 -maxdepth 2 -type d -empty -delete
|
||||
rm -f /var/log/sudo-io-archive/outbox.tar.gz
|
||||
rm -f /var/log/sudo-io-archive/outbox.sha256
|
||||
rm -f /var/log/sudo-io-archive/list
|
||||
rm -f /var/log/sudo-io-archive/cutoff
|
||||
rm -f /var/log/sudo-io-archive/count
|
||||
echo "RESULT COMMITTED $(du -sh /var/log/sudo-io | cut -f1) left"
|
||||
;;
|
||||
status)
|
||||
echo "RESULT STATUS $(du -sh /var/log/sudo-io | cut -f1) $(find /var/log/sudo-io -mindepth 3 -maxdepth 3 -type d | wc -l) sessions outbox=$([ -f /var/log/sudo-io-archive/outbox.tar.gz ] && echo yes || echo no)"
|
||||
;;
|
||||
*) echo "unknown phase $phase"; exit 2 ;;
|
||||
esac
|
||||
@@ -0,0 +1,8 @@
|
||||
[Unit]
|
||||
Description=Archive fleet sudo I/O logs older than 30 days to nh3-dev, then clear them on each host
|
||||
OnFailure=althing-notify-failure@%n.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=%h/development/eshpfi-management/services/sudo-io-archive/sudo-io-archive
|
||||
TimeoutStartSec=4h
|
||||
@@ -0,0 +1,9 @@
|
||||
[Unit]
|
||||
Description=Weekly fleet sudo I/O log archive (Sunday 05:40, after the night's backups, before the 08:00 freshness check)
|
||||
|
||||
[Timer]
|
||||
OnCalendar=Sun *-*-* 05:40:00
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
Reference in New Issue
Block a user