feat(sudo-io-archive): weekly fleet archive of sudo session recordings (pack, verify, then clear); fv-ml1 HF_HOME on tank and root-pool cleanup

2026-10-04 22:48:05 -07:00
parent 626e0c7f94
commit 008bd685aa
8 changed files with 248 additions and 0 deletions
@@ -0,0 +1,43 @@
+# sudo-io-archive — fleet sudo session recordings: archive, back up, clear
+
+Every `/etc/sudoers.d/infra-ops` carries `Defaults:infra-ops log_output`, so sudo records each infra-ops session
+(command line, timing, stdin/stdout/stderr) under `/var/log/sudo-io/XX/YY/ZZ`, with no retention. fv-ml1 had 16 G
+and esh-docker-vm 15 G by 2026-10-04. Prime: "those audits should be archived and backed up, then cleared. That
+should be a regular thing."
+
+## What runs
+
+- **Weekly, Sunday 05:40 PT**: `sudo-io-archive.timer` → `sudo-io-archive.service` (nh3-dev **user** units, installed
+  from this folder into `~/.config/systemd/user/`). A failure pages infra-ops via `althing-notify-failure@`.
+- **Collector** `sudo-io-archive` (nh3-dev): for each host in `hosts`, asks the host to pack every session older than
+  30 days (`SUDO_IO_DAYS`), pulls the archive, checks sha256 + gzip integrity, keeps it as
+  `~/archive/sudo-io/<host>/<host>-<UTC stamp>-older-than-30d.tar.gz` (+ `.sha256`, mode 600), then tells the host
+  to delete exactly what it archived. One ops-log line per host.
+- **Host side** `sudo-io-archive-host.sh`, streamed over ssh into `sudo -n bash -s`, phases `pack <days>`,
+  `commit <sha256>`, `status`. Works under `/var/log/sudo-io-archive/` (outbox, list, cutoff).
+
+## Safety properties
+
+- A session is eligible only when nothing in it changed after the cutoff, so running sessions are never touched.
+- The host deletes nothing until the collector has the archive and has matched its sha256. Removal is
+  `tar --remove-files` over the same list the archive was built from (no `rm` on a variable; the list is refused if
+  any entry is absolute or contains `..`). A failed run leaves the host's logs and outbox; the next run re-ships it.
+- The archives are covered by nh3-dev's backup: VM 102 is in nh3-pve's nightly vzdump to pbs-ana, mirrored to pbs-nh3.
+
+## Run by hand
+
+```bash
+services/sudo-io-archive/sudo-io-archive                  # all hosts
+services/sudo-io-archive/sudo-io-archive fv-ml1           # one host
+ssh infra-ops@<ip> 'sudo -n bash -s -- status' < services/sudo-io-archive/sudo-io-archive-host.sh
+```
+
+Replay an archived session: untar it anywhere, then `sudoreplay -d <dir> <XX/YY/ZZ>` (as root).
+
+## Where the volume comes from (2026-10-04)
+
+Almost all of it is DATA piped through sudo, not commands: one fv-ml1 session is a 14.45 G recording of
+`sudo tar -C /tank/aimodels -cf - erp-tune-v6-nvfp4a16` (a model copy), and esh-docker-vm records a ~380 M
+`sudo rsync --server --sender … /var/lib/docker/volumes/homeassistant…` stream every night. Exempting bulk-transfer
+commands from output recording (`Defaults!/usr/bin/rsync,/usr/bin/tar !log_output` in the infra-ops sudoers) would keep
+the command line in the audit but stop recording the data; that is a policy change for Prime, not made here.