esh-pfi-infrastructure/services/sudo-io-archive/README.md

sudo-io-archive — fleet sudo session recordings: archive, back up, clear

Every /etc/sudoers.d/infra-ops carries Defaults:infra-ops log_output, so sudo records each infra-ops session (command line, timing, stdin/stdout/stderr) under /var/log/sudo-io/XX/YY/ZZ, with no retention. fv-ml1 had 16 G and esh-docker-vm 15 G by 2026-10-04. Prime: "those audits should be archived and backed up, then cleared. That should be a regular thing."

What runs

• Weekly, Sunday 05:40 PT: sudo-io-archive.timer → sudo-io-archive.service (nh3-dev user units, installed from this folder into ~/.config/systemd/user/). A failure pages infra-ops via althing-notify-failure@.
• Collector sudo-io-archive (nh3-dev): for each host in hosts, asks the host to pack every session older than 30 days (SUDO_IO_DAYS), pulls the archive, checks sha256 + gzip integrity, keeps it as ~/archive/sudo-io/<host>/<host>-<UTC stamp>-older-than-30d.tar.gz (+ .sha256, mode 600), then tells the host to delete exactly what it archived. One ops-log line per host.
• Host side sudo-io-archive-host.sh, streamed over ssh into sudo -n bash -s, phases pack <days>, commit <sha256>, status. Works under /var/log/sudo-io-archive/ (outbox, list, cutoff).

Safety properties

• A session is eligible only when nothing in it changed after the cutoff, so running sessions are never touched.
• The host deletes nothing until the collector has the archive and has matched its sha256. Removal is tar --remove-files over the same list the archive was built from (no rm on a variable; the list is refused if any entry is absolute or contains ..). A failed run leaves the host's logs and outbox; the next run re-ships it.
• The archives are covered by nh3-dev's backup: VM 102 is in nh3-pve's nightly vzdump to pbs-ana, mirrored to pbs-nh3.

Run by hand

services/sudo-io-archive/sudo-io-archive                  # all hosts
services/sudo-io-archive/sudo-io-archive fv-ml1           # one host
ssh infra-ops@<ip> 'sudo -n bash -s -- status' < services/sudo-io-archive/sudo-io-archive-host.sh

Replay an archived session: untar it anywhere, then sudoreplay -d <dir> <XX/YY/ZZ> (as root).

Where the volume comes from (2026-10-04)

Almost all of it is DATA piped through sudo, not commands: one fv-ml1 session is a 14.45 G recording of sudo tar -C /tank/aimodels -cf - erp-tune-v6-nvfp4a16 (a model copy), and esh-docker-vm records a ~380 M sudo rsync --server --sender … /var/lib/docker/volumes/homeassistant… stream every night. Exempting bulk-transfer commands from output recording (Defaults!/usr/bin/rsync,/usr/bin/tar !log_output in the infra-ops sudoers) would keep the command line in the audit but stop recording the data; that is a policy change for Prime, not made here.