esh-pfi-infrastructure/services/sudo-io-archive/README.md

sudo-io-archive — fleet sudo session recordings: archive, back up, clear

Every /etc/sudoers.d/infra-ops carries Defaults:infra-ops log_output, so sudo records each infra-ops session (command line, timing, stdin/stdout/stderr) under /var/log/sudo-io/XX/YY/ZZ, with no retention. fv-ml1 had 16 G and esh-docker-vm 15 G by 2026-10-04. Prime: "those audits should be archived and backed up, then cleared. That should be a regular thing."

What runs

• Weekly, Sunday 05:40 PT: sudo-io-archive.timer → sudo-io-archive.service (nh3-dev user units, installed from this folder into ~/.config/systemd/user/). A failure pages infra-ops via althing-notify-failure@.
• Collector sudo-io-archive (nh3-dev): for each host in hosts, asks the host to pack every session older than 30 days (SUDO_IO_DAYS), pulls the archive, checks sha256 + gzip integrity, keeps it as ~/archive/sudo-io/<host>/<host>-<UTC stamp>-older-than-30d.tar.gz (+ .sha256, mode 600), then tells the host to delete exactly what it archived. One ops-log line per host.
• Host side sudo-io-archive-host.sh, streamed over ssh into sudo -n bash -s, phases pack <days>, commit <sha256>, status. Works under /var/log/sudo-io-archive/ (outbox, list, cutoff).

Safety properties

• A session is eligible only when nothing in it changed after the cutoff, so running sessions are never touched.
• The host deletes nothing until the collector has the archive and has matched its sha256. Removal is tar --remove-files over the same list the archive was built from (no rm on a variable; the list is refused if any entry is absolute or contains ..). A failed run leaves the host's logs and outbox; the next run re-ships it.
• The archives are covered by nh3-dev's backup: VM 102 is in nh3-pve's nightly vzdump to pbs-ana, mirrored to pbs-nh3.

Run by hand

services/sudo-io-archive/sudo-io-archive                  # all hosts
services/sudo-io-archive/sudo-io-archive fv-ml1           # one host
ssh infra-ops@<ip> 'sudo -n bash -s -- status' < services/sudo-io-archive/sudo-io-archive-host.sh

Replay an archived session: untar it anywhere, then sudoreplay -d <dir> <XX/YY/ZZ> (as root).

Bulk data is not kept (Prime 2026-10-04: "we don't need to durably back up weights and rsyncs forever")

Almost all of the volume was DATA piped through sudo, not commands: a 14.45 G recording of sudo tar -C /tank/aimodels -cf - erp-tune-v6-nvfp4a16 on fv-ml1, a ~380 M sudo rsync --server --sender of the Home Assistant volume every night on esh-docker-vm, Worldtree volume rsyncs on corviduo-dev, a docker save. Three layers now keep it out:

1. Not recorded. Every infra-ops sudoers file carries a BULK_DATA exemption (rsync, tar, cat, dd, zstd, zcat, gzip, pv, docker save|export|cp): Defaults!BULK_DATA !log_output. sudo still logs who ran the command; it no longer records the bytes. playbooks/sudoers-infra-ops-bulk-no-log-output.yaml rolled it out to all 19 recording hosts on 2026-10-04 (controls on each: sudo id adds a session, sudo tar --version does not), and playbooks/bootstrap-infra-ops-user.yaml writes it for new hosts.
2. Not archived. pack leaves out any stdout/stdin/tty stream over 64 MiB (the session's log, log.json, timing and stderr are still archived) and commit deletes it with the rest. Proven with a labelled 70 MB test session (TE/ST/OV in the 2026-10-04 pfi-pteradactyl archive).
3. Pruned once (2026-10-04). The 36 existing bulk streams on hosts (≈31 G: fv-ml1 15 G, esh-docker-vm 15 G, corviduo-dev, nh3-dev, nh3-docker) were deleted with literal-path rm, keeping each session's metadata, and the three collected archives that held bulk streams were rebuilt without them (*-bulk-pruned.tar.gz; the store went from 4.8 G to 83 M).

Lessons kept in the scripts: pull the outbox WITHOUT sudo (sudo ≥ 1.9.14 truncated piped output, and log_output re-recorded the archive); read the host list on fd 3 (ssh swallowed it from stdin); GNU tar reads -X NUL-separated under --null and only honours it BEFORE -T. Each of those was caught by a check, not by luck.