# sudo-io-archive — fleet sudo session recordings: archive, back up, clear Every `/etc/sudoers.d/infra-ops` carries `Defaults:infra-ops log_output`, so sudo records each infra-ops session (command line, timing, stdin/stdout/stderr) under `/var/log/sudo-io/XX/YY/ZZ`, with no retention. fv-ml1 had 16 G and esh-docker-vm 15 G by 2026-10-04. Prime: "those audits should be archived and backed up, then cleared. That should be a regular thing." ## What runs - **Weekly, Sunday 05:40 PT**: `sudo-io-archive.timer` → `sudo-io-archive.service` (nh3-dev **user** units, installed from this folder into `~/.config/systemd/user/`). A failure pages infra-ops via `althing-notify-failure@`. - **Collector** `sudo-io-archive` (nh3-dev): for each host in `hosts`, asks the host to pack every session older than 30 days (`SUDO_IO_DAYS`), pulls the archive, checks sha256 + gzip integrity, keeps it as `~/archive/sudo-io//--older-than-30d.tar.gz` (+ `.sha256`, mode 600), then tells the host to delete exactly what it archived. One ops-log line per host. - **Host side** `sudo-io-archive-host.sh`, streamed over ssh into `sudo -n bash -s`, phases `pack `, `commit `, `status`. Works under `/var/log/sudo-io-archive/` (outbox, list, cutoff). ## Safety properties - A session is eligible only when nothing in it changed after the cutoff, so running sessions are never touched. - The host deletes nothing until the collector has the archive and has matched its sha256. Removal is `tar --remove-files` over the same list the archive was built from (no `rm` on a variable; the list is refused if any entry is absolute or contains `..`). A failed run leaves the host's logs and outbox; the next run re-ships it. - The archives are covered by nh3-dev's backup: VM 102 is in nh3-pve's nightly vzdump to pbs-ana, mirrored to pbs-nh3. ## Run by hand ```bash services/sudo-io-archive/sudo-io-archive # all hosts services/sudo-io-archive/sudo-io-archive fv-ml1 # one host ssh infra-ops@ 'sudo -n bash -s -- status' < services/sudo-io-archive/sudo-io-archive-host.sh ``` Replay an archived session: untar it anywhere, then `sudoreplay -d ` (as root). ## Bulk data is not kept (Prime 2026-10-04: "we don't need to durably back up weights and rsyncs forever") Almost all of the volume was DATA piped through sudo, not commands: a 14.45 G recording of `sudo tar -C /tank/aimodels -cf - erp-tune-v6-nvfp4a16` on fv-ml1, a ~380 M `sudo rsync --server --sender` of the Home Assistant volume every night on esh-docker-vm, Worldtree volume rsyncs on corviduo-dev, a `docker save`. Three layers now keep it out: 1. **Not recorded.** Every infra-ops sudoers file carries a `BULK_DATA` exemption (rsync, tar, cat, dd, zstd, zcat, gzip, pv, `docker save|export|cp`): `Defaults!BULK_DATA !log_output`. sudo still logs who ran the command; it no longer records the bytes. `playbooks/sudoers-infra-ops-bulk-no-log-output.yaml` rolled it out to all 19 recording hosts on 2026-10-04 (controls on each: `sudo id` adds a session, `sudo tar --version` does not), and `playbooks/bootstrap-infra-ops-user.yaml` writes it for new hosts. 2. **Not archived.** `pack` leaves out any stdout/stdin/tty stream over 64 MiB (the session's log, log.json, timing and stderr are still archived) and `commit` deletes it with the rest. Proven with a labelled 70 MB test session (`TE/ST/OV` in the 2026-10-04 pfi-pteradactyl archive). 3. **Pruned once (2026-10-04).** The 36 existing bulk streams on hosts (≈31 G: fv-ml1 15 G, esh-docker-vm 15 G, corviduo-dev, nh3-dev, nh3-docker) were deleted with literal-path `rm`, keeping each session's metadata, and the three collected archives that held bulk streams were rebuilt without them (`*-bulk-pruned.tar.gz`; the store went from 4.8 G to 83 M). Lessons kept in the scripts: pull the outbox WITHOUT sudo (sudo ≥ 1.9.14 truncated piped output, and log_output re-recorded the archive); read the host list on fd 3 (ssh swallowed it from stdin); GNU tar reads `-X` NUL-separated under `--null` and only honours it BEFORE `-T`. Each of those was caught by a check, not by luck.