# sudo-io-archive — fleet sudo session recordings: archive, back up, clear

Every `/etc/sudoers.d/infra-ops` carries `Defaults:infra-ops log_output`, so sudo records each infra-ops session
(command line, timing, stdin/stdout/stderr) under `/var/log/sudo-io/XX/YY/ZZ`, with no retention. fv-ml1 had 16 G
and esh-docker-vm 15 G by 2026-10-04. Prime: "those audits should be archived and backed up, then cleared. That
should be a regular thing."

## What runs

- **Weekly, Sunday 05:40 PT**: `sudo-io-archive.timer` → `sudo-io-archive.service` (nh3-dev **user** units, installed
  from this folder into `~/.config/systemd/user/`). A failure pages infra-ops via `althing-notify-failure@`.
- **Collector** `sudo-io-archive` (nh3-dev): for each host in `hosts`, asks the host to pack every session older than
  30 days (`SUDO_IO_DAYS`), pulls the archive, checks sha256 + gzip integrity, keeps it as
  `~/archive/sudo-io/<host>/<host>-<UTC stamp>-older-than-30d.tar.gz` (+ `.sha256`, mode 600), then tells the host
  to delete exactly what it archived. One ops-log line per host.
- **Host side** `sudo-io-archive-host.sh`, streamed over ssh into `sudo -n bash -s`, phases `pack <days>`,
  `commit <sha256>`, `status`. Works under `/var/log/sudo-io-archive/` (outbox, list, cutoff).

## Safety properties

- A session is eligible only when nothing in it changed after the cutoff, so running sessions are never touched.
- The host deletes nothing until the collector has the archive and has matched its sha256. Removal is
  `tar --remove-files` over the same list the archive was built from (no `rm` on a variable; the list is refused if
  any entry is absolute or contains `..`). A failed run leaves the host's logs and outbox; the next run re-ships it.
- The archives are covered by nh3-dev's backup: VM 102 is in nh3-pve's nightly vzdump to pbs-ana, mirrored to pbs-nh3.

## Run by hand

```bash
services/sudo-io-archive/sudo-io-archive                  # all hosts
services/sudo-io-archive/sudo-io-archive fv-ml1           # one host
ssh infra-ops@<ip> 'sudo -n bash -s -- status' < services/sudo-io-archive/sudo-io-archive-host.sh
```

Replay an archived session: untar it anywhere, then `sudoreplay -d <dir> <XX/YY/ZZ>` (as root).

## Where the volume comes from (2026-10-04)

Almost all of it is DATA piped through sudo, not commands: one fv-ml1 session is a 14.45 G recording of
`sudo tar -C /tank/aimodels -cf - erp-tune-v6-nvfp4a16` (a model copy), and esh-docker-vm records a ~380 M
`sudo rsync --server --sender … /var/lib/docker/volumes/homeassistant…` stream every night. Exempting bulk-transfer
commands from output recording (`Defaults!/usr/bin/rsync,/usr/bin/tar !log_output` in the infra-ops sudoers) would keep
the command line in the audit but stop recording the data; that is a policy change for Prime, not made here.