44 lines
2.8 KiB
Markdown
44 lines
2.8 KiB
Markdown
# sudo-io-archive — fleet sudo session recordings: archive, back up, clear
|
|
|
|
Every `/etc/sudoers.d/infra-ops` carries `Defaults:infra-ops log_output`, so sudo records each infra-ops session
|
|
(command line, timing, stdin/stdout/stderr) under `/var/log/sudo-io/XX/YY/ZZ`, with no retention. fv-ml1 had 16 G
|
|
and esh-docker-vm 15 G by 2026-10-04. Prime: "those audits should be archived and backed up, then cleared. That
|
|
should be a regular thing."
|
|
|
|
## What runs
|
|
|
|
- **Weekly, Sunday 05:40 PT**: `sudo-io-archive.timer` → `sudo-io-archive.service` (nh3-dev **user** units, installed
|
|
from this folder into `~/.config/systemd/user/`). A failure pages infra-ops via `althing-notify-failure@`.
|
|
- **Collector** `sudo-io-archive` (nh3-dev): for each host in `hosts`, asks the host to pack every session older than
|
|
30 days (`SUDO_IO_DAYS`), pulls the archive, checks sha256 + gzip integrity, keeps it as
|
|
`~/archive/sudo-io/<host>/<host>-<UTC stamp>-older-than-30d.tar.gz` (+ `.sha256`, mode 600), then tells the host
|
|
to delete exactly what it archived. One ops-log line per host.
|
|
- **Host side** `sudo-io-archive-host.sh`, streamed over ssh into `sudo -n bash -s`, phases `pack <days>`,
|
|
`commit <sha256>`, `status`. Works under `/var/log/sudo-io-archive/` (outbox, list, cutoff).
|
|
|
|
## Safety properties
|
|
|
|
- A session is eligible only when nothing in it changed after the cutoff, so running sessions are never touched.
|
|
- The host deletes nothing until the collector has the archive and has matched its sha256. Removal is
|
|
`tar --remove-files` over the same list the archive was built from (no `rm` on a variable; the list is refused if
|
|
any entry is absolute or contains `..`). A failed run leaves the host's logs and outbox; the next run re-ships it.
|
|
- The archives are covered by nh3-dev's backup: VM 102 is in nh3-pve's nightly vzdump to pbs-ana, mirrored to pbs-nh3.
|
|
|
|
## Run by hand
|
|
|
|
```bash
|
|
services/sudo-io-archive/sudo-io-archive # all hosts
|
|
services/sudo-io-archive/sudo-io-archive fv-ml1 # one host
|
|
ssh infra-ops@<ip> 'sudo -n bash -s -- status' < services/sudo-io-archive/sudo-io-archive-host.sh
|
|
```
|
|
|
|
Replay an archived session: untar it anywhere, then `sudoreplay -d <dir> <XX/YY/ZZ>` (as root).
|
|
|
|
## Where the volume comes from (2026-10-04)
|
|
|
|
Almost all of it is DATA piped through sudo, not commands: one fv-ml1 session is a 14.45 G recording of
|
|
`sudo tar -C /tank/aimodels -cf - erp-tune-v6-nvfp4a16` (a model copy), and esh-docker-vm records a ~380 M
|
|
`sudo rsync --server --sender … /var/lib/docker/volumes/homeassistant…` stream every night. Exempting bulk-transfer
|
|
commands from output recording (`Defaults!/usr/bin/rsync,/usr/bin/tar !log_output` in the infra-ops sudoers) would keep
|
|
the command line in the audit but stop recording the data; that is a policy change for Prime, not made here.
|