60 lines
4.1 KiB
Markdown
60 lines
4.1 KiB
Markdown
# sudo-io-archive — fleet sudo session recordings: archive, back up, clear
|
|
|
|
Every `/etc/sudoers.d/infra-ops` carries `Defaults:infra-ops log_output`, so sudo records each infra-ops session
|
|
(command line, timing, stdin/stdout/stderr) under `/var/log/sudo-io/XX/YY/ZZ`, with no retention. fv-ml1 had 16 G
|
|
and esh-docker-vm 15 G by 2026-10-04. Prime: "those audits should be archived and backed up, then cleared. That
|
|
should be a regular thing."
|
|
|
|
## What runs
|
|
|
|
- **Weekly, Sunday 05:40 PT**: `sudo-io-archive.timer` → `sudo-io-archive.service` (nh3-dev **user** units, installed
|
|
from this folder into `~/.config/systemd/user/`). A failure pages infra-ops via `althing-notify-failure@`.
|
|
- **Collector** `sudo-io-archive` (nh3-dev): for each host in `hosts`, asks the host to pack every session older than
|
|
30 days (`SUDO_IO_DAYS`), pulls the archive, checks sha256 + gzip integrity, keeps it as
|
|
`~/archive/sudo-io/<host>/<host>-<UTC stamp>-older-than-30d.tar.gz` (+ `.sha256`, mode 600), then tells the host
|
|
to delete exactly what it archived. One ops-log line per host.
|
|
- **Host side** `sudo-io-archive-host.sh`, streamed over ssh into `sudo -n bash -s`, phases `pack <days>`,
|
|
`commit <sha256>`, `status`. Works under `/var/log/sudo-io-archive/` (outbox, list, cutoff).
|
|
|
|
## Safety properties
|
|
|
|
- A session is eligible only when nothing in it changed after the cutoff, so running sessions are never touched.
|
|
- The host deletes nothing until the collector has the archive and has matched its sha256. Removal is
|
|
`tar --remove-files` over the same list the archive was built from (no `rm` on a variable; the list is refused if
|
|
any entry is absolute or contains `..`). A failed run leaves the host's logs and outbox; the next run re-ships it.
|
|
- The archives are covered by nh3-dev's backup: VM 102 is in nh3-pve's nightly vzdump to pbs-ana, mirrored to pbs-nh3.
|
|
|
|
## Run by hand
|
|
|
|
```bash
|
|
services/sudo-io-archive/sudo-io-archive # all hosts
|
|
services/sudo-io-archive/sudo-io-archive fv-ml1 # one host
|
|
ssh infra-ops@<ip> 'sudo -n bash -s -- status' < services/sudo-io-archive/sudo-io-archive-host.sh
|
|
```
|
|
|
|
Replay an archived session: untar it anywhere, then `sudoreplay -d <dir> <XX/YY/ZZ>` (as root).
|
|
|
|
## Bulk data is not kept (Prime 2026-10-04: "we don't need to durably back up weights and rsyncs forever")
|
|
|
|
Almost all of the volume was DATA piped through sudo, not commands: a 14.45 G recording of
|
|
`sudo tar -C /tank/aimodels -cf - erp-tune-v6-nvfp4a16` on fv-ml1, a ~380 M `sudo rsync --server --sender` of the
|
|
Home Assistant volume every night on esh-docker-vm, Worldtree volume rsyncs on corviduo-dev, a `docker save`. Three
|
|
layers now keep it out:
|
|
|
|
1. **Not recorded.** Every infra-ops sudoers file carries a `BULK_DATA` exemption (rsync, tar, cat, dd, zstd, zcat, gzip,
|
|
pv, `docker save|export|cp`): `Defaults!BULK_DATA !log_output`. sudo still logs who ran the command; it no longer
|
|
records the bytes. `playbooks/sudoers-infra-ops-bulk-no-log-output.yaml` rolled it out to all 19 recording hosts on
|
|
2026-10-04 (controls on each: `sudo id` adds a session, `sudo tar --version` does not), and
|
|
`playbooks/bootstrap-infra-ops-user.yaml` writes it for new hosts.
|
|
2. **Not archived.** `pack` leaves out any stdout/stdin/tty stream over 64 MiB (the session's log, log.json, timing
|
|
and stderr are still archived) and `commit` deletes it with the rest. Proven with a labelled 70 MB test session
|
|
(`TE/ST/OV` in the 2026-10-04 pfi-pteradactyl archive).
|
|
3. **Pruned once (2026-10-04).** The 36 existing bulk streams on hosts (≈31 G: fv-ml1 15 G, esh-docker-vm 15 G,
|
|
corviduo-dev, nh3-dev, nh3-docker) were deleted with literal-path `rm`, keeping each session's metadata, and the three
|
|
collected archives that held bulk streams were rebuilt without them (`*-bulk-pruned.tar.gz`; the store went from
|
|
4.8 G to 83 M).
|
|
|
|
Lessons kept in the scripts: pull the outbox WITHOUT sudo (sudo ≥ 1.9.14 truncated piped output, and log_output
|
|
re-recorded the archive); read the host list on fd 3 (ssh swallowed it from stdin); GNU tar reads `-X` NUL-separated
|
|
under `--null` and only honours it BEFORE `-T`. Each of those was caught by a check, not by luck.
|