feat(sudo-io): stop recording bulk data (sudoers BULK_DATA !log_output fleet-wide), drop >64 MiB streams from archives, prune existing bulk recordings

This commit is contained in:
vh
2026-10-04 22:55:49 -07:00
parent 008bd685aa
commit d9d1dd9325
6 changed files with 105 additions and 18 deletions
+22 -6
View File
@@ -34,10 +34,26 @@ ssh infra-ops@<ip> 'sudo -n bash -s -- status' < services/sudo-io-archive/sudo-i
Replay an archived session: untar it anywhere, then `sudoreplay -d <dir> <XX/YY/ZZ>` (as root).
## Where the volume comes from (2026-10-04)
## Bulk data is not kept (Prime 2026-10-04: "we don't need to durably back up weights and rsyncs forever")
Almost all of it is DATA piped through sudo, not commands: one fv-ml1 session is a 14.45 G recording of
`sudo tar -C /tank/aimodels -cf - erp-tune-v6-nvfp4a16` (a model copy), and esh-docker-vm records a ~380 M
`sudo rsync --server --sender … /var/lib/docker/volumes/homeassistant…` stream every night. Exempting bulk-transfer
commands from output recording (`Defaults!/usr/bin/rsync,/usr/bin/tar !log_output` in the infra-ops sudoers) would keep
the command line in the audit but stop recording the data; that is a policy change for Prime, not made here.
Almost all of the volume was DATA piped through sudo, not commands: a 14.45 G recording of
`sudo tar -C /tank/aimodels -cf - erp-tune-v6-nvfp4a16` on fv-ml1, a ~380 M `sudo rsync --server --sender` of the
Home Assistant volume every night on esh-docker-vm, Worldtree volume rsyncs on corviduo-dev, a `docker save`. Three
layers now keep it out:
1. **Not recorded.** Every infra-ops sudoers file carries a `BULK_DATA` exemption (rsync, tar, cat, dd, zstd, zcat, gzip,
pv, `docker save|export|cp`): `Defaults!BULK_DATA !log_output`. sudo still logs who ran the command; it no longer
records the bytes. `playbooks/sudoers-infra-ops-bulk-no-log-output.yaml` rolled it out to all 19 recording hosts on
2026-10-04 (controls on each: `sudo id` adds a session, `sudo tar --version` does not), and
`playbooks/bootstrap-infra-ops-user.yaml` writes it for new hosts.
2. **Not archived.** `pack` leaves out any stdout/stdin/tty stream over 64 MiB (the session's log, log.json, timing
and stderr are still archived) and `commit` deletes it with the rest. Proven with a labelled 70 MB test session
(`TE/ST/OV` in the 2026-10-04 pfi-pteradactyl archive).
3. **Pruned once (2026-10-04).** The 36 existing bulk streams on hosts (≈31 G: fv-ml1 15 G, esh-docker-vm 15 G,
corviduo-dev, nh3-dev, nh3-docker) were deleted with literal-path `rm`, keeping each session's metadata, and the three
collected archives that held bulk streams were rebuilt without them (`*-bulk-pruned.tar.gz`; the store went from
4.8 G to 83 M).
Lessons kept in the scripts: pull the outbox WITHOUT sudo (sudo ≥ 1.9.14 truncated piped output, and log_output
re-recorded the archive); read the host list on fd 3 (ssh swallowed it from stdin); GNU tar reads `-X` NUL-separated
under `--null` and only honours it BEFORE `-T`. Each of those was caught by a check, not by luck.