feat(sudo-io-archive): weekly fleet archive of sudo session recordings (pack, verify, then clear); fv-ml1 HF_HOME on tank and root-pool cleanup

persistent-memory.md

@@ -119,6 +119,8 @@ _As of 2026-10-04 ~0546 PT. The newest subsection is first; older subsections ca
 
 ### 2026-10-03/04: live now (as of 2026-10-04 ~0546 PT)
 
+- **sudo session recordings: archived weekly since 2026-10-04 (Prime: "archived and backed up, then cleared … a regular thing")**: `services/sudo-io-archive/` (nh3-dev user timer, Sun 05:40; sessions >30 d → `~/archive/sudo-io/<host>/`, sha256-verified, then removed on the host by `tar --remove-files` over the archived list; nh3-dev's vzdump to PBS backs them up). First pass took 4.8 G from 11 hosts. ⚠ The volume is DATA piped through sudo, not commands: a 14.45 G recording of `sudo tar` of a model dir on fv-ml1, and a ~380 M `sudo rsync` of the HA volume nightly on esh-docker-vm. Open: exempt bulk-transfer commands from `log_output` (Prime's call). Lessons in the scripts: pull the outbox WITHOUT sudo (sudo ≥1.9.14 truncated piped output and re-records it), read the host list on fd 3 (ssh ate stdin).
+- **fv-ml1 root pool cleaned (2026-10-04)**: unused vLLM images/containers + build cache (~93 G), user HF/uv/pip caches merged to tank and cleared (zroot 124 G → 194 G free); `HF_HOME=/tank/aimodels/huggingface` system-wide. Docker data-root still on zroot (move = outage window, Prime). → `servers/fv-ml1/README.md`
 - **pfi-ana-webhost DESTROYED 2026-10-04 ~1726 (Prime: retire, build fresh if ever needed)**: `qm destroy 110 --purge`; last image `pbs-ana:backup/vm/110/2026-10-04T10:28:18Z`. All FortiGate objects for it deleted (policy 18, VIPs, address, service, DHCP reservation; **38.120.12.45 now unmapped**), public + internal DNS records removed, inventory moved to `docs/retired/pfi-ana-webhost/`. ⭐ dragonfireacoustics.com = Cloudflare Pages in the CLIENT's account (source on nh3-dev `~/development/adhoc/dragonfireacoustics-site`, pushed 2026-10-04 to private gitea `vh/dragonfireacoustics-site`). **Supersedes the two 2026-09-17 dragonfire entries.**
 - **phasefinal.com on Cloudflare Pages since 2026-10-04 ~1720** (project `phasefinal`, our account `vh@phasefinal`): www + apex CNAME → `phasefinal.pages.dev`; nginx container on ana-docker retired. Privacy page hosting line rewritten (Prime-approved wording). `_headers` + `404.html` added. Apex → www 301 restored as a zone Redirect Rule. Tokens: Prime vaulted an account **token-creator** (`nh3-dev/.config/cloudflare/infra-ops-token-creator`); infra-ops minted **`infra-ops-pages`** with it (vault `…/infra-ops-pages-token`: Pages Write + all-zones DNS/redirect/cache). Deploy: `stacks/phasefinal-web/deploy-pages.sh` (proven with the new token).
 - **VM 101 PFI-ANA-DC (pfi-pve) and VM 104 nh3-laser (nh3-pve) RETIRED from running (Prime 2026-10-04 1634: "we aren't running the DC or the remote laser")**: 101 shut down cleanly, both `onboot 0`, both excluded from their nightly vzdump (last backups kept). That also ends the PVE 9 Secure Boot/VirtIO backup warnings. ⚠ If either is ever started again: enroll the Microsoft 2023 certs first (`qm enroll-efi-keys`, BitLocker check).

servers/fv-ml1/README.md

@@ -18,6 +18,14 @@ Primary AI inference host for PFI.
 - **GPUs:** **4x** NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (95.6 GB VRAM each = 382 GB total, cc 12.0 / sm_120, GPU 0-3) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores. ⚠ This line read "2x" until 2026-09-19; `nvidia-smi` reports four. Read the host, not the doc.
 - **PSUs:** four present (PS1-PS4, all `ok`). ⚠ See § Power — on a single circuit that redundancy does not protect against the failure most likely to happen.
 - **Storage:** ZFS `zroot` (434 GB root) + `tank` pool (raidz2, 8× NVMe, 8.6 TB at `/tank`) — drive inventory below
+- **Root-pool hygiene (2026-10-04, Prime):** `HF_HOME=/tank/aimodels/huggingface` system-wide (`/etc/environment` +
+  `/etc/profile.d/hf-home.sh`), the same tree the model containers mount as `/hf`; group `llm` (lkraven, llmuser,
+  infra-ops) writes it and every dir carries setgid so new files stay group `llm`. All user HF caches were merged
+  into it (`rsync --ignore-existing`, nothing lost) and cleared, as were the uv and pip caches: zroot went from
+  124 G free to 194 G. Left on purpose: lkraven's `models--Cloudflare--clef-flash` (18 G), in use by the
+  2026-10-04 bench; it is already on tank, so delete it once that bench is done. Docker (179 G) is still on zroot;
+  moving its data-root to tank needs a model-seat outage window (Prime's call). sudo session recordings are
+  archived weekly by `services/sudo-io-archive/`.
 - **OS:** Debian 13 (trixie), kernel 6.12.x
 - **Docker:** 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2
 

services/sudo-io-archive/README.md

@@ -0,0 +1,43 @@
+# sudo-io-archive — fleet sudo session recordings: archive, back up, clear
+
+Every `/etc/sudoers.d/infra-ops` carries `Defaults:infra-ops log_output`, so sudo records each infra-ops session
+(command line, timing, stdin/stdout/stderr) under `/var/log/sudo-io/XX/YY/ZZ`, with no retention. fv-ml1 had 16 G
+and esh-docker-vm 15 G by 2026-10-04. Prime: "those audits should be archived and backed up, then cleared. That
+should be a regular thing."
+
+## What runs
+
+- **Weekly, Sunday 05:40 PT**: `sudo-io-archive.timer` → `sudo-io-archive.service` (nh3-dev **user** units, installed
+  from this folder into `~/.config/systemd/user/`). A failure pages infra-ops via `althing-notify-failure@`.
+- **Collector** `sudo-io-archive` (nh3-dev): for each host in `hosts`, asks the host to pack every session older than
+  30 days (`SUDO_IO_DAYS`), pulls the archive, checks sha256 + gzip integrity, keeps it as
+  `~/archive/sudo-io/<host>/<host>-<UTC stamp>-older-than-30d.tar.gz` (+ `.sha256`, mode 600), then tells the host
+  to delete exactly what it archived. One ops-log line per host.
+- **Host side** `sudo-io-archive-host.sh`, streamed over ssh into `sudo -n bash -s`, phases `pack <days>`,
+  `commit <sha256>`, `status`. Works under `/var/log/sudo-io-archive/` (outbox, list, cutoff).
+
+## Safety properties
+
+- A session is eligible only when nothing in it changed after the cutoff, so running sessions are never touched.
+- The host deletes nothing until the collector has the archive and has matched its sha256. Removal is
+  `tar --remove-files` over the same list the archive was built from (no `rm` on a variable; the list is refused if
+  any entry is absolute or contains `..`). A failed run leaves the host's logs and outbox; the next run re-ships it.
+- The archives are covered by nh3-dev's backup: VM 102 is in nh3-pve's nightly vzdump to pbs-ana, mirrored to pbs-nh3.
+
+## Run by hand
+
+```bash
+services/sudo-io-archive/sudo-io-archive                  # all hosts
+services/sudo-io-archive/sudo-io-archive fv-ml1           # one host
+ssh infra-ops@<ip> 'sudo -n bash -s -- status' < services/sudo-io-archive/sudo-io-archive-host.sh
+```
+
+Replay an archived session: untar it anywhere, then `sudoreplay -d <dir> <XX/YY/ZZ>` (as root).
+
+## Where the volume comes from (2026-10-04)
+
+Almost all of it is DATA piped through sudo, not commands: one fv-ml1 session is a 14.45 G recording of
+`sudo tar -C /tank/aimodels -cf - erp-tune-v6-nvfp4a16` (a model copy), and esh-docker-vm records a ~380 M
+`sudo rsync --server --sender … /var/lib/docker/volumes/homeassistant…` stream every night. Exempting bulk-transfer
+commands from output recording (`Defaults!/usr/bin/rsync,/usr/bin/tar !log_output` in the infra-ops sudoers) would keep
+the command line in the audit but stop recording the data; that is a policy change for Prime, not made here.

services/sudo-io-archive/hosts

@@ -0,0 +1,27 @@
+# name          ssh target (infra-ops, NOPASSWD sudo, `Defaults:infra-ops log_output` in /etc/sudoers.d/infra-ops)
+# Hosts with no infra-ops sudo (pfi-postgres, ana-wg, nh3-nas, the SureFire hosts) record nothing and are not listed.
+pfi-pve         infra-ops@10.250.250.31
+nh3-pve         infra-ops@10.100.250.60
+nh3-pve-2       infra-ops@10.100.250.62
+esh-pve         infra-ops@10.0.250.35
+esh-nas-pve     infra-ops@10.0.50.55
+pbs-ana         infra-ops@10.250.50.90
+pbs-nh3         infra-ops@10.100.50.90
+ana-docker      infra-ops@10.250.50.70
+ana-nas         infra-ops@10.250.50.50
+ana-filebot     infra-ops@10.250.50.53
+corviduo-dev    infra-ops@10.250.50.152
+pfi-pteradactyl infra-ops@10.250.50.55
+pfi-tacticalrmm infra-ops@10.250.50.57
+fv-ml1          infra-ops@10.251.50.54
+irv-ml1         infra-ops@irv-ml1.nh3.internal
+pfi-gx10        infra-ops@10.100.50.60
+nh3-dev         infra-ops@10.100.10.50
+nh3-docker      infra-ops@10.100.50.40
+nh3-extdev      infra-ops@10.100.50.42
+nh3-ml1         infra-ops@10.100.50.80
+esh-docker-vm   infra-ops@10.0.50.45
+esh-vm-db       infra-ops@10.0.50.60
+esh-ml1         infra-ops@10.0.50.80
+esh-matter      infra-ops@10.0.90.20
+vm-esh-nas      infra-ops@10.0.50.154

services/sudo-io-archive/sudo-io-archive

@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+# Collector for the fleet's sudo I/O logs. Runs on nh3-dev (weekly user timer, or by hand).
+#   services/sudo-io-archive/sudo-io-archive                 # every host in ./hosts
+#   services/sudo-io-archive/sudo-io-archive fv-ml1 pbs-ana  # just these (names from ./hosts)
+#   SUDO_IO_DAYS=30 (default)   SUDO_IO_ARCHIVE_DIR=~/archive/sudo-io (default)
+#
+# Per host: the host packs sessions older than SUDO_IO_DAYS into one outbox archive (sudo-io-archive-host.sh pack),
+# this side pulls it, checks its sha256 and gzip integrity, keeps it under <dir>/<host>/, and only then tells the
+# host to delete exactly what it archived (… commit <sha>). A host that fails at any step keeps its logs and its
+# outbox; the next run re-ships the same outbox. Backed up because nh3-dev (VM 102 on nh3-pve) is in the nightly
+# vzdump to pbs-ana, which pbs-nh3 mirrors.
+#
+# Nothing here deletes anything locally (auto-memory feedback_no_rm_on_loop_variable): a failed pull leaves a
+# `.part` file behind, which the summary reports.
+set -uo pipefail
+umask 077   # the archives hold whatever the sessions printed; owner-only
+here=$(cd "$(dirname "$0")" && pwd)
+days=${SUDO_IO_DAYS:-30}
+dest=${SUDO_IO_ARCHIVE_DIR:-$HOME/archive/sudo-io}
+hostscript="$here/sudo-io-archive-host.sh"
+ssh_opts=(-o BatchMode=yes -o ConnectTimeout=10 -o ServerAliveInterval=30)
+opslog="$here/../../scripts/ops-log"
+want=("$@")
+fail=0
+summary=()
+
+# hosts come in on fd 3: ssh reads stdin, and on fd 0 the first pull swallowed the rest of the list (2026-10-04)
+while read -r -u 3 name target; do
+  [[ -z ${name:-} || $name == \#* ]] && continue
+  if [ ${#want[@]} -gt 0 ] && [[ ! " ${want[*]} " =~ " $name " ]]; then continue; fi
+  r=$(ssh "${ssh_opts[@]}" "$target" "sudo -n bash -s -- pack $days" < "$hostscript" 2>&1 | grep '^RESULT' | tail -1)
+  read -r _ kind sha sessions files _ <<<"${r:-RESULT ERROR}"
+  case "$kind" in
+    NOSUDOIO|NOTHING) summary+=("$name: ${kind,,}"); continue ;;
+    PACKED) ;;
+    *) summary+=("$name: PACK FAILED (${r:-no answer})"); fail=1; continue ;;
+  esac
+  install -d -m 700 "$dest/$name"
+  f="$dest/$name/$name-$(date -u +%Y%m%dT%H%M%SZ)-older-than-${days}d.tar.gz"
+  if ! ssh -n "${ssh_opts[@]}" "$target" 'cat /var/log/sudo-io-archive/outbox.tar.gz' > "$f.part"; then
+    summary+=("$name: PULL FAILED, left $f.part"); fail=1; continue
+  fi
+  got=$(sha256sum "$f.part" | cut -d' ' -f1)
+  if [ "$got" != "$sha" ] || ! gzip -t "$f.part" 2>/dev/null; then
+    summary+=("$name: VERIFY FAILED (sha $got vs $sha), left $f.part"); fail=1; continue
+  fi
+  mv -n "$f.part" "$f"
+  echo "$sha  $(basename "$f")" > "$f.sha256"
+  c=$(ssh "${ssh_opts[@]}" "$target" "sudo -n bash -s -- commit $sha" < "$hostscript" 2>&1 | grep '^RESULT' | tail -1)
+  if [[ $c != "RESULT COMMITTED"* ]]; then
+    summary+=("$name: archived to $f but COMMIT FAILED (${c:-no answer}); host keeps its logs"); fail=1; continue
+  fi
+  size=$(du -h "$f" | cut -f1)
+  summary+=("$name: $sessions sessions / $files files -> $f ($size); host ${c#RESULT COMMITTED }")
+  [ -x "$opslog" ] && "$opslog" record --host "$name" --action archive --target sudo-io \
+    --detail "sudo I/O logs older than ${days}d: $sessions sessions archived to nh3-dev $f ($size, sha256 ${sha:0:12}) and cleared on the host" >/dev/null 2>&1
+done 3< "$here/hosts"
+
+printf '%s\n' "${summary[@]}"
+exit $fail

services/sudo-io-archive/sudo-io-archive-host.sh

@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
+# Host side of the sudo I/O log archiver. Runs as ROOT on a fleet host, fed over ssh by the collector:
+#   ssh infra-ops@<host> 'sudo -n bash -s -- pack 30'        < sudo-io-archive-host.sh
+#   ssh infra-ops@<host> 'sudo -n bash -s -- commit <sha256>' < sudo-io-archive-host.sh
+#
+# Why this exists: every infra-ops sudoers file carries `Defaults:infra-ops log_output`, so sudo records each
+# session under /var/log/sudo-io/XX/YY/ZZ with no retention at all. fv-ml1 reached 16 G / 14,076 sessions by
+# 2026-10-04. Prime: "those audits should be archived and backed up, then cleared. That should be a regular thing."
+#
+# Safety rules this script keeps (eshpfi auto-memory feedback_no_rm_on_loop_variable):
+#   - every destructive target is a LITERAL path; nothing is deleted through a shell variable;
+#   - the session logs themselves are removed only by `tar --remove-files` reading the SAME list the shipped archive
+#     was built from, and only after the collector has confirmed the archive's sha256 on its side;
+#   - a session is eligible only when NO file in it is newer than the cutoff, so a still-running session is never
+#     touched, and the list is refused if any entry is absolute or contains `..`.
+# Reading stdin is off limits here: bash -s is reading this very script from it.
+set -euo pipefail
+phase=${1:?usage: pack <days> | commit <sha256> | status}
+arg=${2:-}
+
+if [ ! -d /var/log/sudo-io ]; then echo "RESULT NOSUDOIO"; exit 0; fi
+install -d -m 700 /var/log/sudo-io-archive
+
+case "$phase" in
+pack)
+  [[ $arg =~ ^[0-9]+$ ]] || { echo "pack needs a day count"; exit 2; }
+  if [ -f /var/log/sudo-io-archive/outbox.tar.gz ]; then
+    # a previous run packed but never committed: ship that same archive again rather than repacking
+    chgrp infra-ops /var/log/sudo-io-archive /var/log/sudo-io-archive/outbox.tar.gz
+    chmod 750 /var/log/sudo-io-archive
+    chmod 640 /var/log/sudo-io-archive/outbox.tar.gz
+    echo "RESULT PACKED $(cut -d' ' -f1 /var/log/sudo-io-archive/outbox.sha256) $(cat /var/log/sudo-io-archive/count) reused"
+    exit 0
+  fi
+  touch -d "-${arg} days" /var/log/sudo-io-archive/cutoff
+  : > /var/log/sudo-io-archive/list
+  # session dirs are XX/YY/ZZ; include one only if nothing inside it changed after the cutoff
+  while IFS= read -r -d '' rel; do
+    if [ -z "$(find "/var/log/sudo-io/$rel" -newer /var/log/sudo-io-archive/cutoff -print -quit)" ]; then
+      printf '%s\0' "$rel" >> /var/log/sudo-io-archive/list
+    fi
+  done < <(find /var/log/sudo-io -mindepth 3 -maxdepth 3 -type d -printf '%P\0')
+  sessions=$(tr -cd '\0' < /var/log/sudo-io-archive/list | wc -c)
+  if [ "$sessions" -eq 0 ]; then
+    rm -f /var/log/sudo-io-archive/list
+    rm -f /var/log/sudo-io-archive/cutoff
+    echo "RESULT NOTHING"
+    exit 0
+  fi
+  # sh -c so the session dirs land BEFORE -type (plain `xargs find -type f` appends them after the predicate)
+  files=$(cd /var/log/sudo-io && xargs -0 -a /var/log/sudo-io-archive/list sh -c 'find "$@" -type f' sh | wc -l)
+  # gzip, not zstd: it is on every Debian host, and sudo already gzips the stream files inside each session
+  tar -C /var/log/sudo-io --null -T /var/log/sudo-io-archive/list -cf - | gzip -1 > /var/log/sudo-io-archive/outbox.tar.gz
+  gzip -t /var/log/sudo-io-archive/outbox.tar.gz
+  members=$(tar -tzf /var/log/sudo-io-archive/outbox.tar.gz | grep -vc '/$' || true)
+  if [ "$members" -ne "$files" ]; then
+    echo "archive holds $members files, the sessions hold $files: refusing"; exit 3
+  fi
+  echo "$sessions $files" > /var/log/sudo-io-archive/count
+  sha256sum /var/log/sudo-io-archive/outbox.tar.gz > /var/log/sudo-io-archive/outbox.sha256
+  # the collector reads the outbox as plain infra-ops, NOT through sudo: sudo's log_output would record the whole
+  # archive as a new session, and sudo >= 1.9.14 (Debian 13) truncated the piped stream at exit (2026-10-04:
+  # fv-ml1 and corviduo-dev pulls came back 52 K and 7 K short).
+  chgrp infra-ops /var/log/sudo-io-archive /var/log/sudo-io-archive/outbox.tar.gz
+  chmod 750 /var/log/sudo-io-archive
+  chmod 640 /var/log/sudo-io-archive/outbox.tar.gz
+  echo "RESULT PACKED $(cut -d' ' -f1 /var/log/sudo-io-archive/outbox.sha256) $sessions $files"
+  ;;
+commit)
+  [[ $arg =~ ^[0-9a-f]{64}$ ]] || { echo "commit needs the collector's sha256"; exit 2; }
+  [ -f /var/log/sudo-io-archive/outbox.tar.gz ] || { echo "no outbox to commit"; exit 3; }
+  [ "$(cut -d' ' -f1 /var/log/sudo-io-archive/outbox.sha256)" = "$arg" ] || { echo "sha mismatch with the collector: refusing"; exit 3; }
+  [ "$(sha256sum /var/log/sudo-io-archive/outbox.tar.gz | cut -d' ' -f1)" = "$arg" ] || { echo "outbox changed since pack: refusing"; exit 3; }
+  if grep -z -q -E '(^/|(^|/)\.\.(/|$))' /var/log/sudo-io-archive/list; then echo "unsafe path in list: refusing"; exit 3; fi
+  # removal = exactly the archived list, via tar itself (no rm on a variable). Output discarded; the archive that
+  # matters is the one the collector already holds.
+  # --ignore-failed-read: a retried commit (files already gone) must still finish its cleanup, not wedge forever
+  tar -C /var/log/sudo-io --null -T /var/log/sudo-io-archive/list --remove-files --ignore-failed-read -cf - | cat > /dev/null
+  find /var/log/sudo-io -mindepth 1 -maxdepth 2 -type d -empty -delete
+  rm -f /var/log/sudo-io-archive/outbox.tar.gz
+  rm -f /var/log/sudo-io-archive/outbox.sha256
+  rm -f /var/log/sudo-io-archive/list
+  rm -f /var/log/sudo-io-archive/cutoff
+  rm -f /var/log/sudo-io-archive/count
+  echo "RESULT COMMITTED $(du -sh /var/log/sudo-io | cut -f1) left"
+  ;;
+status)
+  echo "RESULT STATUS $(du -sh /var/log/sudo-io | cut -f1) $(find /var/log/sudo-io -mindepth 3 -maxdepth 3 -type d | wc -l) sessions outbox=$([ -f /var/log/sudo-io-archive/outbox.tar.gz ] && echo yes || echo no)"
+  ;;
+*) echo "unknown phase $phase"; exit 2 ;;
+esac

services/sudo-io-archive/sudo-io-archive.service

@@ -0,0 +1,8 @@
+[Unit]
+Description=Archive fleet sudo I/O logs older than 30 days to nh3-dev, then clear them on each host
+OnFailure=althing-notify-failure@%n.service
+
+[Service]
+Type=oneshot
+ExecStart=%h/development/eshpfi-management/services/sudo-io-archive/sudo-io-archive
+TimeoutStartSec=4h

services/sudo-io-archive/sudo-io-archive.timer

@@ -0,0 +1,9 @@
+[Unit]
+Description=Weekly fleet sudo I/O log archive (Sunday 05:40, after the night's backups, before the 08:00 freshness check)
+
+[Timer]
+OnCalendar=Sun *-*-* 05:40:00
+Persistent=true
+
+[Install]
+WantedBy=timers.target