From 008bd685aa55027890491532fc7f695ff975d856 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 4 Oct 2026 22:48:05 -0700 Subject: [PATCH] feat(sudo-io-archive): weekly fleet archive of sudo session recordings (pack, verify, then clear); fv-ml1 HF_HOME on tank and root-pool cleanup --- persistent-memory.md | 2 + servers/fv-ml1/README.md | 8 ++ services/sudo-io-archive/README.md | 43 +++++++++ services/sudo-io-archive/hosts | 27 ++++++ services/sudo-io-archive/sudo-io-archive | 60 ++++++++++++ .../sudo-io-archive/sudo-io-archive-host.sh | 91 +++++++++++++++++++ .../sudo-io-archive/sudo-io-archive.service | 8 ++ .../sudo-io-archive/sudo-io-archive.timer | 9 ++ 8 files changed, 248 insertions(+) create mode 100644 services/sudo-io-archive/README.md create mode 100644 services/sudo-io-archive/hosts create mode 100755 services/sudo-io-archive/sudo-io-archive create mode 100644 services/sudo-io-archive/sudo-io-archive-host.sh create mode 100644 services/sudo-io-archive/sudo-io-archive.service create mode 100644 services/sudo-io-archive/sudo-io-archive.timer diff --git a/persistent-memory.md b/persistent-memory.md index 1b71c10..b91cee8 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -119,6 +119,8 @@ _As of 2026-10-04 ~0546 PT. The newest subsection is first; older subsections ca ### 2026-10-03/04: live now (as of 2026-10-04 ~0546 PT) +- **sudo session recordings: archived weekly since 2026-10-04 (Prime: "archived and backed up, then cleared … a regular thing")**: `services/sudo-io-archive/` (nh3-dev user timer, Sun 05:40; sessions >30 d → `~/archive/sudo-io//`, sha256-verified, then removed on the host by `tar --remove-files` over the archived list; nh3-dev's vzdump to PBS backs them up). First pass took 4.8 G from 11 hosts. ⚠ The volume is DATA piped through sudo, not commands: a 14.45 G recording of `sudo tar` of a model dir on fv-ml1, and a ~380 M `sudo rsync` of the HA volume nightly on esh-docker-vm. Open: exempt bulk-transfer commands from `log_output` (Prime's call). Lessons in the scripts: pull the outbox WITHOUT sudo (sudo ≥1.9.14 truncated piped output and re-records it), read the host list on fd 3 (ssh ate stdin). +- **fv-ml1 root pool cleaned (2026-10-04)**: unused vLLM images/containers + build cache (~93 G), user HF/uv/pip caches merged to tank and cleared (zroot 124 G → 194 G free); `HF_HOME=/tank/aimodels/huggingface` system-wide. Docker data-root still on zroot (move = outage window, Prime). → `servers/fv-ml1/README.md` - **pfi-ana-webhost DESTROYED 2026-10-04 ~1726 (Prime: retire, build fresh if ever needed)**: `qm destroy 110 --purge`; last image `pbs-ana:backup/vm/110/2026-10-04T10:28:18Z`. All FortiGate objects for it deleted (policy 18, VIPs, address, service, DHCP reservation; **38.120.12.45 now unmapped**), public + internal DNS records removed, inventory moved to `docs/retired/pfi-ana-webhost/`. ⭐ dragonfireacoustics.com = Cloudflare Pages in the CLIENT's account (source on nh3-dev `~/development/adhoc/dragonfireacoustics-site`, pushed 2026-10-04 to private gitea `vh/dragonfireacoustics-site`). **Supersedes the two 2026-09-17 dragonfire entries.** - **phasefinal.com on Cloudflare Pages since 2026-10-04 ~1720** (project `phasefinal`, our account `vh@phasefinal`): www + apex CNAME → `phasefinal.pages.dev`; nginx container on ana-docker retired. Privacy page hosting line rewritten (Prime-approved wording). `_headers` + `404.html` added. Apex → www 301 restored as a zone Redirect Rule. Tokens: Prime vaulted an account **token-creator** (`nh3-dev/.config/cloudflare/infra-ops-token-creator`); infra-ops minted **`infra-ops-pages`** with it (vault `…/infra-ops-pages-token`: Pages Write + all-zones DNS/redirect/cache). Deploy: `stacks/phasefinal-web/deploy-pages.sh` (proven with the new token). - **VM 101 PFI-ANA-DC (pfi-pve) and VM 104 nh3-laser (nh3-pve) RETIRED from running (Prime 2026-10-04 1634: "we aren't running the DC or the remote laser")**: 101 shut down cleanly, both `onboot 0`, both excluded from their nightly vzdump (last backups kept). That also ends the PVE 9 Secure Boot/VirtIO backup warnings. ⚠ If either is ever started again: enroll the Microsoft 2023 certs first (`qm enroll-efi-keys`, BitLocker check). diff --git a/servers/fv-ml1/README.md b/servers/fv-ml1/README.md index 5a231fb..b816817 100644 --- a/servers/fv-ml1/README.md +++ b/servers/fv-ml1/README.md @@ -18,6 +18,14 @@ Primary AI inference host for PFI. - **GPUs:** **4x** NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (95.6 GB VRAM each = 382 GB total, cc 12.0 / sm_120, GPU 0-3) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores. ⚠ This line read "2x" until 2026-09-19; `nvidia-smi` reports four. Read the host, not the doc. - **PSUs:** four present (PS1-PS4, all `ok`). ⚠ See § Power — on a single circuit that redundancy does not protect against the failure most likely to happen. - **Storage:** ZFS `zroot` (434 GB root) + `tank` pool (raidz2, 8× NVMe, 8.6 TB at `/tank`) — drive inventory below +- **Root-pool hygiene (2026-10-04, Prime):** `HF_HOME=/tank/aimodels/huggingface` system-wide (`/etc/environment` + + `/etc/profile.d/hf-home.sh`), the same tree the model containers mount as `/hf`; group `llm` (lkraven, llmuser, + infra-ops) writes it and every dir carries setgid so new files stay group `llm`. All user HF caches were merged + into it (`rsync --ignore-existing`, nothing lost) and cleared, as were the uv and pip caches: zroot went from + 124 G free to 194 G. Left on purpose: lkraven's `models--Cloudflare--clef-flash` (18 G), in use by the + 2026-10-04 bench; it is already on tank, so delete it once that bench is done. Docker (179 G) is still on zroot; + moving its data-root to tank needs a model-seat outage window (Prime's call). sudo session recordings are + archived weekly by `services/sudo-io-archive/`. - **OS:** Debian 13 (trixie), kernel 6.12.x - **Docker:** 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2 diff --git a/services/sudo-io-archive/README.md b/services/sudo-io-archive/README.md new file mode 100644 index 0000000..41f4713 --- /dev/null +++ b/services/sudo-io-archive/README.md @@ -0,0 +1,43 @@ +# sudo-io-archive — fleet sudo session recordings: archive, back up, clear + +Every `/etc/sudoers.d/infra-ops` carries `Defaults:infra-ops log_output`, so sudo records each infra-ops session +(command line, timing, stdin/stdout/stderr) under `/var/log/sudo-io/XX/YY/ZZ`, with no retention. fv-ml1 had 16 G +and esh-docker-vm 15 G by 2026-10-04. Prime: "those audits should be archived and backed up, then cleared. That +should be a regular thing." + +## What runs + +- **Weekly, Sunday 05:40 PT**: `sudo-io-archive.timer` → `sudo-io-archive.service` (nh3-dev **user** units, installed + from this folder into `~/.config/systemd/user/`). A failure pages infra-ops via `althing-notify-failure@`. +- **Collector** `sudo-io-archive` (nh3-dev): for each host in `hosts`, asks the host to pack every session older than + 30 days (`SUDO_IO_DAYS`), pulls the archive, checks sha256 + gzip integrity, keeps it as + `~/archive/sudo-io//--older-than-30d.tar.gz` (+ `.sha256`, mode 600), then tells the host + to delete exactly what it archived. One ops-log line per host. +- **Host side** `sudo-io-archive-host.sh`, streamed over ssh into `sudo -n bash -s`, phases `pack `, + `commit `, `status`. Works under `/var/log/sudo-io-archive/` (outbox, list, cutoff). + +## Safety properties + +- A session is eligible only when nothing in it changed after the cutoff, so running sessions are never touched. +- The host deletes nothing until the collector has the archive and has matched its sha256. Removal is + `tar --remove-files` over the same list the archive was built from (no `rm` on a variable; the list is refused if + any entry is absolute or contains `..`). A failed run leaves the host's logs and outbox; the next run re-ships it. +- The archives are covered by nh3-dev's backup: VM 102 is in nh3-pve's nightly vzdump to pbs-ana, mirrored to pbs-nh3. + +## Run by hand + +```bash +services/sudo-io-archive/sudo-io-archive # all hosts +services/sudo-io-archive/sudo-io-archive fv-ml1 # one host +ssh infra-ops@ 'sudo -n bash -s -- status' < services/sudo-io-archive/sudo-io-archive-host.sh +``` + +Replay an archived session: untar it anywhere, then `sudoreplay -d ` (as root). + +## Where the volume comes from (2026-10-04) + +Almost all of it is DATA piped through sudo, not commands: one fv-ml1 session is a 14.45 G recording of +`sudo tar -C /tank/aimodels -cf - erp-tune-v6-nvfp4a16` (a model copy), and esh-docker-vm records a ~380 M +`sudo rsync --server --sender … /var/lib/docker/volumes/homeassistant…` stream every night. Exempting bulk-transfer +commands from output recording (`Defaults!/usr/bin/rsync,/usr/bin/tar !log_output` in the infra-ops sudoers) would keep +the command line in the audit but stop recording the data; that is a policy change for Prime, not made here. diff --git a/services/sudo-io-archive/hosts b/services/sudo-io-archive/hosts new file mode 100644 index 0000000..c5424e6 --- /dev/null +++ b/services/sudo-io-archive/hosts @@ -0,0 +1,27 @@ +# name ssh target (infra-ops, NOPASSWD sudo, `Defaults:infra-ops log_output` in /etc/sudoers.d/infra-ops) +# Hosts with no infra-ops sudo (pfi-postgres, ana-wg, nh3-nas, the SureFire hosts) record nothing and are not listed. +pfi-pve infra-ops@10.250.250.31 +nh3-pve infra-ops@10.100.250.60 +nh3-pve-2 infra-ops@10.100.250.62 +esh-pve infra-ops@10.0.250.35 +esh-nas-pve infra-ops@10.0.50.55 +pbs-ana infra-ops@10.250.50.90 +pbs-nh3 infra-ops@10.100.50.90 +ana-docker infra-ops@10.250.50.70 +ana-nas infra-ops@10.250.50.50 +ana-filebot infra-ops@10.250.50.53 +corviduo-dev infra-ops@10.250.50.152 +pfi-pteradactyl infra-ops@10.250.50.55 +pfi-tacticalrmm infra-ops@10.250.50.57 +fv-ml1 infra-ops@10.251.50.54 +irv-ml1 infra-ops@irv-ml1.nh3.internal +pfi-gx10 infra-ops@10.100.50.60 +nh3-dev infra-ops@10.100.10.50 +nh3-docker infra-ops@10.100.50.40 +nh3-extdev infra-ops@10.100.50.42 +nh3-ml1 infra-ops@10.100.50.80 +esh-docker-vm infra-ops@10.0.50.45 +esh-vm-db infra-ops@10.0.50.60 +esh-ml1 infra-ops@10.0.50.80 +esh-matter infra-ops@10.0.90.20 +vm-esh-nas infra-ops@10.0.50.154 diff --git a/services/sudo-io-archive/sudo-io-archive b/services/sudo-io-archive/sudo-io-archive new file mode 100755 index 0000000..8423002 --- /dev/null +++ b/services/sudo-io-archive/sudo-io-archive @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# Collector for the fleet's sudo I/O logs. Runs on nh3-dev (weekly user timer, or by hand). +# services/sudo-io-archive/sudo-io-archive # every host in ./hosts +# services/sudo-io-archive/sudo-io-archive fv-ml1 pbs-ana # just these (names from ./hosts) +# SUDO_IO_DAYS=30 (default) SUDO_IO_ARCHIVE_DIR=~/archive/sudo-io (default) +# +# Per host: the host packs sessions older than SUDO_IO_DAYS into one outbox archive (sudo-io-archive-host.sh pack), +# this side pulls it, checks its sha256 and gzip integrity, keeps it under //, and only then tells the +# host to delete exactly what it archived (… commit ). A host that fails at any step keeps its logs and its +# outbox; the next run re-ships the same outbox. Backed up because nh3-dev (VM 102 on nh3-pve) is in the nightly +# vzdump to pbs-ana, which pbs-nh3 mirrors. +# +# Nothing here deletes anything locally (auto-memory feedback_no_rm_on_loop_variable): a failed pull leaves a +# `.part` file behind, which the summary reports. +set -uo pipefail +umask 077 # the archives hold whatever the sessions printed; owner-only +here=$(cd "$(dirname "$0")" && pwd) +days=${SUDO_IO_DAYS:-30} +dest=${SUDO_IO_ARCHIVE_DIR:-$HOME/archive/sudo-io} +hostscript="$here/sudo-io-archive-host.sh" +ssh_opts=(-o BatchMode=yes -o ConnectTimeout=10 -o ServerAliveInterval=30) +opslog="$here/../../scripts/ops-log" +want=("$@") +fail=0 +summary=() + +# hosts come in on fd 3: ssh reads stdin, and on fd 0 the first pull swallowed the rest of the list (2026-10-04) +while read -r -u 3 name target; do + [[ -z ${name:-} || $name == \#* ]] && continue + if [ ${#want[@]} -gt 0 ] && [[ ! " ${want[*]} " =~ " $name " ]]; then continue; fi + r=$(ssh "${ssh_opts[@]}" "$target" "sudo -n bash -s -- pack $days" < "$hostscript" 2>&1 | grep '^RESULT' | tail -1) + read -r _ kind sha sessions files _ <<<"${r:-RESULT ERROR}" + case "$kind" in + NOSUDOIO|NOTHING) summary+=("$name: ${kind,,}"); continue ;; + PACKED) ;; + *) summary+=("$name: PACK FAILED (${r:-no answer})"); fail=1; continue ;; + esac + install -d -m 700 "$dest/$name" + f="$dest/$name/$name-$(date -u +%Y%m%dT%H%M%SZ)-older-than-${days}d.tar.gz" + if ! ssh -n "${ssh_opts[@]}" "$target" 'cat /var/log/sudo-io-archive/outbox.tar.gz' > "$f.part"; then + summary+=("$name: PULL FAILED, left $f.part"); fail=1; continue + fi + got=$(sha256sum "$f.part" | cut -d' ' -f1) + if [ "$got" != "$sha" ] || ! gzip -t "$f.part" 2>/dev/null; then + summary+=("$name: VERIFY FAILED (sha $got vs $sha), left $f.part"); fail=1; continue + fi + mv -n "$f.part" "$f" + echo "$sha $(basename "$f")" > "$f.sha256" + c=$(ssh "${ssh_opts[@]}" "$target" "sudo -n bash -s -- commit $sha" < "$hostscript" 2>&1 | grep '^RESULT' | tail -1) + if [[ $c != "RESULT COMMITTED"* ]]; then + summary+=("$name: archived to $f but COMMIT FAILED (${c:-no answer}); host keeps its logs"); fail=1; continue + fi + size=$(du -h "$f" | cut -f1) + summary+=("$name: $sessions sessions / $files files -> $f ($size); host ${c#RESULT COMMITTED }") + [ -x "$opslog" ] && "$opslog" record --host "$name" --action archive --target sudo-io \ + --detail "sudo I/O logs older than ${days}d: $sessions sessions archived to nh3-dev $f ($size, sha256 ${sha:0:12}) and cleared on the host" >/dev/null 2>&1 +done 3< "$here/hosts" + +printf '%s\n' "${summary[@]}" +exit $fail diff --git a/services/sudo-io-archive/sudo-io-archive-host.sh b/services/sudo-io-archive/sudo-io-archive-host.sh new file mode 100644 index 0000000..e5cb93e --- /dev/null +++ b/services/sudo-io-archive/sudo-io-archive-host.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +# Host side of the sudo I/O log archiver. Runs as ROOT on a fleet host, fed over ssh by the collector: +# ssh infra-ops@ 'sudo -n bash -s -- pack 30' < sudo-io-archive-host.sh +# ssh infra-ops@ 'sudo -n bash -s -- commit ' < sudo-io-archive-host.sh +# +# Why this exists: every infra-ops sudoers file carries `Defaults:infra-ops log_output`, so sudo records each +# session under /var/log/sudo-io/XX/YY/ZZ with no retention at all. fv-ml1 reached 16 G / 14,076 sessions by +# 2026-10-04. Prime: "those audits should be archived and backed up, then cleared. That should be a regular thing." +# +# Safety rules this script keeps (eshpfi auto-memory feedback_no_rm_on_loop_variable): +# - every destructive target is a LITERAL path; nothing is deleted through a shell variable; +# - the session logs themselves are removed only by `tar --remove-files` reading the SAME list the shipped archive +# was built from, and only after the collector has confirmed the archive's sha256 on its side; +# - a session is eligible only when NO file in it is newer than the cutoff, so a still-running session is never +# touched, and the list is refused if any entry is absolute or contains `..`. +# Reading stdin is off limits here: bash -s is reading this very script from it. +set -euo pipefail +phase=${1:?usage: pack | commit | status} +arg=${2:-} + +if [ ! -d /var/log/sudo-io ]; then echo "RESULT NOSUDOIO"; exit 0; fi +install -d -m 700 /var/log/sudo-io-archive + +case "$phase" in +pack) + [[ $arg =~ ^[0-9]+$ ]] || { echo "pack needs a day count"; exit 2; } + if [ -f /var/log/sudo-io-archive/outbox.tar.gz ]; then + # a previous run packed but never committed: ship that same archive again rather than repacking + chgrp infra-ops /var/log/sudo-io-archive /var/log/sudo-io-archive/outbox.tar.gz + chmod 750 /var/log/sudo-io-archive + chmod 640 /var/log/sudo-io-archive/outbox.tar.gz + echo "RESULT PACKED $(cut -d' ' -f1 /var/log/sudo-io-archive/outbox.sha256) $(cat /var/log/sudo-io-archive/count) reused" + exit 0 + fi + touch -d "-${arg} days" /var/log/sudo-io-archive/cutoff + : > /var/log/sudo-io-archive/list + # session dirs are XX/YY/ZZ; include one only if nothing inside it changed after the cutoff + while IFS= read -r -d '' rel; do + if [ -z "$(find "/var/log/sudo-io/$rel" -newer /var/log/sudo-io-archive/cutoff -print -quit)" ]; then + printf '%s\0' "$rel" >> /var/log/sudo-io-archive/list + fi + done < <(find /var/log/sudo-io -mindepth 3 -maxdepth 3 -type d -printf '%P\0') + sessions=$(tr -cd '\0' < /var/log/sudo-io-archive/list | wc -c) + if [ "$sessions" -eq 0 ]; then + rm -f /var/log/sudo-io-archive/list + rm -f /var/log/sudo-io-archive/cutoff + echo "RESULT NOTHING" + exit 0 + fi + # sh -c so the session dirs land BEFORE -type (plain `xargs find -type f` appends them after the predicate) + files=$(cd /var/log/sudo-io && xargs -0 -a /var/log/sudo-io-archive/list sh -c 'find "$@" -type f' sh | wc -l) + # gzip, not zstd: it is on every Debian host, and sudo already gzips the stream files inside each session + tar -C /var/log/sudo-io --null -T /var/log/sudo-io-archive/list -cf - | gzip -1 > /var/log/sudo-io-archive/outbox.tar.gz + gzip -t /var/log/sudo-io-archive/outbox.tar.gz + members=$(tar -tzf /var/log/sudo-io-archive/outbox.tar.gz | grep -vc '/$' || true) + if [ "$members" -ne "$files" ]; then + echo "archive holds $members files, the sessions hold $files: refusing"; exit 3 + fi + echo "$sessions $files" > /var/log/sudo-io-archive/count + sha256sum /var/log/sudo-io-archive/outbox.tar.gz > /var/log/sudo-io-archive/outbox.sha256 + # the collector reads the outbox as plain infra-ops, NOT through sudo: sudo's log_output would record the whole + # archive as a new session, and sudo >= 1.9.14 (Debian 13) truncated the piped stream at exit (2026-10-04: + # fv-ml1 and corviduo-dev pulls came back 52 K and 7 K short). + chgrp infra-ops /var/log/sudo-io-archive /var/log/sudo-io-archive/outbox.tar.gz + chmod 750 /var/log/sudo-io-archive + chmod 640 /var/log/sudo-io-archive/outbox.tar.gz + echo "RESULT PACKED $(cut -d' ' -f1 /var/log/sudo-io-archive/outbox.sha256) $sessions $files" + ;; +commit) + [[ $arg =~ ^[0-9a-f]{64}$ ]] || { echo "commit needs the collector's sha256"; exit 2; } + [ -f /var/log/sudo-io-archive/outbox.tar.gz ] || { echo "no outbox to commit"; exit 3; } + [ "$(cut -d' ' -f1 /var/log/sudo-io-archive/outbox.sha256)" = "$arg" ] || { echo "sha mismatch with the collector: refusing"; exit 3; } + [ "$(sha256sum /var/log/sudo-io-archive/outbox.tar.gz | cut -d' ' -f1)" = "$arg" ] || { echo "outbox changed since pack: refusing"; exit 3; } + if grep -z -q -E '(^/|(^|/)\.\.(/|$))' /var/log/sudo-io-archive/list; then echo "unsafe path in list: refusing"; exit 3; fi + # removal = exactly the archived list, via tar itself (no rm on a variable). Output discarded; the archive that + # matters is the one the collector already holds. + # --ignore-failed-read: a retried commit (files already gone) must still finish its cleanup, not wedge forever + tar -C /var/log/sudo-io --null -T /var/log/sudo-io-archive/list --remove-files --ignore-failed-read -cf - | cat > /dev/null + find /var/log/sudo-io -mindepth 1 -maxdepth 2 -type d -empty -delete + rm -f /var/log/sudo-io-archive/outbox.tar.gz + rm -f /var/log/sudo-io-archive/outbox.sha256 + rm -f /var/log/sudo-io-archive/list + rm -f /var/log/sudo-io-archive/cutoff + rm -f /var/log/sudo-io-archive/count + echo "RESULT COMMITTED $(du -sh /var/log/sudo-io | cut -f1) left" + ;; +status) + echo "RESULT STATUS $(du -sh /var/log/sudo-io | cut -f1) $(find /var/log/sudo-io -mindepth 3 -maxdepth 3 -type d | wc -l) sessions outbox=$([ -f /var/log/sudo-io-archive/outbox.tar.gz ] && echo yes || echo no)" + ;; +*) echo "unknown phase $phase"; exit 2 ;; +esac diff --git a/services/sudo-io-archive/sudo-io-archive.service b/services/sudo-io-archive/sudo-io-archive.service new file mode 100644 index 0000000..ca104cb --- /dev/null +++ b/services/sudo-io-archive/sudo-io-archive.service @@ -0,0 +1,8 @@ +[Unit] +Description=Archive fleet sudo I/O logs older than 30 days to nh3-dev, then clear them on each host +OnFailure=althing-notify-failure@%n.service + +[Service] +Type=oneshot +ExecStart=%h/development/eshpfi-management/services/sudo-io-archive/sudo-io-archive +TimeoutStartSec=4h diff --git a/services/sudo-io-archive/sudo-io-archive.timer b/services/sudo-io-archive/sudo-io-archive.timer new file mode 100644 index 0000000..c87dc33 --- /dev/null +++ b/services/sudo-io-archive/sudo-io-archive.timer @@ -0,0 +1,9 @@ +[Unit] +Description=Weekly fleet sudo I/O log archive (Sunday 05:40, after the night's backups, before the 08:00 freshness check) + +[Timer] +OnCalendar=Sun *-*-* 05:40:00 +Persistent=true + +[Install] +WantedBy=timers.target