vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sudo-io-archive): weekly fleet archive of sudo session recordings (pack, verify, then clear); fv-ml1 HF_HOME on tank and root-pool cleanup

Browse Source
This commit is contained in:
vh
2026-10-04 22:48:05 -07:00
parent 626e0c7f94
commit 008bd685aa
8 changed files with 248 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+2
View File
@@ -119,6 +119,8 @@ _As of 2026-10-04 ~0546 PT. The newest subsection is first; older subsections ca
### 2026-10-03/04: live now (as of 2026-10-04 ~0546 PT)
- **sudo session recordings: archived weekly since 2026-10-04 (Prime: "archived and backed up, then cleared … a regular thing")**: `services/sudo-io-archive/` (nh3-dev user timer, Sun 05:40; sessions >30 d → `~/archive/sudo-io/<host>/`, sha256-verified, then removed on the host by `tar --remove-files` over the archived list; nh3-dev's vzdump to PBS backs them up). First pass took 4.8 G from 11 hosts. ⚠ The volume is DATA piped through sudo, not commands: a 14.45 G recording of `sudo tar` of a model dir on fv-ml1, and a ~380 M `sudo rsync` of the HA volume nightly on esh-docker-vm. Open: exempt bulk-transfer commands from `log_output` (Prime's call). Lessons in the scripts: pull the outbox WITHOUT sudo (sudo ≥1.9.14 truncated piped output and re-records it), read the host list on fd 3 (ssh ate stdin).
- **fv-ml1 root pool cleaned (2026-10-04)**: unused vLLM images/containers + build cache (~93 G), user HF/uv/pip caches merged to tank and cleared (zroot 124 G → 194 G free); `HF_HOME=/tank/aimodels/huggingface` system-wide. Docker data-root still on zroot (move = outage window, Prime). → `servers/fv-ml1/README.md`
- **pfi-ana-webhost DESTROYED 2026-10-04 ~1726 (Prime: retire, build fresh if ever needed)**: `qm destroy 110 --purge`; last image `pbs-ana:backup/vm/110/2026-10-04T10:28:18Z`. All FortiGate objects for it deleted (policy 18, VIPs, address, service, DHCP reservation; **38.120.12.45 now unmapped**), public + internal DNS records removed, inventory moved to `docs/retired/pfi-ana-webhost/`. ⭐ dragonfireacoustics.com = Cloudflare Pages in the CLIENT's account (source on nh3-dev `~/development/adhoc/dragonfireacoustics-site`, pushed 2026-10-04 to private gitea `vh/dragonfireacoustics-site`). **Supersedes the two 2026-09-17 dragonfire entries.**
- **phasefinal.com on Cloudflare Pages since 2026-10-04 ~1720** (project `phasefinal`, our account `vh@phasefinal`): www + apex CNAME → `phasefinal.pages.dev`; nginx container on ana-docker retired. Privacy page hosting line rewritten (Prime-approved wording). `_headers` + `404.html` added. Apex → www 301 restored as a zone Redirect Rule. Tokens: Prime vaulted an account **token-creator** (`nh3-dev/.config/cloudflare/infra-ops-token-creator`); infra-ops minted **`infra-ops-pages`** with it (vault `…/infra-ops-pages-token`: Pages Write + all-zones DNS/redirect/cache). Deploy: `stacks/phasefinal-web/deploy-pages.sh` (proven with the new token).
- **VM 101 PFI-ANA-DC (pfi-pve) and VM 104 nh3-laser (nh3-pve) RETIRED from running (Prime 2026-10-04 1634: "we aren't running the DC or the remote laser")**: 101 shut down cleanly, both `onboot 0`, both excluded from their nightly vzdump (last backups kept). That also ends the PVE 9 Secure Boot/VirtIO backup warnings. ⚠ If either is ever started again: enroll the Microsoft 2023 certs first (`qm enroll-efi-keys`, BitLocker check).