vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: destroyed-guest backup residue pruned on pbs-ana and the pbs-nh3 mirror; mirror has no retention

Browse Source
This commit is contained in:
vh
2026-10-06 13:08:41 -07:00
parent e931b319a2
commit ef67546b9c
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+1 -1
View File
@@ -119,7 +119,7 @@ _As of 2026-10-05 ~2035 PT. The newest subsection is first; older subsections ca
- **Nothing is mid-action.** No claims held (`scripts/ops-log claims`); eshpfi pushed through `1210b4a`.
- **New SVOS on nh3-svos (VM 111, nh3-pve-2, 10.100.50.43): DEPLOYED and ACCEPTED by svos-dev 2026-10-05 ~2024** (thread `01M47ET6XH0M7JZSH98Z19SKR4` closed). Separate from the High Seat (`svos.service` :8770 on nh3-dev) and Miranda, which are NOT migrated. Front door `https://svos.phasefinal.com`, mesh only, Basic auth login `Vuong` (Prime has the password; vault `nh3-svos/basic-auth-vuong`). ⚠ **After every SVOS release** (`/opt/svos/current` moves): `sudo /usr/local/sbin/svos-front-inventory`, and keep the previous release dir while it is "previous" (svos-dev's runbook carries both). Only open item is Prime's: whether his iPhone home-screen app keeps the login. → `servers/nh3-svos/README.md`
- **Backup check (2026-10-06): the 0800 run was clean apart from one false STALE, now FIXED.** nh3-pve-2 coverage read clean, svos drill B was ACKNOWLEDGED, nothing was DEGRADED. The false STALE was `pbs/ana-pve/vm/110`, pfi-ana-webhost, destroyed 10-04 with `qm destroy --purge`. Three fixes landed in `scripts/check-backup-freshness.sh`. infra-hermes, `0a348e5`: destroyed guests are now reported as policy residue (GUEST DESTROYED), and an ssh in the read loop now uses `-n`, because without it the whole PBS layer silently vanished. infra-ops, `d920cf3`: snapshot age now comes from the snapshot NAME, not the directory mtime, which PBS bumps at every verify and every 30-day re-verify. Verified exit 0; positive control at 5h flags 10. The vm/110 and esh vm/102 snapshots on pbs-ana are residue; deleting them is destructive, so ask Prime before pruning.
- **Backup check (2026-10-06): the 0800 run was clean apart from one false STALE, now FIXED.** nh3-pve-2 coverage read clean, svos drill B was ACKNOWLEDGED, nothing was DEGRADED. The false STALE was `pbs/ana-pve/vm/110`, pfi-ana-webhost, destroyed 10-04 with `qm destroy --purge`. Three fixes landed in `scripts/check-backup-freshness.sh`. infra-hermes, `0a348e5`: destroyed guests are now reported as policy residue (GUEST DESTROYED), and an ssh in the read loop now uses `-n`, because without it the whole PBS layer silently vanished. infra-ops, `d920cf3`: snapshot age now comes from the snapshot NAME, not the directory mtime, which PBS bumps at every verify and every 30-day re-verify. Verified exit 0; positive control at 5h flags 10. Residue PRUNED on Prime's word (2026-10-06 1307): on pbs-ana, ana-pve/vm/110 (8 snapshots) and esh-pve/vm/102 (7, including the protected final 09-10 image, unprotected first; Prime's 09-19 ruling said that VM had no needed state); on the pbs-nh3 mirror, 162 + 10. The checker now reads 0 GUEST DESTROYED, exit 0. ⚠ **Open for Prime: the mirror keeps everything forever.** Its prune job has no keep-* rules and its sync has remove-vanished false: 3,932 snapshots against 266 on the primary, 5.49 TiB of chunks against 0.93 TiB, on an nh3-nas volume at 79%.
- **Waiting on Prime (updated 2026-10-05 2035):** push Booth commit `ce095c1` (touch-layout fix, `~/development/booth`, browser test + mutation); a trusted HTTPS name for ESH Homepage (https://10.0.50.45 serves Traefik's default cert; the ESH ACME resolver, Route53, is dead); nh3-dev reboot (runs kernel 6.1.0-49, 6.1.0-53 installed; its engine is ALREADY 29.8.2 since 10-05 1413, so the old "reboot + engine bump" wording was stale, checked 2035); Docker engine bumps held on nh3-docker (carries the post office) and esh-docker-vm (NFS D-state history); fv-ml1's held ZFS 2.3.9, nvidia-container-toolkit 1.20, and a new kernel only together with a newer NVIDIA driver (6.12.111 broke the 580.65 DKMS build). ⚠ **irv-ml1: DO NOT PATCH (Prime).** ana-nas is out of scope.
- **Scheduled / to verify:** 10-05 0815 + 0900 infra-hermes jobs (sfsrv-ana `pre-pbs4` snapshot delete after its 04:00 backup; drop `pve/root-pre-pve9` on pfi-pve + esh-pve) — outcome not checked in this session, read `scripts/ops-log tail`; esh-nas-pve `nvme/ROOT@pre-pve9` destroy ~10-10 (ours); 10-11 0540 first weekly sudo-io archiver run (check it), 10-11 0900 infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet.
- **Worldtree memory gate: the 20261004T143011Z FAIL did not recur.** That run filed the forbidden `u-negative-ephemeral` fact once (run 1, negative 1/2), and an esh-pve reboot overlapped its run 1. The next two batches PASSED: 20261005T143010Z and 20261006T143010Z (user filing 11/12 = 0.92 in both runs, floor 0.80, negative 2/2). Read from `Worldtree/docs/eval/memory_acceptance_runs/<stamp>/summary.md` on 10-06 after infra-hermes' 0747 PASS report. That is 2 passes after 1 failure on a stochastic gate: a single filing slip is not ruled out, so keep reading the daily verdict rather than calling it fixed.