esh-pfi-infrastructure/persistent-memory.d/2026-10-06-backup-checker-fixes-and-mirror-retention.md

[2026-10-06] Backup checker: destroyed guests, the stdin swallow and snapshot age; residue pruned; mirror retention matched

• Backup check (2026-10-06): the 0800 run was clean apart from one false STALE, now FIXED. nh3-pve-2 coverage read clean, svos drill B was ACKNOWLEDGED, nothing was DEGRADED. The false STALE was pbs/ana-pve/vm/110, pfi-ana-webhost, destroyed 10-04 with qm destroy --purge. Three fixes landed in scripts/check-backup-freshness.sh. infra-hermes, 0a348e5: destroyed guests are now reported as policy residue (GUEST DESTROYED), and an ssh in the read loop now uses -n, because without it the whole PBS layer silently vanished. infra-ops, d920cf3: snapshot age now comes from the snapshot NAME, not the directory mtime, which PBS bumps at every verify and every 30-day re-verify. Verified exit 0; positive control at 5h flags 10. Residue PRUNED on Prime's word (2026-10-06 1307): on pbs-ana, ana-pve/vm/110 (8 snapshots) and esh-pve/vm/102 (7, including the protected final 09-10 image, unprotected first; Prime's 09-19 ruling said that VM had no needed state); on the pbs-nh3 mirror, 162 + 10. The checker now reads 0 GUEST DESTROYED, exit 0. Mirror retention SET (Prime "match history", 2026-10-06 1310): pbs-nh3 prune job default-backups-mirror-1b7d4123- now keeps last 3 / daily 14 / weekly 8 / monthly 12 / yearly 5, the same as pbs-ana. It was keep-all: 3,932 snapshots against 266, 5.49 TiB against 0.93 TiB, nh3-nas at 79%. Dry-run: keep 738, remove 3,194. First run at 00:00 on 10-07, then daily GC. remove-vanished stays false (deliberate). ⚠ Check on 10-07 that the prune ran and GC freed the space. Old cfg: /root/prune.cfg.bak-20261006 on pbs-nh3.

• The false STALE came from qm destroy 110 --purge (10-04 1726, pfi-ana-webhost retirement): --purge also removed 110 from the 03:00 job's exclude list, so the checker saw a "covered" guest with no new snapshots.

• infra-hermes's first cut crashed under set -u (case "$guest" in vm/*) never matches a namespaced path); infra-ops sent the */vm/* + pre-set fix and infra-hermes adopted it. The worse bug, found only by running it: the inventory ssh inside the while-read loop swallowed the loop's stdin, so the whole PBS layer silently vanished while the verdict stayed green.

• Snapshot dir mtime moves on every verify/notes edit (pbs-ana verify: 02:30 daily, outdated-after 30). Age now comes from the snapshot name. Auto-memory reference_pbs_snapshot_mtime_moves_on_verify.

• infra-hermes added a runbook rule: alarm-script changes get a two-seat review before they ship (24b7195).

• Prime "prune backups" (1307): residue deleted on pbs-ana (ana-pve/vm/110 ×8, esh-pve/vm/102 ×7 incl. the protected 09-10 final image) and on the pbs-nh3 mirror (162 + 10; sync has remove-vanished false). The mirror's prune job had no keep rules (keep-all); Prime "match history" set it to the primary's rules.