10 lines
2.9 KiB
Markdown
10 lines
2.9 KiB
Markdown
# `[2026-10-06]` Backup checker: destroyed guests, the stdin swallow and snapshot age; residue pruned; mirror retention matched
|
||
|
||
- **Backup check (2026-10-06): the 0800 run was clean apart from one false STALE, now FIXED.** nh3-pve-2 coverage read clean, svos drill B was ACKNOWLEDGED, nothing was DEGRADED. The false STALE was `pbs/ana-pve/vm/110`, pfi-ana-webhost, destroyed 10-04 with `qm destroy --purge`. Three fixes landed in `scripts/check-backup-freshness.sh`. infra-hermes, `0a348e5`: destroyed guests are now reported as policy residue (GUEST DESTROYED), and an ssh in the read loop now uses `-n`, because without it the whole PBS layer silently vanished. infra-ops, `d920cf3`: snapshot age now comes from the snapshot NAME, not the directory mtime, which PBS bumps at every verify and every 30-day re-verify. Verified exit 0; positive control at 5h flags 10. Residue PRUNED on Prime's word (2026-10-06 1307): on pbs-ana, ana-pve/vm/110 (8 snapshots) and esh-pve/vm/102 (7, including the protected final 09-10 image, unprotected first; Prime's 09-19 ruling said that VM had no needed state); on the pbs-nh3 mirror, 162 + 10. The checker now reads 0 GUEST DESTROYED, exit 0. **Mirror retention SET (Prime "match history", 2026-10-06 1310):** pbs-nh3 prune job `default-backups-mirror-1b7d4123-` now keeps last 3 / daily 14 / weekly 8 / monthly 12 / yearly 5, the same as pbs-ana. It was keep-all: 3,932 snapshots against 266, 5.49 TiB against 0.93 TiB, nh3-nas at 79%. Dry-run: keep 738, remove 3,194. First run at 00:00 on 10-07, then daily GC. remove-vanished stays false (deliberate). ⚠ **Check on 10-07** that the prune ran and GC freed the space. Old cfg: `/root/prune.cfg.bak-20261006` on pbs-nh3.
|
||
|
||
- The false STALE came from `qm destroy 110 --purge` (10-04 1726, pfi-ana-webhost retirement): --purge also removed 110 from the 03:00 job's exclude list, so the checker saw a "covered" guest with no new snapshots.
|
||
- infra-hermes's first cut crashed under `set -u` (`case "$guest" in vm/*)` never matches a namespaced path); infra-ops sent the `*/vm/*` + pre-set fix and infra-hermes adopted it. The worse bug, found only by running it: the inventory ssh inside the while-read loop swallowed the loop's stdin, so the whole PBS layer silently vanished while the verdict stayed green.
|
||
- Snapshot dir mtime moves on every verify/notes edit (pbs-ana verify: 02:30 daily, `outdated-after 30`). Age now comes from the snapshot name. Auto-memory `reference_pbs_snapshot_mtime_moves_on_verify`.
|
||
- infra-hermes added a runbook rule: alarm-script changes get a two-seat review before they ship (`24b7195`).
|
||
- Prime "prune backups" (1307): residue deleted on pbs-ana (ana-pve/vm/110 ×8, esh-pve/vm/102 ×7 incl. the protected 09-10 final image) and on the pbs-nh3 mirror (162 + 10; sync has remove-vanished false). The mirror's prune job had no keep rules (keep-all); Prime "match history" set it to the primary's rules.
|