diff --git a/docs/runbooks/backups.md b/docs/runbooks/backups.md index 8a13da9..bd32b0a 100644 --- a/docs/runbooks/backups.md +++ b/docs/runbooks/backups.md @@ -28,8 +28,8 @@ layer has failed. | **irv-ml1** | **bare metal** | ❌ none (not a VM) | ✅ → rest-server-**nh3** | ⚠️ **restic is the ONLY net** | | nh3-docker | VM (nh3-pve) | ✅ `nh3-pve` | ✅ → rest-server-**nh3** | no | | esh-docker-vm | VM (esh-pve) | ✅ `esh-pve` | ✅ → rest-server-**ana** | no | -| esh-vm-db | VM (esh-pve-nas?) | ❓ **no `esh-pve-nas` ns — VERIFY** | ✅ → rest-server-**ana** | ⚠️ maybe restic-only (a DB!) | -| vm-esh-nas | VM (esh-pve-nas) | ❓ **no `esh-pve-nas` ns — VERIFY** | ✅ → rest-server-**ana** | ⚠️ maybe restic-only | +| esh-vm-db | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only (a DB!)** | +| vm-esh-nas | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only** | | other pfi-pve / nh3-pve VMs/CTs | VM/CT | ✅ respective ns | (PBS only) | no | | SureFire `sfsrv-pve` | tenant VMs | ✅ `sfsrv-pve` ns | (PBS only) | no | @@ -150,19 +150,27 @@ the stop→umount→rm-ghost→remount→start variant. ## Known gaps / TODO -- [ ] **No backup-freshness alerting** — the systemic miss; the ana - outage went unnoticed ~6.5 weeks. Add a check (e.g. a daily job that - alerts if any repo's newest snapshot is > 48 h old, or wire - resticprofile's check/`--json` output to a notifier). -- [ ] **fstab hardening** — change ana-docker's `/mnt/backup` entry from - `defaults` to `_netdev,nofail,x-systemd.automount,x-systemd.mount-timeout=30,retry` - so it self-heals on reboot/NAS-blip instead of getting stuck. -- [ ] **ana-ml2 has no PBS net** (bare metal) — restic is its only layer. - Acceptable IF restic stays healthy + alerted; bulk `/tank` models are - re-downloadable, but bespoke quants/configs/scripts are not. -- [ ] **Verify esh-pve-nas coverage** — vm-esh-nas + **esh-vm-db (a - postgres+mongo DB VM)** have no PBS namespace. Confirm whether they - get a PBS image anywhere; if not, restic is their sole net. -- [ ] **Rotate rest-server repo passwords** — the 5 per-repo basic-auth - creds were exposed during the 2026-06-20 diagnosis (Backrest config - dump). Add to the disclosed-keys rotation queue. +- [x] **Backup-freshness alerting — DONE (2026-06-20).** + `scripts/check-backup-freshness.sh` (the 2-min check, exit 1 on + stale/down) + a daily **systemd user timer on nh3-dev** at 08:00 + (`scripts/install-backup-freshness-timer.sh`) → `backup-freshness-alert.sh` + posts an **althing alert to infra-ops** on any stale/down layer. Run the + check by hand anytime. (Channel is althing for now — swap in email/ntfy if + you want a louder one.) +- [x] **fstab hardening — DONE (2026-06-20).** ana-docker `/mnt/backup` → + `noauto,x-systemd.automount,x-systemd.mount-timeout=30` (autofs self-heals + on a NAS blip instead of getting stuck `failed`; activates on next reboot). + `/etc/fstab.bak-pre-harden` saved. **NOTE:** `/mnt/compose` is still bare + `defaults` (same latent risk) — harden it the same way when convenient. +- [ ] **ana-ml2 has no PBS net** (bare metal) — restic is its only layer; now + healthy + alerted. Bulk `/tank` models are re-downloadable; bespoke + quants/configs/scripts are the real loss-risk. +- [x] **esh-pve-nas coverage — VERIFIED (2026-06-20): NOT PBS-covered.** No + `esh-pve-nas` namespace exists on PBS-ANA, so **esh-vm-db (postgres+mongo) + + vm-esh-nas are restic-only.** For the DB VM, restic-with-dumps is the + *preferred* method (vs a VM image) **IF** the resticprofile includes + `pg_dump`/`mongodump` — confirm that. Optionally add esh-pve-nas as a PBS + source. ESH is home-lab (no SLA). +- [ ] **Rotate rest-server repo passwords** — the 5 per-repo basic-auth creds + were exposed during the 2026-06-20 diagnosis. **BELAYED** — operator + handling offline. diff --git a/persistent-memory.md b/persistent-memory.md index 4eea4b0..9570758 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -200,8 +200,15 @@ _As of 2026-06-20:_ 22:53Z** (total 19→20); ana-docker's larger catch-up was still running at hand-off (will land). esh-docker-vm/esh-vm-db/vm-esh-nas auto-catch-up at their next 01:00 timers (or trigger manually). **Full topology + 2-min freshness-check now documented - in `docs/runbooks/backups.md`.** STILL OPEN (prevention): backup-freshness alert + - fstab `_netdev`/retry hardening + verify esh-pve-nas PBS gap + rotate the 5 creds. + in `docs/runbooks/backups.md`.** **PREVENTION DONE (2026-06-20):** (1) freshness + alert — `scripts/check-backup-freshness.sh` + daily systemd user timer on nh3-dev + (08:00) → althing alert to infra-ops on any stale/down layer (proven end-to-end); + (2) fstab hardened — ana-docker `/mnt/backup` → `noauto,x-systemd.automount,…` + (autofs self-heals, active next boot; `.bak-pre-harden` saved); (3) esh-pve-nas + VERIFIED **not** PBS-covered → esh-vm-db (a DB!) + vm-esh-nas are restic-only. + STILL OPEN: rotate the 5 rest-server creds (operator, offline); harden + `/mnt/compose` too (also bare defaults); the 3 esh + ana-docker restic clients + clear at tonight's 01:00. - **ana-docker disk incident RESOLVED — root cause was a 94 GB unrotated `langfuse-clickhouse` log, NOT (just) image bloat.** ClickHouse logs @@ -270,6 +277,16 @@ _As of 2026-06-20:_ ## Recent decisions +- `[2026-06-20]` **rest-server-ana recovered + backup prevention shipped + worldtree-dev + admin keys provisioned** (operator-directed). rest-server-ana fixed (mount-a + + force-recreate via `infra-ops@ana-docker`); freshness alert + fstab hardening landed + (see Current-state BACKUP entry + `docs/runbooks/backups.md`). Minted worldtree-dev + **admin-tier Heimdall keys** on demo (key_id d113207c) + personal (f4f75adb) via + `POST /admin/keys {tier:admin}` + `POST /admin/users/worldtree-dev/tier`, verified + read+write, dropped mode-600 to `~/.wt-admin-{demo,personal}` on nh3-dev → worldtree-dev + now self-serves key minting (cuts infra-ops out of the per-key loop, like the + claude-bot gitea token). Cred rotation (5 rest-server pw) BELAYED per operator. + - `[2026-06-20]` **Worldtree capability-gateway aliases stood up; NO `wt-` prefix (operator) — reuse the existing capability convention.** gen/ gen-reasoning/classifier/summarizer/chat-judge/reranker/scalar-judge diff --git a/scripts/backup-freshness-alert.sh b/scripts/backup-freshness-alert.sh new file mode 100755 index 0000000..0552ceb --- /dev/null +++ b/scripts/backup-freshness-alert.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +# backup-freshness-alert.sh — daily wrapper around check-backup-freshness.sh. +# Runs the check; on any stale/down layer (exit!=0) posts an althing alert to +# infra-ops so the silent-failure class (the 2026-05-06→06-20 ana outage that +# went unnoticed ~6.5 weeks) can't recur. Installed as a systemd user timer on +# nh3-dev via scripts/install-backup-freshness-timer.sh. +set -uo pipefail +REPO=/home/lkraven/development/eshpfi-management +ALTHING=/home/lkraven/.local/bin/althing-cli + +out=$("$REPO/scripts/check-backup-freshness.sh" 2>&1); rc=$? +printf '%s\n' "$out" + +if [ "$rc" -ne 0 ]; then + printf 'Automated daily backup-freshness check found STALE or DOWN backup layer(s) on the PFI fleet.\nRunbook: docs/runbooks/backups.md (topology, 2-min check, rest-server-ana recovery).\n\n%s\n' "$out" \ + | "$ALTHING" post --to infra-ops --subject "🔴 Backup freshness ALERT ($(date '+%Y-%m-%d'))" 2>&1 \ + || echo "WARN: althing alert post failed — the check still ran (exit $rc); investigate manually." +fi +exit "$rc" diff --git a/scripts/check-backup-freshness.sh b/scripts/check-backup-freshness.sh new file mode 100755 index 0000000..3097679 --- /dev/null +++ b/scripts/check-backup-freshness.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash +# check-backup-freshness.sh — the "are we actually backed up?" check. +# +# Walks every backup layer and flags anything whose newest snapshot is older +# than the threshold (default 48h) or any down endpoint. Prints a report; +# exits 0 if everything is fresh, 1 if anything is stale/down. Designed to be +# run by a daily timer that alerts on non-zero exit (see +# scripts/install-backup-freshness-timer.sh), or by hand anytime. +# +# Companion to docs/runbooks/backups.md. Read-only — only SSH stat/curl. +# +# BACKUP_MAX_AGE_HOURS=48 scripts/check-backup-freshness.sh +set -uo pipefail + +MAX_AGE_H="${BACKUP_MAX_AGE_HOURS:-48}" +SSH="ssh -o ConnectTimeout=8 -o BatchMode=yes" +now=$(date +%s) +stale=() ; fresh=() ; errors=() + +# newest snapshot epoch under a remote glob (echoes epoch or empty) +newest_epoch() { # $1=host $2=glob + $SSH "$1" "stat -c %Y $2 2>/dev/null | sort -n | tail -1" 2>/dev/null +} +report() { # $1=label $2=epoch("" = none) + local label="$1" ep="$2" + if [ -z "$ep" ]; then stale+=("$label: NO SNAPSHOTS / unreachable"); return; fi + local age=$(( (now - ep) / 3600 )) + local when; when=$(date -d "@$ep" '+%Y-%m-%d %H:%M' 2>/dev/null) + if [ "$age" -gt "$MAX_AGE_H" ]; then stale+=("$label: ${age}h old (newest $when)") + else fresh+=("$label: ${age}h old (newest $when)"); fi +} + +echo "=== Backup freshness (threshold ${MAX_AGE_H}h) — $(date '+%Y-%m-%d %H:%M %Z') ===" + +# --- Layer: restic file+DB, ANA side (rest-server-ana) --- +for c in ana-docker ana-ml2 esh-docker-vm esh-vm-db vm-esh-nas; do + report "restic/ana/$c" "$(newest_epoch ana-nas "/mnt/backup/restic/repo/ana/$c/snapshots/*")" +done +# --- Layer: restic file+DB, NH3 side (rest-server-nh3) --- +for c in irv-ml1 nh3-docker; do + report "restic/nh3/$c" "$(newest_epoch nh3-nas "/volume1/Backup/restic/$c/snapshots/*")" +done +# --- Layer: PBS VM images (newest per guest, all namespaces) --- +pbs=$($SSH pbs-ana 'for ns in /mnt/pbs-datastore/ns/*/; do n=$(basename "$ns") + for d in vm ct; do for g in "$ns$d"/*/; do [ -d "$g" ] || continue + nb=$(ls -d "$g"20*T* 2>/dev/null | sort | tail -1) + [ -n "$nb" ] && echo "$n/$d/$(basename "$g") $(stat -c %Y "$nb")" + done; done; done' 2>/dev/null) +if [ -z "$pbs" ]; then errors+=("PBS-ANA: unreachable or no snapshots"); else + while read -r guest ep; do [ -n "$guest" ] && report "pbs/$guest" "$ep"; done <<<"$pbs" +fi + +# --- rest-server endpoint health (401 = up & serving) --- +for ep in "rest-server-ana http://10.250.50.70:8000/" "rest-server-nh3 http://10.100.50.50:8000/"; do + set -- $ep + code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 6 "$2" 2>/dev/null) + [ "$code" = "401" ] && fresh+=("$1: up (401)") || stale+=("$1: endpoint code=$code (expected 401)") +done + +echo +echo "FRESH (${#fresh[@]}):"; printf ' ✅ %s\n' "${fresh[@]}" +if [ "${#stale[@]}" -gt 0 ] || [ "${#errors[@]}" -gt 0 ]; then + echo; echo "STALE / PROBLEMS (${#stale[@]}+${#errors[@]}):" + printf ' 🔴 %s\n' "${stale[@]}" "${errors[@]}" + echo; echo "RESULT: STALE — see docs/runbooks/backups.md" + exit 1 +fi +echo; echo "RESULT: all backups fresh" +exit 0 diff --git a/scripts/install-backup-freshness-timer.sh b/scripts/install-backup-freshness-timer.sh new file mode 100755 index 0000000..129d27b --- /dev/null +++ b/scripts/install-backup-freshness-timer.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +# install-backup-freshness-timer.sh — install/refresh the daily backup-freshness +# alert as a systemd USER timer on nh3-dev (the only host with SSH to all backup +# stores + althing-cli). Idempotent; re-run after editing the wrapper/check. +# Requires linger (loginctl enable-linger lkraven) so it fires without a login. +set -euo pipefail +UNIT_DIR="$HOME/.config/systemd/user" +REPO=/home/lkraven/development/eshpfi-management +mkdir -p "$UNIT_DIR" + +cat > "$UNIT_DIR/backup-freshness.service" < "$UNIT_DIR/backup-freshness.timer" <