vh
a5691ce796
Work by a parallel session on 2026-09-12; committed here with the rest of the day's changes. Rationale in persistent-memory.d/2026-09-12-esh-vm-db-restic-repair.md. The only visible symptom was a systemd-failed unit from a Sep 6 repository network timeout after boot. The real fault was quieter and much worse: the pre-backup hook logged failures as WARN and returned zero, so pg_dumpall could fail every single night -- it used TCP localhost and wanted a password nobody supplied -- while restic dutifully backed up the stale April 23 dump still sitting in the staging directory and reported success. Mongo was fine, which is part of why it went unnoticed. Postgres now dumps over the /var/run/postgresql socket with peer auth and -w, and both database failures now fail the backup rather than masking it, while still preserving any prior per-DB dump rather than truncating to nothing. An ERRORS counter replaces the warn-and-continue path, and the staging directory is overridable via RESTIC_STAGE_DIR so the new test can exercise it. Adds backup.contract.md, retry.conf and test_pre_backup.py -- three red-green regression tests covering the failure modes above. systemd drop-ins on both jobs add network-online ordering plus Restart=on-failure with a 5m delay and a 3-per-hour limit, which addresses the original boot-timeout symptom. Verified against a real run: snapshot bc5eeaff at 07:01 PDT with a fresh 3.46MB PG dump, retrieved from the repository with decompression and completion marker checked (not a full restore). Repository check passed, 99 snapshots. The old hook and stale dump are preserved root-only at /var/lib/restic/repair-20260912.
67 lines
2.4 KiB
Bash
67 lines
2.4 KiB
Bash
#!/bin/bash
|
|
# pre-backup.sh — esh-vm-db.
|
|
# Runs as root from resticprofile's `run-before`.
|
|
#
|
|
# Produces consistent DB dumps in /var/lib/restic/stage/. Two DBs here:
|
|
# - Postgres 15 (port 5432, local) — pg_dumpall all databases
|
|
# - MongoDB (port 27017, local) — mongodump all databases
|
|
#
|
|
# Peer-data DBs (paperless-ng on Postgres) are the primary consumers;
|
|
# raw volume capture isn't in the restic source list so these dumps
|
|
# are the ONLY way restic sees DB data.
|
|
#
|
|
# Required dump failures return nonzero so restic cannot report stale DB data
|
|
# as a successful fresh backup. Previous successful dumps are preserved.
|
|
|
|
set -euo pipefail
|
|
ERRORS=0
|
|
|
|
STAGE=${RESTIC_STAGE_DIR:-/var/lib/restic/stage}
|
|
install -d -m 0700 "$STAGE"
|
|
|
|
log() { printf '%s pre-backup(esh-vm-db): %s\n' "$(date -Is)" "$*"; }
|
|
warn() { printf '%s pre-backup(esh-vm-db): WARN: %s\n' "$(date -Is)" "$*" >&2; }
|
|
error() { ERRORS=$((ERRORS + 1)); warn "$*"; }
|
|
|
|
# ---- Postgres ----------------------------------------------------------
|
|
PG_DUMP="$STAGE/pg_dumpall.sql.gz"
|
|
if sudo -u postgres pg_isready -h /var/run/postgresql -p 5432 > /dev/null 2>&1; then
|
|
log "pg_dumpall starting → $PG_DUMP"
|
|
if (cd /; sudo -u postgres pg_dumpall -w -h /var/run/postgresql -p 5432) | gzip > "$PG_DUMP.tmp"; then
|
|
mv "$PG_DUMP.tmp" "$PG_DUMP"
|
|
log "pg_dumpall done ($(du -h "$PG_DUMP" | cut -f1))"
|
|
else
|
|
error "pg_dumpall failed (exit $?); keeping previous dump if any"
|
|
rm -f "$PG_DUMP.tmp"
|
|
fi
|
|
else
|
|
error "postgres not ready on :5432 — skipping pg_dumpall"
|
|
fi
|
|
|
|
# ---- MongoDB -----------------------------------------------------------
|
|
MONGO_DIR="$STAGE/mongodump"
|
|
if mongosh --quiet --eval 'db.adminCommand({ping: 1}).ok' | grep -q '^1$'; then
|
|
log "mongodump starting → $MONGO_DIR"
|
|
rm -rf "$MONGO_DIR.tmp"
|
|
if mongodump --out "$MONGO_DIR.tmp" > /dev/null 2>&1; then
|
|
rm -rf "$MONGO_DIR"
|
|
mv "$MONGO_DIR.tmp" "$MONGO_DIR"
|
|
log "mongodump done ($(du -sh "$MONGO_DIR" | cut -f1))"
|
|
else
|
|
error "mongodump failed (exit $?); keeping previous dump if any"
|
|
rm -rf "$MONGO_DIR.tmp"
|
|
fi
|
|
else
|
|
error "mongo not reachable via mongosh — skipping mongodump"
|
|
fi
|
|
|
|
# ---- Retention on stage dir --------------------------------------------
|
|
# restic dedupes identical dumps at the chunk level, so we can safely keep
|
|
# overwriting the same files. No explicit rotation needed here.
|
|
|
|
if [ "$ERRORS" -ne 0 ]; then
|
|
log "pre-backup FAILED: $ERRORS required dump(s) failed"
|
|
exit 1
|
|
fi
|
|
log "pre-backup complete"
|