vh
d8f59a15d9
esh-ml1 is outside vzdump, so augaman's face gallery reaches backup only through restic. New playbooks/esh-ml1-restic.yaml installs restic 0.14.0 (the same Debian package as the other ESH hosts) and resticprofile 0.33.1 (pinned, sha256-checked). It uploads configs/restic/esh-ml1/ and schedules a daily 0100 PT backup plus a Sunday 0500 PT check to rest-server-ana. The CT runs UTC, so both schedules name the zone explicitly. pre-backup.sh is fail-closed: it runs augaman's own backup CLI, and any failure, including a stopped container, aborts the run. Tested with a stub docker that exits 1: the run returned 1, and neither the snapshot count nor last-success moved. The restore was verified at identity level against augaman-dev's public-domain canary (snapshot fd3061a1: the restored copy's digest over identities and samples matches the live gallery). That meets the operator gate for real enrollments. The repository URL is read through repository-file rather than restic.env. resticprofile schedule copies env-file values into world-readable systemd units, which publishes the rest-server password on the env-file hosts (observed on esh-docker-vm). This is recorded in the backups runbook under Known gaps, and the playbook verifies no generated unit contains the URL. esh-ml1 is added to the freshness check's expected ana-side repos and to the runbook tables. augaman moves to v0.1.2 (dependency layer keyed on the lock without the project; per-crop embedding). pytest -m gpu tests/vision passes 3/3 on the card, and the canary survived the container recreate.
36 lines
1.4 KiB
Bash
Executable File
36 lines
1.4 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# pre-backup hook for esh-ml1 — deployed to /etc/restic/pre-backup.sh (root, 0755).
|
|
#
|
|
# Takes an online-consistent copy of augaman's face gallery with augaman's own
|
|
# backup CLI (SQLite backup API, integrity_check, atomic rename, mode 0600) into
|
|
# /var/lib/restic/stage/augaman/gallery.db, which restic then snapshots.
|
|
#
|
|
# FAIL-CLOSED. Any failure exits non-zero, which makes resticprofile abort the
|
|
# run: no snapshot, no last-success stamp, and the fleet freshness check alerts.
|
|
# That includes the augaman container being stopped. A gallery that is not being
|
|
# backed up must be loud, not a quietly stale file in every snapshot.
|
|
|
|
set -euo pipefail
|
|
|
|
STAGE_FILE=/var/lib/restic/stage/augaman/gallery.db
|
|
MAX_AGE_S=300
|
|
|
|
if ! out=$(docker exec augaman python -m augaman.gallery.backup \
|
|
--db /data/gallery.db --dest /backup/gallery.db 2>&1); then
|
|
echo "pre-backup: augaman gallery backup FAILED: ${out}" >&2
|
|
exit 1
|
|
fi
|
|
echo "pre-backup: augaman gallery backup: ${out}"
|
|
|
|
# The CLI said it succeeded; confirm the file restic is about to read is the
|
|
# one it just wrote (non-empty, and written within the last few minutes).
|
|
if [ ! -s "$STAGE_FILE" ]; then
|
|
echo "pre-backup: ${STAGE_FILE} is missing or empty after a successful CLI run" >&2
|
|
exit 1
|
|
fi
|
|
age=$(( $(date +%s) - $(stat -c %Y "$STAGE_FILE") ))
|
|
if [ "$age" -gt "$MAX_AGE_S" ]; then
|
|
echo "pre-backup: ${STAGE_FILE} is ${age}s old; the CLI did not refresh it" >&2
|
|
exit 1
|
|
fi
|