feat(esh-ml1): restic backup of augaman's gallery; augaman v0.1.2
esh-ml1 is outside vzdump, so augaman's face gallery reaches backup only through restic. New playbooks/esh-ml1-restic.yaml installs restic 0.14.0 (the same Debian package as the other ESH hosts) and resticprofile 0.33.1 (pinned, sha256-checked). It uploads configs/restic/esh-ml1/ and schedules a daily 0100 PT backup plus a Sunday 0500 PT check to rest-server-ana. The CT runs UTC, so both schedules name the zone explicitly. pre-backup.sh is fail-closed: it runs augaman's own backup CLI, and any failure, including a stopped container, aborts the run. Tested with a stub docker that exits 1: the run returned 1, and neither the snapshot count nor last-success moved. The restore was verified at identity level against augaman-dev's public-domain canary (snapshot fd3061a1: the restored copy's digest over identities and samples matches the live gallery). That meets the operator gate for real enrollments. The repository URL is read through repository-file rather than restic.env. resticprofile schedule copies env-file values into world-readable systemd units, which publishes the rest-server password on the env-file hosts (observed on esh-docker-vm). This is recorded in the backups runbook under Known gaps, and the playbook verifies no generated unit contains the URL. esh-ml1 is added to the freshness check's expected ana-side repos and to the runbook tables. augaman moves to v0.1.2 (dependency layer keyed on the lock without the project; per-crop embedding). pytest -m gpu tests/vision passes 3/3 on the card, and the canary survived the container recreate.
This commit is contained in:
@@ -0,0 +1,89 @@
|
||||
# restic / esh-ml1
|
||||
|
||||
ESH GPU LXC (CT 110 on esh-pve). **Outside vzdump on purpose**: everything on it
|
||||
is rebuilt from playbooks and stacks. The exception is **augaman's face gallery**
|
||||
(biometric data, not reconstructable), and this restic profile exists for it.
|
||||
|
||||
Writes cross-site to `rest-server-ana` at `10.250.50.70:8000/esh-ml1/`, user
|
||||
`esh-ml1`. Wired 2026-09-27 by [`playbooks/esh-ml1-restic.yaml`](../../../playbooks/esh-ml1-restic.yaml).
|
||||
|
||||
## What's backed up
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `/var/lib/restic/stage` | `augaman/gallery.db`, written by `pre-backup.sh` each run |
|
||||
| `/opt/docker/compose` | the deployed compose files and `.env`s (a few KB) |
|
||||
|
||||
Daily **0100 PT**, weekly check Sun **0500 PT**. The CT's clock is UTC, so both
|
||||
schedules carry `America/Los_Angeles` explicitly.
|
||||
|
||||
## pre-backup.sh is fail-closed
|
||||
|
||||
It runs augaman's own backup CLI (`docker exec augaman python -m
|
||||
augaman.gallery.backup ...`: SQLite backup API, `integrity_check`, atomic rename,
|
||||
mode 0600), then checks that the staged file is non-empty and under 5 minutes
|
||||
old. Any failure, **including the augaman container being stopped**, aborts the
|
||||
run: no snapshot, no `last-success`, and the fleet freshness check alerts.
|
||||
|
||||
Tested by making it fail (2026-09-27): with a stub `docker` that exits 1 on PATH,
|
||||
`resticprofile backup` returned 1, the snapshot count stayed 1 → 1, and
|
||||
`last-success` did not move.
|
||||
|
||||
## ⚠ Deviation from the other hosts: `repository-file`, not `restic.env`
|
||||
|
||||
`resticprofile schedule` copies `env-file` values into the generated systemd
|
||||
units, which are world-readable. On the env-file hosts that publishes the
|
||||
rest-server password to every local user (see `docs/runbooks/backups.md` →
|
||||
Known gaps). Here the URL lives in `/etc/restic/repository` (root 0400) and the
|
||||
unit carries only the path. The playbook's verify step fails if a unit ever
|
||||
contains `rest:http`.
|
||||
|
||||
## Secrets (seeded by hand; never in this repo)
|
||||
|
||||
| host file | vault item |
|
||||
|---|---|
|
||||
| `/etc/restic/password` (root 0400), encryption passphrase | `esh-ml1/etc/restic/password` |
|
||||
| `/etc/restic/repository` (root 0400), `rest:http://esh-ml1:<pw>@10.250.50.70:8000/esh-ml1/` | `esh-ml1/etc/restic/repository` |
|
||||
|
||||
To re-seed a rebuilt CT from the vault (from nh3-dev):
|
||||
|
||||
```bash
|
||||
printf '%s\n' "$(secret get esh-ml1/etc/restic/password)" \
|
||||
| ssh infra-ops@10.0.50.80 'sudo install -d -m 0755 /etc/restic && sudo install -o root -g root -m 0400 /dev/stdin /etc/restic/password'
|
||||
printf '%s\n' "$(secret get esh-ml1/etc/restic/repository)" \
|
||||
| ssh infra-ops@10.0.50.80 'sudo install -o root -g root -m 0400 /dev/stdin /etc/restic/repository'
|
||||
scripts/elway esh-ml1 --playbook playbooks/esh-ml1-restic.yaml
|
||||
```
|
||||
|
||||
The rest-server side is a bcrypt line for `esh-ml1` in
|
||||
`/mnt/backup/restic/repo/ana/.htpasswd` on ana-docker. That file is on
|
||||
root-squashed NFS, so edit it as `lkraven` (`ssh ana-docker`), not with sudo.
|
||||
rest-server picked up the new line without a restart. The NH3 Synology's
|
||||
`.htpasswd` does **not** have an `esh-ml1` line yet, so a DR move of this client
|
||||
to rest-server-nh3 needs one added first.
|
||||
|
||||
## Restoring the gallery
|
||||
|
||||
```bash
|
||||
sudo -n bash -c 'R="restic --repository-file /etc/restic/repository --password-file /etc/restic/password"
|
||||
t=$(mktemp -d); $R restore latest --target $t --include /var/lib/restic/stage/augaman/gallery.db
|
||||
ls -l $t/var/lib/restic/stage/augaman/gallery.db'
|
||||
```
|
||||
|
||||
The live gallery is the `augaman_gallery` named volume (`/data/gallery.db` in
|
||||
the container). Putting a restored copy back is augaman's procedure; ask
|
||||
augaman-dev before overwriting a live gallery.
|
||||
|
||||
**Verified 2026-09-27, two ways.**
|
||||
- **Mechanics**, against snapshot `2685d3f6` (empty gallery): the restored file
|
||||
was byte-identical to the staged one (sha256), and `integrity_check` was ok.
|
||||
- **Identities**, against snapshot `fd3061a1`, taken while augaman-dev's
|
||||
public-domain canary was enrolled (1 identity, 3 samples). The restored copy
|
||||
and the live gallery gave the same sha256 digest over every identity row
|
||||
(id, name_key, created_at) and every sample (id, embedder, embedding), and
|
||||
`integrity_check` was ok.
|
||||
|
||||
Together these satisfy the operator ruling's gate (a scheduled off-box backup
|
||||
plus one verified restore) before real enrollments. To re-run the check, digest
|
||||
both copies the same way. The empty snapshot `2685d3f6` is a ready-made
|
||||
negative control: it must NOT match a non-empty live gallery.
|
||||
Executable
+35
@@ -0,0 +1,35 @@
|
||||
#!/usr/bin/env bash
|
||||
# pre-backup hook for esh-ml1 — deployed to /etc/restic/pre-backup.sh (root, 0755).
|
||||
#
|
||||
# Takes an online-consistent copy of augaman's face gallery with augaman's own
|
||||
# backup CLI (SQLite backup API, integrity_check, atomic rename, mode 0600) into
|
||||
# /var/lib/restic/stage/augaman/gallery.db, which restic then snapshots.
|
||||
#
|
||||
# FAIL-CLOSED. Any failure exits non-zero, which makes resticprofile abort the
|
||||
# run: no snapshot, no last-success stamp, and the fleet freshness check alerts.
|
||||
# That includes the augaman container being stopped. A gallery that is not being
|
||||
# backed up must be loud, not a quietly stale file in every snapshot.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
STAGE_FILE=/var/lib/restic/stage/augaman/gallery.db
|
||||
MAX_AGE_S=300
|
||||
|
||||
if ! out=$(docker exec augaman python -m augaman.gallery.backup \
|
||||
--db /data/gallery.db --dest /backup/gallery.db 2>&1); then
|
||||
echo "pre-backup: augaman gallery backup FAILED: ${out}" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "pre-backup: augaman gallery backup: ${out}"
|
||||
|
||||
# The CLI said it succeeded; confirm the file restic is about to read is the
|
||||
# one it just wrote (non-empty, and written within the last few minutes).
|
||||
if [ ! -s "$STAGE_FILE" ]; then
|
||||
echo "pre-backup: ${STAGE_FILE} is missing or empty after a successful CLI run" >&2
|
||||
exit 1
|
||||
fi
|
||||
age=$(( $(date +%s) - $(stat -c %Y "$STAGE_FILE") ))
|
||||
if [ "$age" -gt "$MAX_AGE_S" ]; then
|
||||
echo "pre-backup: ${STAGE_FILE} is ${age}s old; the CLI did not refresh it" >&2
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,71 @@
|
||||
# resticprofile config for esh-ml1.
|
||||
#
|
||||
# ESH GPU LXC (CT 110 on esh-pve). The host is OUTSIDE vzdump on purpose:
|
||||
# everything on it is rebuilt from playbooks + stacks, and models re-download.
|
||||
# The one exception is augaman's face gallery (biometric data, not
|
||||
# reconstructable), so this profile exists for that and little else.
|
||||
#
|
||||
# Writes cross-site to rest-server-ana (10.250.50.70:8000/esh-ml1/) because
|
||||
# the ESH site has no local rest-server (same as esh-docker-vm / esh-vm-db).
|
||||
#
|
||||
# ⚠ DELIBERATE DEVIATION from the other hosts: the repository URL (which
|
||||
# embeds the rest-server basic-auth password) is read from
|
||||
# /etc/restic/repository via `repository-file`, NOT from an env-file.
|
||||
# `resticprofile schedule` copies env-file values into the generated systemd
|
||||
# unit, and units under /etc/systemd/system are world-readable (0644), so
|
||||
# the env-file pattern publishes the credential to every local user. With
|
||||
# repository-file the unit carries only the path. See README.md.
|
||||
#
|
||||
# ⚠ The CT runs in UTC; the schedules carry an explicit zone so they fire at
|
||||
# the same wall-clock time as the rest of the fleet.
|
||||
|
||||
version: "1"
|
||||
|
||||
global:
|
||||
priority: low
|
||||
ionice: true
|
||||
ionice-class: 2
|
||||
ionice-level: 7
|
||||
min-memory: 100
|
||||
|
||||
default:
|
||||
repository-file: /etc/restic/repository
|
||||
password-file: /etc/restic/password
|
||||
initialize: false
|
||||
lock: /var/lock/restic-esh-ml1.lock
|
||||
|
||||
backup:
|
||||
verbose: 1
|
||||
run-before:
|
||||
# Fail-closed: if augaman's backup CLI fails (or the container is down),
|
||||
# the whole run fails, last-success is not written, and the freshness
|
||||
# check alerts. A stale gallery copy must never be snapshotted as current.
|
||||
- /etc/restic/pre-backup.sh
|
||||
run-after:
|
||||
- date +%s > /var/lib/restic/last-success
|
||||
source:
|
||||
- /var/lib/restic/stage # augaman/gallery.db, written by pre-backup.sh
|
||||
- /opt/docker/compose # the deployed compose files + .env (small)
|
||||
tag:
|
||||
- host:esh-ml1
|
||||
- site:esh
|
||||
- fleet:home-lab
|
||||
schedule: "*-*-* 01:00:00 America/Los_Angeles"
|
||||
schedule-permission: system
|
||||
schedule-log: /var/log/restic-backup.log
|
||||
|
||||
forget:
|
||||
keep-daily: 7
|
||||
keep-weekly: 4
|
||||
keep-monthly: 12
|
||||
keep-yearly: 3
|
||||
tag:
|
||||
- host:esh-ml1
|
||||
# No schedule: rest-server-ana is --append-only, so forget always fails
|
||||
# from the client. It runs by hand during the prune ceremony.
|
||||
|
||||
check:
|
||||
read-data-subset: 10%
|
||||
schedule: "Sun *-*-* 05:00:00 America/Los_Angeles"
|
||||
schedule-permission: system
|
||||
schedule-log: /var/log/restic-check.log
|
||||
Reference in New Issue
Block a user