vh
d8f59a15d9
esh-ml1 is outside vzdump, so augaman's face gallery reaches backup only through restic. New playbooks/esh-ml1-restic.yaml installs restic 0.14.0 (the same Debian package as the other ESH hosts) and resticprofile 0.33.1 (pinned, sha256-checked). It uploads configs/restic/esh-ml1/ and schedules a daily 0100 PT backup plus a Sunday 0500 PT check to rest-server-ana. The CT runs UTC, so both schedules name the zone explicitly. pre-backup.sh is fail-closed: it runs augaman's own backup CLI, and any failure, including a stopped container, aborts the run. Tested with a stub docker that exits 1: the run returned 1, and neither the snapshot count nor last-success moved. The restore was verified at identity level against augaman-dev's public-domain canary (snapshot fd3061a1: the restored copy's digest over identities and samples matches the live gallery). That meets the operator gate for real enrollments. The repository URL is read through repository-file rather than restic.env. resticprofile schedule copies env-file values into world-readable systemd units, which publishes the rest-server password on the env-file hosts (observed on esh-docker-vm). This is recorded in the backups runbook under Known gaps, and the playbook verifies no generated unit contains the URL. esh-ml1 is added to the freshness check's expected ana-side repos and to the runbook tables. augaman moves to v0.1.2 (dependency layer keyed on the lock without the project; per-crop embedding). pytest -m gpu tests/vision passes 3/3 on the card, and the canary survived the container recreate.
restic / esh-ml1
ESH GPU LXC (CT 110 on esh-pve). Outside vzdump on purpose: everything on it is rebuilt from playbooks and stacks. The exception is augaman's face gallery (biometric data, not reconstructable), and this restic profile exists for it.
Writes cross-site to rest-server-ana at 10.250.50.70:8000/esh-ml1/, user
esh-ml1. Wired 2026-09-27 by playbooks/esh-ml1-restic.yaml.
What's backed up
| Path | Purpose |
|---|---|
/var/lib/restic/stage |
augaman/gallery.db, written by pre-backup.sh each run |
/opt/docker/compose |
the deployed compose files and .envs (a few KB) |
Daily 0100 PT, weekly check Sun 0500 PT. The CT's clock is UTC, so both
schedules carry America/Los_Angeles explicitly.
pre-backup.sh is fail-closed
It runs augaman's own backup CLI (docker exec augaman python -m augaman.gallery.backup ...: SQLite backup API, integrity_check, atomic rename,
mode 0600), then checks that the staged file is non-empty and under 5 minutes
old. Any failure, including the augaman container being stopped, aborts the
run: no snapshot, no last-success, and the fleet freshness check alerts.
Tested by making it fail (2026-09-27): with a stub docker that exits 1 on PATH,
resticprofile backup returned 1, the snapshot count stayed 1 → 1, and
last-success did not move.
⚠ Deviation from the other hosts: repository-file, not restic.env
resticprofile schedule copies env-file values into the generated systemd
units, which are world-readable. On the env-file hosts that publishes the
rest-server password to every local user (see docs/runbooks/backups.md →
Known gaps). Here the URL lives in /etc/restic/repository (root 0400) and the
unit carries only the path. The playbook's verify step fails if a unit ever
contains rest:http.
Secrets (seeded by hand; never in this repo)
| host file | vault item |
|---|---|
/etc/restic/password (root 0400), encryption passphrase |
esh-ml1/etc/restic/password |
/etc/restic/repository (root 0400), rest:http://esh-ml1:<pw>@10.250.50.70:8000/esh-ml1/ |
esh-ml1/etc/restic/repository |
To re-seed a rebuilt CT from the vault (from nh3-dev):
printf '%s\n' "$(secret get esh-ml1/etc/restic/password)" \
| ssh infra-ops@10.0.50.80 'sudo install -d -m 0755 /etc/restic && sudo install -o root -g root -m 0400 /dev/stdin /etc/restic/password'
printf '%s\n' "$(secret get esh-ml1/etc/restic/repository)" \
| ssh infra-ops@10.0.50.80 'sudo install -o root -g root -m 0400 /dev/stdin /etc/restic/repository'
scripts/elway esh-ml1 --playbook playbooks/esh-ml1-restic.yaml
The rest-server side is a bcrypt line for esh-ml1 in
/mnt/backup/restic/repo/ana/.htpasswd on ana-docker. That file is on
root-squashed NFS, so edit it as lkraven (ssh ana-docker), not with sudo.
rest-server picked up the new line without a restart. The NH3 Synology's
.htpasswd does not have an esh-ml1 line yet, so a DR move of this client
to rest-server-nh3 needs one added first.
Restoring the gallery
sudo -n bash -c 'R="restic --repository-file /etc/restic/repository --password-file /etc/restic/password"
t=$(mktemp -d); $R restore latest --target $t --include /var/lib/restic/stage/augaman/gallery.db
ls -l $t/var/lib/restic/stage/augaman/gallery.db'
The live gallery is the augaman_gallery named volume (/data/gallery.db in
the container). Putting a restored copy back is augaman's procedure; ask
augaman-dev before overwriting a live gallery.
Verified 2026-09-27, two ways.
- Mechanics, against snapshot
2685d3f6(empty gallery): the restored file was byte-identical to the staged one (sha256), andintegrity_checkwas ok. - Identities, against snapshot
fd3061a1, taken while augaman-dev's public-domain canary was enrolled (1 identity, 3 samples). The restored copy and the live gallery gave the same sha256 digest over every identity row (id, name_key, created_at) and every sample (id, embedder, embedding), andintegrity_checkwas ok.
Together these satisfy the operator ruling's gate (a scheduled off-box backup
plus one verified restore) before real enrollments. To re-run the check, digest
both copies the same way. The empty snapshot 2685d3f6 is a ready-made
negative control: it must NOT match a non-empty live gallery.