vh
6e203dcb99
resticprofile schedule copies env-file values into the generated units, which
are 0644, so RESTIC_REPOSITORY (the rest-server basic-auth password included)
was readable by every local user on every restic host.
New playbooks/restic-repository-file.yaml:
- derives /etc/restic/repository (root 0400) from restic.env;
- uploads the profile switched to repository-file, but only when the live
profile's sha matches the repo copy it was edited from (drift guard);
- checks the repository is reachable through the new profile (cat config);
- regenerates the units and verifies they exist and contain no rest:http.
Applied to ana-docker, fv-ml1 (configs/restic/ana-ml2), esh-docker-vm,
esh-vm-db, irv-ml1, nh3-dev and nh3-docker. An independent check across all
eight restic hosts (these seven plus esh-ml1) found 0 leaking units. nh3-docker's
scheduled unit ran a real backup afterwards (snapshot a29b889d). Each host's
URL and passphrase are vaulted as <host>/etc/restic/{repository,password}.
restic.env is kept (root 0600) because the per-host READMEs and the freshness
probe source it. A rotation must update the vault, restic.env and repository.
vm-esh-nas has no infra-ops account. Its in-place migration script is staged
for Prime to run with sudo, and its repo profile is pre-edited to match.
Also mirrors augaman-dev's 401df2d (compose header only). The config hash on
esh-ml1 is unchanged.
4.3 KiB
restic / esh-ml1
ESH GPU LXC (CT 110 on esh-pve). Outside vzdump on purpose: everything on it is rebuilt from playbooks and stacks. The exception is augaman's face gallery (biometric data, not reconstructable), and this restic profile exists for it.
Writes cross-site to rest-server-ana at 10.250.50.70:8000/esh-ml1/, user
esh-ml1. Wired 2026-09-27 by playbooks/esh-ml1-restic.yaml.
What's backed up
| Path | Purpose |
|---|---|
/var/lib/restic/stage |
augaman/gallery.db, written by pre-backup.sh each run |
/opt/docker/compose |
the deployed compose files and .envs (a few KB) |
Daily 0100 PT, weekly check Sun 0500 PT. The CT's clock is UTC, so both
schedules carry America/Los_Angeles explicitly.
pre-backup.sh is fail-closed
It runs augaman's own backup CLI (docker exec augaman python -m augaman.gallery.backup ...: SQLite backup API, integrity_check, atomic rename,
mode 0600), then checks that the staged file is non-empty and under 5 minutes
old. Any failure, including the augaman container being stopped, aborts the
run: no snapshot, no last-success, and the fleet freshness check alerts.
Tested by making it fail (2026-09-27): with a stub docker that exits 1 on PATH,
resticprofile backup returned 1, the snapshot count stayed 1 → 1, and
last-success did not move.
repository-file, not restic.env (now the fleet standard)
resticprofile schedule copies env-file values into the generated systemd
units, which are world-readable. That published the rest-server password on
every other host until 2026-09-27, when the rest of the fleet was moved to this
pattern (see docs/runbooks/backups.md → Known gaps). Here the URL lives in
/etc/restic/repository (root 0400), and the unit carries only the path. The playbook's verify step fails if a unit ever
contains rest:http.
Secrets (seeded by hand; never in this repo)
| host file | vault item |
|---|---|
/etc/restic/password (root 0400), encryption passphrase |
esh-ml1/etc/restic/password |
/etc/restic/repository (root 0400), rest:http://esh-ml1:<pw>@10.250.50.70:8000/esh-ml1/ |
esh-ml1/etc/restic/repository |
To re-seed a rebuilt CT from the vault (from nh3-dev):
printf '%s\n' "$(secret get esh-ml1/etc/restic/password)" \
| ssh infra-ops@10.0.50.80 'sudo install -d -m 0755 /etc/restic && sudo install -o root -g root -m 0400 /dev/stdin /etc/restic/password'
printf '%s\n' "$(secret get esh-ml1/etc/restic/repository)" \
| ssh infra-ops@10.0.50.80 'sudo install -o root -g root -m 0400 /dev/stdin /etc/restic/repository'
scripts/elway esh-ml1 --playbook playbooks/esh-ml1-restic.yaml
The rest-server side is a bcrypt line for esh-ml1 in
/mnt/backup/restic/repo/ana/.htpasswd on ana-docker. That file is on
root-squashed NFS, so edit it as lkraven (ssh ana-docker), not with sudo.
rest-server picked up the new line without a restart. The NH3 Synology's
.htpasswd does not have an esh-ml1 line yet, so a DR move of this client
to rest-server-nh3 needs one added first.
Restoring the gallery
sudo -n bash -c 'R="restic --repository-file /etc/restic/repository --password-file /etc/restic/password"
t=$(mktemp -d); $R restore latest --target $t --include /var/lib/restic/stage/augaman/gallery.db
ls -l $t/var/lib/restic/stage/augaman/gallery.db'
The live gallery is the augaman_gallery named volume (/data/gallery.db in
the container). Putting a restored copy back is augaman's procedure; ask
augaman-dev before overwriting a live gallery.
Verified 2026-09-27, two ways.
- Mechanics, against snapshot
2685d3f6(empty gallery): the restored file was byte-identical to the staged one (sha256), andintegrity_checkwas ok. - Identities, against snapshot
fd3061a1, taken while augaman-dev's public-domain canary was enrolled (1 identity, 3 samples). The restored copy and the live gallery gave the same sha256 digest over every identity row (id, name_key, created_at) and every sample (id, embedder, embedding), andintegrity_checkwas ok.
Together these satisfy the operator ruling's gate (a scheduled off-box backup
plus one verified restore) before real enrollments. To re-run the check, digest
both copies the same way. The empty snapshot 2685d3f6 is a ready-made
negative control: it must NOT match a non-empty live gallery.