vh
6e203dcb99
resticprofile schedule copies env-file values into the generated units, which
are 0644, so RESTIC_REPOSITORY (the rest-server basic-auth password included)
was readable by every local user on every restic host.
New playbooks/restic-repository-file.yaml:
- derives /etc/restic/repository (root 0400) from restic.env;
- uploads the profile switched to repository-file, but only when the live
profile's sha matches the repo copy it was edited from (drift guard);
- checks the repository is reachable through the new profile (cat config);
- regenerates the units and verifies they exist and contain no rest:http.
Applied to ana-docker, fv-ml1 (configs/restic/ana-ml2), esh-docker-vm,
esh-vm-db, irv-ml1, nh3-dev and nh3-docker. An independent check across all
eight restic hosts (these seven plus esh-ml1) found 0 leaking units. nh3-docker's
scheduled unit ran a real backup afterwards (snapshot a29b889d). Each host's
URL and passphrase are vaulted as <host>/etc/restic/{repository,password}.
restic.env is kept (root 0600) because the per-host READMEs and the freshness
probe source it. A rotation must update the vault, restic.env and repository.
vm-esh-nas has no infra-ops account. Its in-place migration script is staged
for Prime to run with sudo, and its repo profile is pre-edited to match.
Also mirrors augaman-dev's 401df2d (compose header only). The config hash on
esh-ml1 is unchanged.
90 lines
4.3 KiB
Markdown
90 lines
4.3 KiB
Markdown
# restic / esh-ml1
|
|
|
|
ESH GPU LXC (CT 110 on esh-pve). **Outside vzdump on purpose**: everything on it
|
|
is rebuilt from playbooks and stacks. The exception is **augaman's face gallery**
|
|
(biometric data, not reconstructable), and this restic profile exists for it.
|
|
|
|
Writes cross-site to `rest-server-ana` at `10.250.50.70:8000/esh-ml1/`, user
|
|
`esh-ml1`. Wired 2026-09-27 by [`playbooks/esh-ml1-restic.yaml`](../../../playbooks/esh-ml1-restic.yaml).
|
|
|
|
## What's backed up
|
|
|
|
| Path | Purpose |
|
|
|---|---|
|
|
| `/var/lib/restic/stage` | `augaman/gallery.db`, written by `pre-backup.sh` each run |
|
|
| `/opt/docker/compose` | the deployed compose files and `.env`s (a few KB) |
|
|
|
|
Daily **0100 PT**, weekly check Sun **0500 PT**. The CT's clock is UTC, so both
|
|
schedules carry `America/Los_Angeles` explicitly.
|
|
|
|
## pre-backup.sh is fail-closed
|
|
|
|
It runs augaman's own backup CLI (`docker exec augaman python -m
|
|
augaman.gallery.backup ...`: SQLite backup API, `integrity_check`, atomic rename,
|
|
mode 0600), then checks that the staged file is non-empty and under 5 minutes
|
|
old. Any failure, **including the augaman container being stopped**, aborts the
|
|
run: no snapshot, no `last-success`, and the fleet freshness check alerts.
|
|
|
|
Tested by making it fail (2026-09-27): with a stub `docker` that exits 1 on PATH,
|
|
`resticprofile backup` returned 1, the snapshot count stayed 1 → 1, and
|
|
`last-success` did not move.
|
|
|
|
## `repository-file`, not `restic.env` (now the fleet standard)
|
|
|
|
`resticprofile schedule` copies `env-file` values into the generated systemd
|
|
units, which are world-readable. That published the rest-server password on
|
|
every other host until 2026-09-27, when the rest of the fleet was moved to this
|
|
pattern (see `docs/runbooks/backups.md` → Known gaps). Here the URL lives in
|
|
`/etc/restic/repository` (root 0400), and the unit carries only the path. The playbook's verify step fails if a unit ever
|
|
contains `rest:http`.
|
|
|
|
## Secrets (seeded by hand; never in this repo)
|
|
|
|
| host file | vault item |
|
|
|---|---|
|
|
| `/etc/restic/password` (root 0400), encryption passphrase | `esh-ml1/etc/restic/password` |
|
|
| `/etc/restic/repository` (root 0400), `rest:http://esh-ml1:<pw>@10.250.50.70:8000/esh-ml1/` | `esh-ml1/etc/restic/repository` |
|
|
|
|
To re-seed a rebuilt CT from the vault (from nh3-dev):
|
|
|
|
```bash
|
|
printf '%s\n' "$(secret get esh-ml1/etc/restic/password)" \
|
|
| ssh infra-ops@10.0.50.80 'sudo install -d -m 0755 /etc/restic && sudo install -o root -g root -m 0400 /dev/stdin /etc/restic/password'
|
|
printf '%s\n' "$(secret get esh-ml1/etc/restic/repository)" \
|
|
| ssh infra-ops@10.0.50.80 'sudo install -o root -g root -m 0400 /dev/stdin /etc/restic/repository'
|
|
scripts/elway esh-ml1 --playbook playbooks/esh-ml1-restic.yaml
|
|
```
|
|
|
|
The rest-server side is a bcrypt line for `esh-ml1` in
|
|
`/mnt/backup/restic/repo/ana/.htpasswd` on ana-docker. That file is on
|
|
root-squashed NFS, so edit it as `lkraven` (`ssh ana-docker`), not with sudo.
|
|
rest-server picked up the new line without a restart. The NH3 Synology's
|
|
`.htpasswd` does **not** have an `esh-ml1` line yet, so a DR move of this client
|
|
to rest-server-nh3 needs one added first.
|
|
|
|
## Restoring the gallery
|
|
|
|
```bash
|
|
sudo -n bash -c 'R="restic --repository-file /etc/restic/repository --password-file /etc/restic/password"
|
|
t=$(mktemp -d); $R restore latest --target $t --include /var/lib/restic/stage/augaman/gallery.db
|
|
ls -l $t/var/lib/restic/stage/augaman/gallery.db'
|
|
```
|
|
|
|
The live gallery is the `augaman_gallery` named volume (`/data/gallery.db` in
|
|
the container). Putting a restored copy back is augaman's procedure; ask
|
|
augaman-dev before overwriting a live gallery.
|
|
|
|
**Verified 2026-09-27, two ways.**
|
|
- **Mechanics**, against snapshot `2685d3f6` (empty gallery): the restored file
|
|
was byte-identical to the staged one (sha256), and `integrity_check` was ok.
|
|
- **Identities**, against snapshot `fd3061a1`, taken while augaman-dev's
|
|
public-domain canary was enrolled (1 identity, 3 samples). The restored copy
|
|
and the live gallery gave the same sha256 digest over every identity row
|
|
(id, name_key, created_at) and every sample (id, embedder, embedding), and
|
|
`integrity_check` was ok.
|
|
|
|
Together these satisfy the operator ruling's gate (a scheduled off-box backup
|
|
plus one verified restore) before real enrollments. To re-run the check, digest
|
|
both copies the same way. The empty snapshot `2685d3f6` is a ready-made
|
|
negative control: it must NOT match a non-empty live gallery.
|