feat(esh-ml1): restic backup of augaman's gallery; augaman v0.1.2
esh-ml1 is outside vzdump, so augaman's face gallery reaches backup only through restic. New playbooks/esh-ml1-restic.yaml installs restic 0.14.0 (the same Debian package as the other ESH hosts) and resticprofile 0.33.1 (pinned, sha256-checked). It uploads configs/restic/esh-ml1/ and schedules a daily 0100 PT backup plus a Sunday 0500 PT check to rest-server-ana. The CT runs UTC, so both schedules name the zone explicitly. pre-backup.sh is fail-closed: it runs augaman's own backup CLI, and any failure, including a stopped container, aborts the run. Tested with a stub docker that exits 1: the run returned 1, and neither the snapshot count nor last-success moved. The restore was verified at identity level against augaman-dev's public-domain canary (snapshot fd3061a1: the restored copy's digest over identities and samples matches the live gallery). That meets the operator gate for real enrollments. The repository URL is read through repository-file rather than restic.env. resticprofile schedule copies env-file values into world-readable systemd units, which publishes the rest-server password on the env-file hosts (observed on esh-docker-vm). This is recorded in the backups runbook under Known gaps, and the playbook verifies no generated unit contains the URL. esh-ml1 is added to the freshness check's expected ana-side repos and to the runbook tables. augaman moves to v0.1.2 (dependency layer keyed on the lock without the project; per-crop embedding). pytest -m gpu tests/vision passes 3/3 on the card, and the canary survived the container recreate.
This commit is contained in:
@@ -0,0 +1,89 @@
|
||||
# restic / esh-ml1
|
||||
|
||||
ESH GPU LXC (CT 110 on esh-pve). **Outside vzdump on purpose**: everything on it
|
||||
is rebuilt from playbooks and stacks. The exception is **augaman's face gallery**
|
||||
(biometric data, not reconstructable), and this restic profile exists for it.
|
||||
|
||||
Writes cross-site to `rest-server-ana` at `10.250.50.70:8000/esh-ml1/`, user
|
||||
`esh-ml1`. Wired 2026-09-27 by [`playbooks/esh-ml1-restic.yaml`](../../../playbooks/esh-ml1-restic.yaml).
|
||||
|
||||
## What's backed up
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `/var/lib/restic/stage` | `augaman/gallery.db`, written by `pre-backup.sh` each run |
|
||||
| `/opt/docker/compose` | the deployed compose files and `.env`s (a few KB) |
|
||||
|
||||
Daily **0100 PT**, weekly check Sun **0500 PT**. The CT's clock is UTC, so both
|
||||
schedules carry `America/Los_Angeles` explicitly.
|
||||
|
||||
## pre-backup.sh is fail-closed
|
||||
|
||||
It runs augaman's own backup CLI (`docker exec augaman python -m
|
||||
augaman.gallery.backup ...`: SQLite backup API, `integrity_check`, atomic rename,
|
||||
mode 0600), then checks that the staged file is non-empty and under 5 minutes
|
||||
old. Any failure, **including the augaman container being stopped**, aborts the
|
||||
run: no snapshot, no `last-success`, and the fleet freshness check alerts.
|
||||
|
||||
Tested by making it fail (2026-09-27): with a stub `docker` that exits 1 on PATH,
|
||||
`resticprofile backup` returned 1, the snapshot count stayed 1 → 1, and
|
||||
`last-success` did not move.
|
||||
|
||||
## ⚠ Deviation from the other hosts: `repository-file`, not `restic.env`
|
||||
|
||||
`resticprofile schedule` copies `env-file` values into the generated systemd
|
||||
units, which are world-readable. On the env-file hosts that publishes the
|
||||
rest-server password to every local user (see `docs/runbooks/backups.md` →
|
||||
Known gaps). Here the URL lives in `/etc/restic/repository` (root 0400) and the
|
||||
unit carries only the path. The playbook's verify step fails if a unit ever
|
||||
contains `rest:http`.
|
||||
|
||||
## Secrets (seeded by hand; never in this repo)
|
||||
|
||||
| host file | vault item |
|
||||
|---|---|
|
||||
| `/etc/restic/password` (root 0400), encryption passphrase | `esh-ml1/etc/restic/password` |
|
||||
| `/etc/restic/repository` (root 0400), `rest:http://esh-ml1:<pw>@10.250.50.70:8000/esh-ml1/` | `esh-ml1/etc/restic/repository` |
|
||||
|
||||
To re-seed a rebuilt CT from the vault (from nh3-dev):
|
||||
|
||||
```bash
|
||||
printf '%s\n' "$(secret get esh-ml1/etc/restic/password)" \
|
||||
| ssh infra-ops@10.0.50.80 'sudo install -d -m 0755 /etc/restic && sudo install -o root -g root -m 0400 /dev/stdin /etc/restic/password'
|
||||
printf '%s\n' "$(secret get esh-ml1/etc/restic/repository)" \
|
||||
| ssh infra-ops@10.0.50.80 'sudo install -o root -g root -m 0400 /dev/stdin /etc/restic/repository'
|
||||
scripts/elway esh-ml1 --playbook playbooks/esh-ml1-restic.yaml
|
||||
```
|
||||
|
||||
The rest-server side is a bcrypt line for `esh-ml1` in
|
||||
`/mnt/backup/restic/repo/ana/.htpasswd` on ana-docker. That file is on
|
||||
root-squashed NFS, so edit it as `lkraven` (`ssh ana-docker`), not with sudo.
|
||||
rest-server picked up the new line without a restart. The NH3 Synology's
|
||||
`.htpasswd` does **not** have an `esh-ml1` line yet, so a DR move of this client
|
||||
to rest-server-nh3 needs one added first.
|
||||
|
||||
## Restoring the gallery
|
||||
|
||||
```bash
|
||||
sudo -n bash -c 'R="restic --repository-file /etc/restic/repository --password-file /etc/restic/password"
|
||||
t=$(mktemp -d); $R restore latest --target $t --include /var/lib/restic/stage/augaman/gallery.db
|
||||
ls -l $t/var/lib/restic/stage/augaman/gallery.db'
|
||||
```
|
||||
|
||||
The live gallery is the `augaman_gallery` named volume (`/data/gallery.db` in
|
||||
the container). Putting a restored copy back is augaman's procedure; ask
|
||||
augaman-dev before overwriting a live gallery.
|
||||
|
||||
**Verified 2026-09-27, two ways.**
|
||||
- **Mechanics**, against snapshot `2685d3f6` (empty gallery): the restored file
|
||||
was byte-identical to the staged one (sha256), and `integrity_check` was ok.
|
||||
- **Identities**, against snapshot `fd3061a1`, taken while augaman-dev's
|
||||
public-domain canary was enrolled (1 identity, 3 samples). The restored copy
|
||||
and the live gallery gave the same sha256 digest over every identity row
|
||||
(id, name_key, created_at) and every sample (id, embedder, embedding), and
|
||||
`integrity_check` was ok.
|
||||
|
||||
Together these satisfy the operator ruling's gate (a scheduled off-box backup
|
||||
plus one verified restore) before real enrollments. To re-run the check, digest
|
||||
both copies the same way. The empty snapshot `2685d3f6` is a ready-made
|
||||
negative control: it must NOT match a non-empty live gallery.
|
||||
Executable
+35
@@ -0,0 +1,35 @@
|
||||
#!/usr/bin/env bash
|
||||
# pre-backup hook for esh-ml1 — deployed to /etc/restic/pre-backup.sh (root, 0755).
|
||||
#
|
||||
# Takes an online-consistent copy of augaman's face gallery with augaman's own
|
||||
# backup CLI (SQLite backup API, integrity_check, atomic rename, mode 0600) into
|
||||
# /var/lib/restic/stage/augaman/gallery.db, which restic then snapshots.
|
||||
#
|
||||
# FAIL-CLOSED. Any failure exits non-zero, which makes resticprofile abort the
|
||||
# run: no snapshot, no last-success stamp, and the fleet freshness check alerts.
|
||||
# That includes the augaman container being stopped. A gallery that is not being
|
||||
# backed up must be loud, not a quietly stale file in every snapshot.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
STAGE_FILE=/var/lib/restic/stage/augaman/gallery.db
|
||||
MAX_AGE_S=300
|
||||
|
||||
if ! out=$(docker exec augaman python -m augaman.gallery.backup \
|
||||
--db /data/gallery.db --dest /backup/gallery.db 2>&1); then
|
||||
echo "pre-backup: augaman gallery backup FAILED: ${out}" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "pre-backup: augaman gallery backup: ${out}"
|
||||
|
||||
# The CLI said it succeeded; confirm the file restic is about to read is the
|
||||
# one it just wrote (non-empty, and written within the last few minutes).
|
||||
if [ ! -s "$STAGE_FILE" ]; then
|
||||
echo "pre-backup: ${STAGE_FILE} is missing or empty after a successful CLI run" >&2
|
||||
exit 1
|
||||
fi
|
||||
age=$(( $(date +%s) - $(stat -c %Y "$STAGE_FILE") ))
|
||||
if [ "$age" -gt "$MAX_AGE_S" ]; then
|
||||
echo "pre-backup: ${STAGE_FILE} is ${age}s old; the CLI did not refresh it" >&2
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,71 @@
|
||||
# resticprofile config for esh-ml1.
|
||||
#
|
||||
# ESH GPU LXC (CT 110 on esh-pve). The host is OUTSIDE vzdump on purpose:
|
||||
# everything on it is rebuilt from playbooks + stacks, and models re-download.
|
||||
# The one exception is augaman's face gallery (biometric data, not
|
||||
# reconstructable), so this profile exists for that and little else.
|
||||
#
|
||||
# Writes cross-site to rest-server-ana (10.250.50.70:8000/esh-ml1/) because
|
||||
# the ESH site has no local rest-server (same as esh-docker-vm / esh-vm-db).
|
||||
#
|
||||
# ⚠ DELIBERATE DEVIATION from the other hosts: the repository URL (which
|
||||
# embeds the rest-server basic-auth password) is read from
|
||||
# /etc/restic/repository via `repository-file`, NOT from an env-file.
|
||||
# `resticprofile schedule` copies env-file values into the generated systemd
|
||||
# unit, and units under /etc/systemd/system are world-readable (0644), so
|
||||
# the env-file pattern publishes the credential to every local user. With
|
||||
# repository-file the unit carries only the path. See README.md.
|
||||
#
|
||||
# ⚠ The CT runs in UTC; the schedules carry an explicit zone so they fire at
|
||||
# the same wall-clock time as the rest of the fleet.
|
||||
|
||||
version: "1"
|
||||
|
||||
global:
|
||||
priority: low
|
||||
ionice: true
|
||||
ionice-class: 2
|
||||
ionice-level: 7
|
||||
min-memory: 100
|
||||
|
||||
default:
|
||||
repository-file: /etc/restic/repository
|
||||
password-file: /etc/restic/password
|
||||
initialize: false
|
||||
lock: /var/lock/restic-esh-ml1.lock
|
||||
|
||||
backup:
|
||||
verbose: 1
|
||||
run-before:
|
||||
# Fail-closed: if augaman's backup CLI fails (or the container is down),
|
||||
# the whole run fails, last-success is not written, and the freshness
|
||||
# check alerts. A stale gallery copy must never be snapshotted as current.
|
||||
- /etc/restic/pre-backup.sh
|
||||
run-after:
|
||||
- date +%s > /var/lib/restic/last-success
|
||||
source:
|
||||
- /var/lib/restic/stage # augaman/gallery.db, written by pre-backup.sh
|
||||
- /opt/docker/compose # the deployed compose files + .env (small)
|
||||
tag:
|
||||
- host:esh-ml1
|
||||
- site:esh
|
||||
- fleet:home-lab
|
||||
schedule: "*-*-* 01:00:00 America/Los_Angeles"
|
||||
schedule-permission: system
|
||||
schedule-log: /var/log/restic-backup.log
|
||||
|
||||
forget:
|
||||
keep-daily: 7
|
||||
keep-weekly: 4
|
||||
keep-monthly: 12
|
||||
keep-yearly: 3
|
||||
tag:
|
||||
- host:esh-ml1
|
||||
# No schedule: rest-server-ana is --append-only, so forget always fails
|
||||
# from the client. It runs by hand during the prune ceremony.
|
||||
|
||||
check:
|
||||
read-data-subset: 10%
|
||||
schedule: "Sun *-*-* 05:00:00 America/Los_Angeles"
|
||||
schedule-permission: system
|
||||
schedule-log: /var/log/restic-check.log
|
||||
@@ -29,6 +29,7 @@ layer has failed.
|
||||
| nh3-docker | VM (nh3-pve) | ✅ `nh3-pve` | ✅ → rest-server-**nh3** | no |
|
||||
| esh-docker-vm | VM (esh-pve) | ✅ `esh-pve` | ✅ → rest-server-**ana** | no |
|
||||
| esh-vm-db | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only (a DB!)** |
|
||||
| **esh-ml1** | CT 110 (esh-pve) | ❌ none (**outside vzdump on purpose**: rebuilt from playbooks) | ✅ → rest-server-**ana** (augaman gallery + compose dir only) | ⚠️ **restic is the ONLY net for augaman's biometric gallery** |
|
||||
| vm-esh-nas | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only** |
|
||||
| other pfi-pve / nh3-pve VMs/CTs | VM/CT | ✅ respective ns | (PBS only) | no |
|
||||
| SureFire `sfsrv-pve` | tenant VMs | ✅ `sfsrv-pve` ns | (PBS only) | no |
|
||||
@@ -64,7 +65,7 @@ auth, append-only, private repos). The split is by site:
|
||||
|
||||
| rest-server | Endpoint | Backing store | Clients |
|
||||
|---|---|---|---|
|
||||
| **rest-server-ana** | `http://10.250.50.70:8000` (container `rest-server` on ana-docker) | `ana-nas:/mnt/backup/restic/repo/ana` (NFS bind → `/data`) | ana-docker, **ana-ml2**, esh-docker-vm, esh-vm-db, vm-esh-nas |
|
||||
| **rest-server-ana** | `http://10.250.50.70:8000` (container `rest-server` on ana-docker) | `ana-nas:/mnt/backup/restic/repo/ana` (NFS bind → `/data`) | ana-docker, **ana-ml2**, esh-docker-vm, esh-ml1, esh-vm-db, vm-esh-nas |
|
||||
| **rest-server-nh3** | `http://10.100.50.50:8000` (on nh3-nas) | `nh3-nas:/volume1/Backup/restic/<client>` | **irv-ml1**, nh3-docker |
|
||||
|
||||
- Per-client repos live as subdirs of the rest-server data dir
|
||||
@@ -115,7 +116,7 @@ Run these any time you need to answer "are we backed up?"
|
||||
|
||||
```bash
|
||||
# --- restic ANA side: newest snapshot per client (want: today/yesterday) ---
|
||||
ssh ana-nas 'for c in ana-docker ana-ml2 esh-docker-vm esh-vm-db vm-esh-nas; do
|
||||
ssh ana-nas 'for c in ana-docker ana-ml2 esh-docker-vm esh-ml1 esh-vm-db vm-esh-nas; do
|
||||
echo -n "$c: "; ls -t /mnt/backup/restic/repo/ana/$c/snapshots/ 2>/dev/null | head -1 \
|
||||
| xargs -I{} stat -c "%y" /mnt/backup/restic/repo/ana/$c/snapshots/{} 2>/dev/null || echo MISSING
|
||||
done'
|
||||
@@ -177,6 +178,25 @@ the stop→umount→rm-ghost→remount→start variant.
|
||||
|
||||
## Known gaps / TODO
|
||||
|
||||
### ⚠ `resticprofile schedule` publishes the repo credential (found 2026-09-27)
|
||||
|
||||
On a host whose profile loads `RESTIC_REPOSITORY` from `env-file:
|
||||
/etc/restic/restic.env`, `resticprofile schedule` copies that value, **including
|
||||
the embedded rest-server basic-auth password**, into
|
||||
`/etc/systemd/system/resticprofile-*@profile-default.service` as an
|
||||
`Environment=` line. Those units are world-readable, so `systemctl cat` shows the
|
||||
credential to any local user; the 0600 on `restic.env` protects nothing.
|
||||
Observed on esh-docker-vm (read without sudo). Blast radius: the credential lets
|
||||
a caller read the encrypted blobs and append to that one repo (rest-server is
|
||||
`--append-only`, and the encryption passphrase is a separate file). It does not
|
||||
decrypt anything.
|
||||
|
||||
**esh-ml1 does not have this problem:** its profile uses `repository-file:
|
||||
/etc/restic/repository` (root 0400), so the unit carries only the path, and
|
||||
`playbooks/esh-ml1-restic.yaml` verifies that no unit contains `rest:http`.
|
||||
Moving the other hosts to `repository-file` is not done; it belongs with the
|
||||
(operator-held) rest-server password rotation below.
|
||||
|
||||
### ✅ restic content assertion (2026-09-22, CLOSED)
|
||||
|
||||
⚠ **I reported this gap wrongly first.** I ran `grep -ic restic` against
|
||||
|
||||
@@ -144,13 +144,13 @@ _As of 2026-09-26 ~1620 PT._
|
||||
|
||||
### augaman: face recognition for Cicada
|
||||
|
||||
- **v0.1.1 DEPLOYED on esh-ml1:8040 2026-09-26 2347 PT** (`stacks/augaman`),
|
||||
healthy on CUDA, in `nvidia-smi`. Built on-box from a `git archive` of the tag.
|
||||
- **infra-ops owes the gallery backup NEXT:** esh-ml1 has no restic.
|
||||
`BACKUP_DIR=/var/lib/restic/stage/augaman` exists; needs restic + timer running
|
||||
the backup CLI, then one verified restore. **Fixtures only until then** (Prime).
|
||||
- `pytest -m gpu` FAILS 3/3 on the batch-vs-single tolerance (≥6e-5 vs 1e-5);
|
||||
placement passes. Reported to augaman-dev: the contract is theirs to rule on.
|
||||
- **v0.1.2 LIVE on esh-ml1:8040** (2026-09-27; v0.1.1 first deployed 2026-09-26 2347 PT),
|
||||
`stacks/augaman`, healthy on CUDA, in `nvidia-smi`. Built on-box from a `git archive`
|
||||
of the tag. `pytest -m gpu tests/vision` 3/3 PASS on v0.1.2.
|
||||
- **Gallery backup WIRED + RESTORE-VERIFIED 2026-09-27** (Prime OK'd off-site): restic
|
||||
daily 0100 PT → rest-server-ana `esh-ml1/`, fail-closed hook, secrets vaulted under
|
||||
`esh-ml1/etc/restic/`. Identity-level restore matched the live gallery (canary,
|
||||
snapshot `fd3061a1`). **The backup gate for real enrollments is MET.**
|
||||
|
||||
### esh-ml1
|
||||
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
# esh-ml1 restic: backs up augaman's face gallery (and the deployed compose dir)
|
||||
# to rest-server-ana. Config + rationale: configs/restic/esh-ml1/.
|
||||
#
|
||||
# Run: scripts/elway esh-ml1 --playbook playbooks/esh-ml1-restic.yaml
|
||||
#
|
||||
# PRE-REQUISITE, NOT DONE HERE (secrets never live in this repo): seed the two
|
||||
# root-only secret files from the vault, and the rest-server-ana htpasswd entry
|
||||
# for user `esh-ml1`. The exact commands are in configs/restic/esh-ml1/README.md.
|
||||
# The "Secrets are seeded" step below halts the run if they are missing.
|
||||
#
|
||||
# Idempotent: a second run should report ok/skipped everywhere except the
|
||||
# re-uploads and the schedule refresh.
|
||||
|
||||
vars:
|
||||
resticprofile_version: 0.33.1
|
||||
resticprofile_sha256: 5fecd20de21811a750a4d61e841b2258fdf018bbfffad136c96078df27b452df
|
||||
|
||||
steps:
|
||||
- name: Install restic (Debian 12 package, 0.14.0, same as esh-docker-vm and esh-vm-db)
|
||||
sudo: true
|
||||
shell: DEBIAN_FRONTEND=noninteractive apt-get install -y -q restic
|
||||
when: "! command -v restic >/dev/null"
|
||||
|
||||
- name: Install resticprofile {{ resticprofile_version }} (pinned, sha256-checked)
|
||||
sudo: true
|
||||
shell: |
|
||||
set -eu
|
||||
tmp=$(mktemp -d)
|
||||
trap 'rm -rf "$tmp"' EXIT
|
||||
tgz="resticprofile_{{ resticprofile_version }}_linux_amd64.tar.gz"
|
||||
curl -fsSL -o "$tmp/$tgz" "https://github.com/creativeprojects/resticprofile/releases/download/v{{ resticprofile_version }}/$tgz"
|
||||
echo "{{ resticprofile_sha256 }} $tmp/$tgz" | sha256sum -c -
|
||||
tar -xzf "$tmp/$tgz" -C "$tmp" resticprofile
|
||||
install -o root -g root -m 0755 "$tmp/resticprofile" /usr/local/bin/resticprofile
|
||||
when: "! /usr/local/bin/resticprofile version 2>/dev/null | grep -q 'version {{ resticprofile_version }} '"
|
||||
|
||||
- name: Directories (stage is root-only; its augaman subdir belongs to the container user 10001)
|
||||
sudo: true
|
||||
shell: |
|
||||
set -eu
|
||||
install -d -o root -g root -m 0755 /etc/restic
|
||||
install -d -o root -g root -m 0700 /var/lib/restic/stage
|
||||
install -d -o 10001 -g 10001 -m 0700 /var/lib/restic/stage/augaman
|
||||
when: "! sudo -n sh -c 'test \"$(stat -c %U:%a /var/lib/restic/stage)\" = root:700 && test \"$(stat -c %u:%a /var/lib/restic/stage/augaman)\" = 10001:700 && test -d /etc/restic'"
|
||||
|
||||
- name: Secrets are seeded (password + repository, root 0400)
|
||||
sudo: true
|
||||
shell: |
|
||||
set -eu
|
||||
for f in /etc/restic/password /etc/restic/repository; do
|
||||
test -s "$f" || { echo "missing $f: seed it from the vault (configs/restic/esh-ml1/README.md)"; exit 1; }
|
||||
test "$(stat -c %U:%a "$f")" = root:400 || { echo "$f must be root 0400"; exit 1; }
|
||||
done
|
||||
changed_when: "false"
|
||||
|
||||
- name: restic understands --repository-file
|
||||
shell: restic --help | grep -q -- --repository-file
|
||||
changed_when: "false"
|
||||
|
||||
- name: Upload profiles.yaml
|
||||
sudo: true
|
||||
upload:
|
||||
src: configs/restic/esh-ml1/profiles.yaml
|
||||
dest: /etc/restic/profiles.yaml
|
||||
mode: "0644"
|
||||
|
||||
- name: Upload pre-backup.sh
|
||||
sudo: true
|
||||
upload:
|
||||
src: configs/restic/esh-ml1/pre-backup.sh
|
||||
dest: /etc/restic/pre-backup.sh
|
||||
mode: "0755"
|
||||
|
||||
- name: Initialise the repository (once)
|
||||
sudo: true
|
||||
shell: resticprofile --no-ansi --config /etc/restic/profiles.yaml --name default init
|
||||
when: "! sudo -n restic --repository-file /etc/restic/repository --password-file /etc/restic/password cat config >/dev/null 2>&1"
|
||||
|
||||
- name: Install the systemd timers
|
||||
sudo: true
|
||||
shell: resticprofile --no-ansi --config /etc/restic/profiles.yaml --name default schedule
|
||||
|
||||
verify:
|
||||
- name: Backup and check timers are active
|
||||
shell: systemctl is-active --quiet resticprofile-backup@profile-default.timer && systemctl is-active --quiet resticprofile-check@profile-default.timer
|
||||
changed_when: "false"
|
||||
|
||||
# The units must EXIST before "no match" means anything: a grep over a path that
|
||||
# is not there also "finds nothing". `rest:http` is the exact form the leak takes
|
||||
# on the env-file hosts (seen in esh-docker-vm's unit, 2026-09-27).
|
||||
- name: No repository URL (and so no rest-server password) in the generated units
|
||||
shell: |
|
||||
set -eu
|
||||
d=/etc/systemd/system
|
||||
for u in resticprofile-backup@profile-default.service resticprofile-check@profile-default.service; do
|
||||
test -f "$d/$u" || { echo "missing unit $u"; exit 1; }
|
||||
if grep -q 'rest:http' "$d/$u"; then echo "$u embeds the repository URL"; exit 1; fi
|
||||
done
|
||||
changed_when: "false"
|
||||
|
||||
- name: pre-backup.sh parses
|
||||
shell: bash -n /etc/restic/pre-backup.sh
|
||||
changed_when: "false"
|
||||
@@ -160,7 +160,7 @@ discover_repos() { # $1=host $2=parent dir — prints "name<TAB>newest_epoch"
|
||||
done" 2>/dev/null
|
||||
}
|
||||
|
||||
EXPECTED_ANA="${EXPECTED_RESTIC_ANA:-ana-docker ana-ml2 esh-docker-vm esh-vm-db vm-esh-nas}"
|
||||
EXPECTED_ANA="${EXPECTED_RESTIC_ANA:-ana-docker ana-ml2 esh-docker-vm esh-ml1 esh-vm-db vm-esh-nas}"
|
||||
EXPECTED_NH3="${EXPECTED_RESTIC_NH3:-irv-ml1 nh3-docker nh3-dev}"
|
||||
|
||||
found_ana=""; found_nh3=""
|
||||
|
||||
@@ -19,7 +19,7 @@ README. The reward seat (`/scalar-judge`) is still esh-ml1 only.
|
||||
| **OS** | Debian 12, unprivileged, `nesting=1,keyctl=1` |
|
||||
| **Size** | 6 cores, 16 GB RAM + 2 GB swap, 80 GB rootfs on `local-lvm` |
|
||||
| **Boot** | `onboot: 1`, `startup: order=30` — after esh-scale (1), esh-vm-db (10) and esh-vm-docker (20), so a GPU fault never delays ESH's DNS or mesh route |
|
||||
| **Backups** | **None, on purpose.** esh-pve's vzdump job lists vmids explicitly and 110 is not one. Everything is rebuilt from the playbooks and the stack; models re-download. |
|
||||
| **Backups** | **No vzdump, on purpose.** esh-pve's vzdump job lists vmids explicitly and 110 is not one. Everything is rebuilt from the playbooks and the stack; models re-download. **Exception: augaman's gallery**, backed up by restic daily at 0100 PT to rest-server-ana ([`configs/restic/esh-ml1/`](../../configs/restic/esh-ml1/README.md), `playbooks/esh-ml1-restic.yaml`). |
|
||||
|
||||
## What it serves
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
# compose.yaml. Do not add an env_file.
|
||||
|
||||
# Built locally on esh-ml1 from the release tag (see README "Building").
|
||||
IMAGE=augaman:0.1.1
|
||||
IMAGE=augaman:0.1.2
|
||||
|
||||
PORT=8040
|
||||
HOST_IP=10.0.50.80
|
||||
|
||||
@@ -26,8 +26,13 @@ CLI, which writes `gallery.db` (mode 0600) into `BACKUP_DIR` =
|
||||
restore has been verified (the restored copy reports the same identities), only
|
||||
public-domain test fixtures may be enrolled. No household faces.**
|
||||
|
||||
Backup status: **NOT WIRED** as of the first deploy (2026-09-26). esh-ml1 has no
|
||||
restic yet.
|
||||
Backup status (2026-09-27): **WIRED AND RESTORE-VERIFIED, so the gate is met.**
|
||||
restic runs daily at 0100 PT to rest-server-ana, with a fail-closed pre-backup
|
||||
hook ([`configs/restic/esh-ml1/`](../../configs/restic/esh-ml1/README.md)). The
|
||||
restore was verified against augaman-dev's public-domain canary identity (1
|
||||
identity, 3 samples): the copy restored from snapshot `fd3061a1` matched the
|
||||
live gallery (identities, samples and embeddings, by digest), and
|
||||
`integrity_check` returned ok.
|
||||
|
||||
```bash
|
||||
docker exec augaman python -m augaman.gallery.backup --db /data/gallery.db --dest /backup/gallery.db
|
||||
@@ -47,11 +52,17 @@ cd /opt/docker/src/augaman-vX.Y.Z && docker build -t augaman:X.Y.Z .
|
||||
```
|
||||
|
||||
The build fetches the two pinned models and SHA-256-checks them; a mismatch fails
|
||||
the build. v0.1.0 built in under 2 minutes cold; first deployed version is v0.1.1 (tightly-cropped-face detector fix).
|
||||
the build. v0.1.0 built in under 2 minutes cold; v0.1.1 was the first version
|
||||
deployed (2026-09-26), then v0.1.2 (2026-09-27).
|
||||
|
||||
⚠ **Disk:** the build took the rootfs from 66% to 80% (image ~5 GB plus build
|
||||
cache). Beszel alerts at 85%. After a rebuild, remove the old image and run
|
||||
`docker builder prune` once the new version is verified.
|
||||
⚠ **Disk:** a build that has to install the third-party packages takes ~8–11 GB
|
||||
transiently (image ~5 GB plus the ~3 GB uv download cache). Beszel alerts at 85%.
|
||||
Before v0.1.2, every version bump re-installed them, and the v0.1.1 build hit 90%.
|
||||
From v0.1.2 the packages install from a layer keyed on `uv export
|
||||
--no-emit-project`, so a bump that changes no dependency reuses that layer. Keep
|
||||
the layer cache (`docker builder prune --filter type=exec.cachemount` drops only
|
||||
the download cache); a full `docker builder prune` forces the next build to
|
||||
re-install everything.
|
||||
|
||||
## Startup is fail-closed on CUDA
|
||||
|
||||
|
||||
Reference in New Issue
Block a user