Prime bootstrapped infra-ops on vm-esh-nas with playbooks/bootstrap-infra-ops-user.yaml.
It got the fleet-pinned uid/gid 850, NOPASSWD sudo with log_output, the docker
group and a 0700 home. That let playbooks/restic-repository-file.yaml migrate
the last restic host: the live profile matched the repo's pre-change sha,
its units no longer carry the URL, its secrets are vaulted, and the live and repo
profiles now match (a5ea75ea). All eight restic hosts are clean.
The staged helper script is gone, both from Prime's home on the host and from the
repo. The docs that described vm-esh-nas as lkraven-only are updated.
resticprofile schedule copies env-file values into the generated units, which
are 0644, so RESTIC_REPOSITORY (the rest-server basic-auth password included)
was readable by every local user on every restic host.
New playbooks/restic-repository-file.yaml:
- derives /etc/restic/repository (root 0400) from restic.env;
- uploads the profile switched to repository-file, but only when the live
profile's sha matches the repo copy it was edited from (drift guard);
- checks the repository is reachable through the new profile (cat config);
- regenerates the units and verifies they exist and contain no rest:http.
Applied to ana-docker, fv-ml1 (configs/restic/ana-ml2), esh-docker-vm,
esh-vm-db, irv-ml1, nh3-dev and nh3-docker. An independent check across all
eight restic hosts (these seven plus esh-ml1) found 0 leaking units. nh3-docker's
scheduled unit ran a real backup afterwards (snapshot a29b889d). Each host's
URL and passphrase are vaulted as <host>/etc/restic/{repository,password}.
restic.env is kept (root 0600) because the per-host READMEs and the freshness
probe source it. A rotation must update the vault, restic.env and repository.
vm-esh-nas has no infra-ops account. Its in-place migration script is staged
for Prime to run with sudo, and its repo profile is pre-edited to match.
Also mirrors augaman-dev's 401df2d (compose header only). The config hash on
esh-ml1 is unchanged.
Forget against an --append-only rest-server fails every night (delete
ops blocked). The resulting daily failure cluttered service status and
logs without ever actually retiring old snapshots. Schedule is now
removed from the forget block in all six profiles; the keep-daily /
keep-weekly / keep-monthly / keep-yearly policy remains so manual
invocations (during prune ceremonies, when --append-only is
temporarily off) honor the intended retention.
Files:
configs/restic/ana-docker/profiles.yaml
configs/restic/ana-ml2/profiles.yaml
configs/restic/nh3-docker/profiles.yaml
configs/restic/esh-docker-vm/profiles.yaml
configs/restic/vm-esh-nas/profiles.yaml
configs/restic/nh3-dev/profiles.yaml
Each file has an inline comment marking why the schedule was dropped
so a future reader doesn't re-add it thinking it was an oversight.
STATUS.md: removed the "install Backrest nightly-restart timer" line
item. User confirmed the UI timeout hits even at startup, so periodic
restart wouldn't actually help. Root cause remains deferred.
The Anaheim NAS exports /mnt/backup with root_squash, so sudo-as-root
on ana-docker becomes nobody on the NFS side and `sudo tee` gets EACCES.
The repo tree is owned by lkraven:lkraven precisely because of this —
all writes go through the lkraven UID.
Drop sudo from the example; plain `>>` append as lkraven works.
Second ESH Docker host — light (filezilla + dockge + agents, /opt/docker
is 56 KB). Cross-site writes to the Anaheim rest-server since ESH has
no local rest-server.
Critical detail: vm-esh-nas NFS-mounts /mnt/{share,music,books,media}
from 10.0.50.50 (~400 TB). Profile's exclude patterns explicitly
reject those paths as a safety net on top of the source list not
including them — a careless future edit to sources can't nuke the
backup repo by pulling in 400 TB.
Sources include /home/ (lkraven/nas/user dirs) in addition to the
usual /opt/docker /etc /root /var/lib/docker/volumes — this host has
multiple user accounts worth preserving dotfiles for.
No pre-backup hook: no relational DBs.
README walks through the full setup: install restic (not present on
this host), add htpasswd entry on rest-server-ana, install creds,
init, install resticprofile, schedule timers. Also flags the 3.8 GB
RAM constraint.
vh