esh-pfi-infrastructure/persistent-memory.d/2026-09-27-restic-repository-file-fleetwide.md

[2026-09-27] restic: repository URL out of the systemd units on all 8 hosts, and vaulted (Prime)

resticprofile schedule copies env-file values into the generated units under /etc/systemd/system (0644). So RESTIC_REPOSITORY, including the rest-server basic-auth password, was readable by any local user on every env-file host (found on esh-docker-vm; systemctl cat works without sudo). Blast radius: the password allows reading the encrypted blobs and appending to one repo, and nothing more (rest-server is --append-only, and the passphrase is a separate file).

Fix (6e203dc, 30f2c97): playbooks/restic-repository-file.yaml does four things:

• derives /etc/restic/repository (root 0400) from restic.env;
• uploads the profile switched to repository-file, guarded by the live profile's pre-change sha;
• runs cat config through the new profile;
• re-schedules, then verifies the units exist and contain no rest:http.

Applied to ana-docker, fv-ml1 (configs/restic/ana-ml2), esh-docker-vm, esh-vm-db, irv-ml1, nh3-dev, nh3-docker, and vm-esh-nas once Prime had bootstrapped infra-ops there. esh-ml1 was built that way. An independent check across all 8 found 0 leaking units. nh3-docker's scheduled unit ran a real backup (a29b889d). Every host's URL and passphrase are vaulted as <host>/etc/restic/{repository,password}.

• restic.env is kept (root 0600): the per-host READMEs and the freshness probe source it. A rotation must update the vault, restic.env and repository.
• The rotation itself stays Prime's (backups runbook, Known gaps). The move stops the ongoing exposure but does not un-expose what was readable.
• Before the edit, esh-docker-vm's and irv-ml1's live profiles had drifted ahead of the repo, and were pulled in first (c698751).

In-flight state at retirement (moved here at the 2026-10-06 snapshot)

• Seven hosts were moved from env-file to repository-file (playbooks/restic-repository-file.yaml), so the systemd units no longer carry the rest-server password. Secrets are vaulted as <host>/etc/restic/{repository,password}. restic.env is KEPT for manual snippets, so a rotation must update the vault, restic.env and repository.
• vm-esh-nas done too (Prime bootstrapped infra-ops there at uid 850, NOPASSWD). All 8 hosts are clean and vaulted.
• The passwords were readable until today, so the rotation (Prime's) remains the real fix.
• 2026-09-27 0859 freshness check: all restic and PBS repos ✅. Those 0100 runs PRE-date the move (done ~0140–0200).
• VERIFIED 2026-09-28 (infra-hermes, read-only): the first nightly on repository-file (0100 PT) succeeded on all 8 hosts (unit Result=success AND a today-dated snapshot in each repo), and the 0800 freshness check was all-green, PBS included. nh3-dev also passed a content restore (af5580c3).