esh-pfi-infrastructure/configs/restic/README.md

configs/restic

Per-host restic backup configs, deployed into /etc/restic/ on each server and driven by resticprofile + systemd timers.

Writes flow to two rest-servers: the Anaheim one on ana-docker stores data to an NFS mount backed by the Debian file server at 10.250.50.50; the NH3 one on the Synology at 10.100.50.50 stores to local Btrfs. Cross-site rsync keeps each side holding a mirror of the other.

Layout

configs/restic/
├── README.md                  # this file
└── <host>/
    ├── profiles.yaml          # committed, zero secrets
    ├── pre-backup.sh          # committed, zero secrets
    └── README.md              # per-host notes (paths, containers, quirks)

On each server, deployed to /etc/restic/:

/etc/restic/
├── profiles.yaml              # scp'd from configs/restic/<host>/profiles.yaml
├── pre-backup.sh              # scp'd, 0755, root:root
├── password                   # 0400 root:root — client-side encryption passphrase
├── repository                 # 0400 root:root — rest:http://user:pw@host:port/path/ (read via repository-file)
└── restic.env                 # 0600 root:root — RESTIC_REPOSITORY=… (same URL; kept for manual `. restic.env` snippets)
/var/lib/restic/
├── stage/                     # temp staging for DB dumps; owned by root, 0700
└── last-success               # unix timestamp of the last successful run

Why this shape

• No secrets in committed config. profiles.yaml references RESTIC_PASSWORD_FILE=/etc/restic/password and reads the repository URL through repository-file: /etc/restic/repository. Both files live only on the host, root 0400. Both are also vaulted as <host>/etc/restic/{password,repository} (2026-09-27).
• Creds go in the URL, not netrc. restic's rest backend doesn't consult ~/.netrc — HTTP basic-auth has to be embedded in the repository URL.
• ⚠ Never env-file for the URL. Until 2026-09-27 the profiles loaded it with env-file: /etc/restic/restic.env. But resticprofile schedule copies env-file values into the generated systemd units, which are 0644, so every local user could read the rest-server password. Every host was moved with playbooks/restic-repository-file.yaml (vm-esh-nas once Prime had bootstrapped infra-ops there, the same day). restic.env is kept (root 0600, not a leak) because the per-host READMEs and the freshness probe source it. On a password rotation, update the vault, restic.env AND repository.
• Pre-hook runs DB dumps into a staging dir, then restic backup includes that dir alongside the regular paths. One snapshot = one point-in-time.
• restic forget is scheduled; restic prune is not. Rest-server's --append-only blocks prune from the client side by design. Prune is a manual ceremony (flip the flag, run prune, flip back).

Install restic + resticprofile on each host

# Current-enough restic. Debian 12 ships 0.14 (too old for some flags);
# Debian 13 ships 0.18. If you're on 12, grab the .deb from the upstream
# release page instead.
sudo apt install -y restic    # or install 0.18+ from github.com/restic/restic/releases

# resticprofile is not in Debian. Download the .deb from its release page.
v=$(curl -sI https://github.com/creativeprojects/resticprofile/releases/latest \
     | awk -F'/' '/^location:/{sub(/\r/,""); print $NF}')
v=${v#v}
url="https://github.com/creativeprojects/resticprofile/releases/download/v${v}/resticprofile_${v}_linux_amd64.deb"
curl -sL -o /tmp/resticprofile.deb "$url"
sudo dpkg -i /tmp/resticprofile.deb
rm /tmp/resticprofile.deb
resticprofile version

Per-host deploy flow (done once per host)

HOST=ana-docker     # or ana-ml2, nh3-docker, esh-docker-vm

# 1. Create the target dir (one-time)
ssh -t "$HOST" 'sudo install -d -o root -g root -m 0755 /etc/restic'
ssh -t "$HOST" 'sudo install -d -o root -g root -m 0700 /var/lib/restic/stage'

# 2. Seed the two secret files on the host (never in this repo):
#    - the client-side encryption passphrase (the one used at `restic init`)
#    - an env-file with the full RESTIC_REPOSITORY URL including HTTP creds
#    - the same URL alone in /etc/restic/repository (what the profile reads)
#    Vault them first: secret put <host>/etc/restic/password, …/repository
ssh -t "$HOST" 'sudo install -o root -g root -m 0400 /dev/null /etc/restic/password'
ssh -t "$HOST" 'sudo install -o root -g root -m 0600 /dev/null /etc/restic/restic.env'
ssh -t "$HOST" 'sudo install -o root -g root -m 0400 /dev/null /etc/restic/repository'

# Seed content (replace <…> with real values from your vault):
ssh -t "$HOST" "echo '<encryption-passphrase>' | sudo tee /etc/restic/password >/dev/null"
ssh -t "$HOST" "echo 'RESTIC_REPOSITORY=rest:http://<user>:<http-pw>@<rest-server>:8000/<user>/' | sudo tee /etc/restic/restic.env >/dev/null"
ssh -t "$HOST" "echo 'rest:http://<user>:<http-pw>@<rest-server>:8000/<user>/' | sudo tee /etc/restic/repository >/dev/null"

# 3. Push the committed config + pre-hook
scp "configs/restic/$HOST/profiles.yaml"  "$HOST:/tmp/profiles.yaml"
scp "configs/restic/$HOST/pre-backup.sh"  "$HOST:/tmp/pre-backup.sh"
ssh -t "$HOST" "
  sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml
  sudo install -o root -g root -m 0755 /tmp/pre-backup.sh /etc/restic/pre-backup.sh
  rm -f /tmp/profiles.yaml /tmp/pre-backup.sh
"

# 4. Test the profile before scheduling
ssh -t "$HOST" 'sudo resticprofile --config /etc/restic/profiles.yaml --name default backup --dry-run'

# 5. When dry-run looks clean, wire up systemd timers
ssh -t "$HOST" 'sudo resticprofile --config /etc/restic/profiles.yaml schedule'
#    → installs restic-backup@<profile>.{service,timer} units for backup/forget/check

# 6. Verify
ssh -t "$HOST" 'sudo systemctl list-timers | grep restic'

Prune ceremony (quarterly or as needed)

Per-repo, when enough forgotten-but-still-on-disk snapshots accumulate:

# On the rest-server host (ana-docker or the Synology):
# 1. Stop the append-only rest-server, start one without the flag
#    (simplest: edit the stack's .env, remove --append-only from OPTIONS,
#    docker compose up -d)

# 2. From the client host, run prune
ssh -t <host> 'sudo resticprofile --config /etc/restic/profiles.yaml --name default prune'

# 3. Put --append-only back on the rest-server and docker compose up -d.

Alternatively stand up a second rest-server stack on port 8001 without --append-only and point prune runs at that endpoint; keep the append-only one for daily writes.

Monitoring

Each successful run writes /var/lib/restic/last-success with the current unix timestamp (this is the post-backup hook in profiles.yaml). Beszel can be configured to alert when that file's mtime exceeds ~36 hours.

ssh <host> 'stat -c "%y %n" /var/lib/restic/last-success 2>/dev/null || echo no successful run yet'