vh
6e203dcb99
resticprofile schedule copies env-file values into the generated units, which
are 0644, so RESTIC_REPOSITORY (the rest-server basic-auth password included)
was readable by every local user on every restic host.
New playbooks/restic-repository-file.yaml:
- derives /etc/restic/repository (root 0400) from restic.env;
- uploads the profile switched to repository-file, but only when the live
profile's sha matches the repo copy it was edited from (drift guard);
- checks the repository is reachable through the new profile (cat config);
- regenerates the units and verifies they exist and contain no rest:http.
Applied to ana-docker, fv-ml1 (configs/restic/ana-ml2), esh-docker-vm,
esh-vm-db, irv-ml1, nh3-dev and nh3-docker. An independent check across all
eight restic hosts (these seven plus esh-ml1) found 0 leaking units. nh3-docker's
scheduled unit ran a real backup afterwards (snapshot a29b889d). Each host's
URL and passphrase are vaulted as <host>/etc/restic/{repository,password}.
restic.env is kept (root 0600) because the per-host READMEs and the freshness
probe source it. A rotation must update the vault, restic.env and repository.
vm-esh-nas has no infra-ops account. Its in-place migration script is staged
for Prime to run with sudo, and its repo profile is pre-edited to match.
Also mirrors augaman-dev's 401df2d (compose header only). The config hash on
esh-ml1 is unchanged.
127 lines
6.9 KiB
Markdown
127 lines
6.9 KiB
Markdown
# configs/restic
|
|
|
|
Per-host restic backup configs, deployed into `/etc/restic/` on each server and driven by `resticprofile` + `systemd` timers.
|
|
|
|
Writes flow to two rest-servers: the Anaheim one on ana-docker stores data to an NFS mount backed by the Debian file server at `10.250.50.50`; the NH3 one on the Synology at `10.100.50.50` stores to local Btrfs. Cross-site rsync keeps each side holding a mirror of the other.
|
|
|
|
## Layout
|
|
|
|
```
|
|
configs/restic/
|
|
├── README.md # this file
|
|
└── <host>/
|
|
├── profiles.yaml # committed, zero secrets
|
|
├── pre-backup.sh # committed, zero secrets
|
|
└── README.md # per-host notes (paths, containers, quirks)
|
|
```
|
|
|
|
On each server, deployed to `/etc/restic/`:
|
|
|
|
```
|
|
/etc/restic/
|
|
├── profiles.yaml # scp'd from configs/restic/<host>/profiles.yaml
|
|
├── pre-backup.sh # scp'd, 0755, root:root
|
|
├── password # 0400 root:root — client-side encryption passphrase
|
|
├── repository # 0400 root:root — rest:http://user:pw@host:port/path/ (read via repository-file)
|
|
└── restic.env # 0600 root:root — RESTIC_REPOSITORY=… (same URL; kept for manual `. restic.env` snippets)
|
|
/var/lib/restic/
|
|
├── stage/ # temp staging for DB dumps; owned by root, 0700
|
|
└── last-success # unix timestamp of the last successful run
|
|
```
|
|
|
|
## Why this shape
|
|
|
|
- **No secrets in committed config.** `profiles.yaml` references `RESTIC_PASSWORD_FILE=/etc/restic/password` and reads the repository URL through `repository-file: /etc/restic/repository`. Both files live only on the host, root 0400. Both are also vaulted as `<host>/etc/restic/{password,repository}` (2026-09-27).
|
|
- **Creds go in the URL, not netrc.** restic's rest backend doesn't consult `~/.netrc` — HTTP basic-auth has to be embedded in the repository URL.
|
|
- ⚠ **Never `env-file` for the URL.** Until 2026-09-27 the profiles loaded it with `env-file: /etc/restic/restic.env`. But `resticprofile schedule` copies env-file values into the generated systemd units, which are 0644, so every local user could read the rest-server password. All hosts except vm-esh-nas were moved with `playbooks/restic-repository-file.yaml`. vm-esh-nas has no infra-ops account; its script is `configs/restic/vm-esh-nas/migrate-repository-file.sh`, and Prime runs it with sudo. `restic.env` is kept (root 0600, not a leak) because the per-host READMEs and the freshness probe source it. **On a password rotation, update the vault, `restic.env` AND `repository`.**
|
|
- **Pre-hook runs DB dumps into a staging dir**, then `restic backup` includes that dir alongside the regular paths. One snapshot = one point-in-time.
|
|
- **`restic forget` is scheduled; `restic prune` is not.** Rest-server's `--append-only` blocks prune from the client side by design. Prune is a manual ceremony (flip the flag, run prune, flip back).
|
|
|
|
## Install restic + resticprofile on each host
|
|
|
|
```bash
|
|
# Current-enough restic. Debian 12 ships 0.14 (too old for some flags);
|
|
# Debian 13 ships 0.18. If you're on 12, grab the .deb from the upstream
|
|
# release page instead.
|
|
sudo apt install -y restic # or install 0.18+ from github.com/restic/restic/releases
|
|
|
|
# resticprofile is not in Debian. Download the .deb from its release page.
|
|
v=$(curl -sI https://github.com/creativeprojects/resticprofile/releases/latest \
|
|
| awk -F'/' '/^location:/{sub(/\r/,""); print $NF}')
|
|
v=${v#v}
|
|
url="https://github.com/creativeprojects/resticprofile/releases/download/v${v}/resticprofile_${v}_linux_amd64.deb"
|
|
curl -sL -o /tmp/resticprofile.deb "$url"
|
|
sudo dpkg -i /tmp/resticprofile.deb
|
|
rm /tmp/resticprofile.deb
|
|
resticprofile version
|
|
```
|
|
|
|
## Per-host deploy flow (done once per host)
|
|
|
|
```bash
|
|
HOST=ana-docker # or ana-ml2, nh3-docker, esh-docker-vm
|
|
|
|
# 1. Create the target dir (one-time)
|
|
ssh -t "$HOST" 'sudo install -d -o root -g root -m 0755 /etc/restic'
|
|
ssh -t "$HOST" 'sudo install -d -o root -g root -m 0700 /var/lib/restic/stage'
|
|
|
|
# 2. Seed the two secret files on the host (never in this repo):
|
|
# - the client-side encryption passphrase (the one used at `restic init`)
|
|
# - an env-file with the full RESTIC_REPOSITORY URL including HTTP creds
|
|
# - the same URL alone in /etc/restic/repository (what the profile reads)
|
|
# Vault them first: secret put <host>/etc/restic/password, …/repository
|
|
ssh -t "$HOST" 'sudo install -o root -g root -m 0400 /dev/null /etc/restic/password'
|
|
ssh -t "$HOST" 'sudo install -o root -g root -m 0600 /dev/null /etc/restic/restic.env'
|
|
ssh -t "$HOST" 'sudo install -o root -g root -m 0400 /dev/null /etc/restic/repository'
|
|
|
|
# Seed content (replace <…> with real values from your vault):
|
|
ssh -t "$HOST" "echo '<encryption-passphrase>' | sudo tee /etc/restic/password >/dev/null"
|
|
ssh -t "$HOST" "echo 'RESTIC_REPOSITORY=rest:http://<user>:<http-pw>@<rest-server>:8000/<user>/' | sudo tee /etc/restic/restic.env >/dev/null"
|
|
ssh -t "$HOST" "echo 'rest:http://<user>:<http-pw>@<rest-server>:8000/<user>/' | sudo tee /etc/restic/repository >/dev/null"
|
|
|
|
# 3. Push the committed config + pre-hook
|
|
scp "configs/restic/$HOST/profiles.yaml" "$HOST:/tmp/profiles.yaml"
|
|
scp "configs/restic/$HOST/pre-backup.sh" "$HOST:/tmp/pre-backup.sh"
|
|
ssh -t "$HOST" "
|
|
sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml
|
|
sudo install -o root -g root -m 0755 /tmp/pre-backup.sh /etc/restic/pre-backup.sh
|
|
rm -f /tmp/profiles.yaml /tmp/pre-backup.sh
|
|
"
|
|
|
|
# 4. Test the profile before scheduling
|
|
ssh -t "$HOST" 'sudo resticprofile --config /etc/restic/profiles.yaml --name default backup --dry-run'
|
|
|
|
# 5. When dry-run looks clean, wire up systemd timers
|
|
ssh -t "$HOST" 'sudo resticprofile --config /etc/restic/profiles.yaml schedule'
|
|
# → installs restic-backup@<profile>.{service,timer} units for backup/forget/check
|
|
|
|
# 6. Verify
|
|
ssh -t "$HOST" 'sudo systemctl list-timers | grep restic'
|
|
```
|
|
|
|
## Prune ceremony (quarterly or as needed)
|
|
|
|
Per-repo, when enough forgotten-but-still-on-disk snapshots accumulate:
|
|
|
|
```bash
|
|
# On the rest-server host (ana-docker or the Synology):
|
|
# 1. Stop the append-only rest-server, start one without the flag
|
|
# (simplest: edit the stack's .env, remove --append-only from OPTIONS,
|
|
# docker compose up -d)
|
|
|
|
# 2. From the client host, run prune
|
|
ssh -t <host> 'sudo resticprofile --config /etc/restic/profiles.yaml --name default prune'
|
|
|
|
# 3. Put --append-only back on the rest-server and docker compose up -d.
|
|
```
|
|
|
|
Alternatively stand up a second rest-server stack on port 8001 without `--append-only` and point prune runs at that endpoint; keep the append-only one for daily writes.
|
|
|
|
## Monitoring
|
|
|
|
Each successful run writes `/var/lib/restic/last-success` with the current unix timestamp (this is the `post-backup` hook in `profiles.yaml`). Beszel can be configured to alert when that file's mtime exceeds ~36 hours.
|
|
|
|
```bash
|
|
ssh <host> 'stat -c "%y %n" /var/lib/restic/last-success 2>/dev/null || echo no successful run yet'
|
|
```
|