Initial commit: PFI fleet inventory, stacks, tooling, and backup pipeline
Captures the full workspace state built up to this point:
- CLAUDE.md + README.md describing conventions and the four-host fleet
(ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
- Per-host notes under servers/<host>/ with ssh-target fallback files
and latest system-details snapshots (two in-compose credential leaks
scrubbed; the upstream compose files still need to move those to .env).
- scripts/: server_inspect.sh (read-only remote diagnostic),
refresh-server-info.sh (dir-driven discovery + snapshot capture with
validation warnings), add-host.sh, sync-stacks.sh (pull
compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
- stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
infinity reference. All use the .env-driven + traefik-net + homepage
label pattern.
- configs/restic/ana-docker/: first resticprofile config + pre-backup
hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
for the other three hosts to come.
- docs/pfi/: general infrastructure reference carried over.
- .gitignore excludes .env, stacks-mirror/, and assorted secret/state
filenames to prevent re-leaks on later commits.
This commit is contained in:
@@ -0,0 +1,118 @@
|
||||
# configs/restic
|
||||
|
||||
Per-host restic backup configs, deployed into `/etc/restic/` on each server and driven by `resticprofile` + `systemd` timers.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
configs/restic/
|
||||
├── README.md # this file
|
||||
└── <host>/
|
||||
├── profiles.yaml # committed, zero secrets
|
||||
├── pre-backup.sh # committed, zero secrets
|
||||
└── README.md # per-host notes (paths, containers, quirks)
|
||||
```
|
||||
|
||||
On each server, deployed to `/etc/restic/`:
|
||||
|
||||
```
|
||||
/etc/restic/
|
||||
├── profiles.yaml # scp'd from configs/restic/<host>/profiles.yaml
|
||||
├── pre-backup.sh # scp'd, 0755, root:root
|
||||
├── password # 0400 root:root — client-side encryption passphrase
|
||||
└── restic.env # 0600 root:root — RESTIC_REPOSITORY=rest:http://user:pw@host:port/path/
|
||||
/var/lib/restic/
|
||||
├── stage/ # temp staging for DB dumps; owned by root, 0700
|
||||
└── last-success # unix timestamp of the last successful run
|
||||
```
|
||||
|
||||
## Why this shape
|
||||
|
||||
- **No secrets in committed config.** `profiles.yaml` references `RESTIC_PASSWORD_FILE=/etc/restic/password` and loads `RESTIC_REPOSITORY` from `restic.env`. Both files live only on the host, 0400/0600 root-owned.
|
||||
- **Creds go in the URL, not netrc.** restic's rest backend doesn't consult `~/.netrc` — HTTP basic-auth has to be embedded in the repository URL. Keeping that URL in an env-file (not the committed YAML) means the secret stays on the host.
|
||||
- **Pre-hook runs DB dumps into a staging dir**, then `restic backup` includes that dir alongside the regular paths. One snapshot = one point-in-time.
|
||||
- **`restic forget` is scheduled; `restic prune` is not.** Rest-server's `--append-only` blocks prune from the client side by design. Prune is a manual ceremony (flip the flag, run prune, flip back).
|
||||
|
||||
## Install restic + resticprofile on each host
|
||||
|
||||
```bash
|
||||
# Current-enough restic. Debian 12 ships 0.14 (too old for some flags);
|
||||
# Debian 13 ships 0.18. If you're on 12, grab the .deb from the upstream
|
||||
# release page instead.
|
||||
sudo apt install -y restic # or install 0.18+ from github.com/restic/restic/releases
|
||||
|
||||
# resticprofile is not in Debian. Download the .deb from its release page.
|
||||
v=$(curl -sI https://github.com/creativeprojects/resticprofile/releases/latest \
|
||||
| awk -F'/' '/^location:/{sub(/\r/,""); print $NF}')
|
||||
v=${v#v}
|
||||
url="https://github.com/creativeprojects/resticprofile/releases/download/v${v}/resticprofile_${v}_linux_amd64.deb"
|
||||
curl -sL -o /tmp/resticprofile.deb "$url"
|
||||
sudo dpkg -i /tmp/resticprofile.deb
|
||||
rm /tmp/resticprofile.deb
|
||||
resticprofile version
|
||||
```
|
||||
|
||||
## Per-host deploy flow (done once per host)
|
||||
|
||||
```bash
|
||||
HOST=ana-docker # or ana-ml2, nh3-docker, esh-docker-vm
|
||||
|
||||
# 1. Create the target dir (one-time)
|
||||
ssh -t "$HOST" 'sudo install -d -o root -g root -m 0755 /etc/restic'
|
||||
ssh -t "$HOST" 'sudo install -d -o root -g root -m 0700 /var/lib/restic/stage'
|
||||
|
||||
# 2. Seed the two secret files on the host (never in this repo):
|
||||
# - the client-side encryption passphrase (the one used at `restic init`)
|
||||
# - an env-file with the full RESTIC_REPOSITORY URL including HTTP creds
|
||||
ssh -t "$HOST" 'sudo install -o root -g root -m 0400 /dev/null /etc/restic/password'
|
||||
ssh -t "$HOST" 'sudo install -o root -g root -m 0600 /dev/null /etc/restic/restic.env'
|
||||
|
||||
# Seed content (replace <…> with real values from your vault):
|
||||
ssh -t "$HOST" "echo '<encryption-passphrase>' | sudo tee /etc/restic/password >/dev/null"
|
||||
ssh -t "$HOST" "echo 'RESTIC_REPOSITORY=rest:http://<user>:<http-pw>@<rest-server>:8000/<user>/' | sudo tee /etc/restic/restic.env >/dev/null"
|
||||
|
||||
# 3. Push the committed config + pre-hook
|
||||
scp "configs/restic/$HOST/profiles.yaml" "$HOST:/tmp/profiles.yaml"
|
||||
scp "configs/restic/$HOST/pre-backup.sh" "$HOST:/tmp/pre-backup.sh"
|
||||
ssh -t "$HOST" "
|
||||
sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml
|
||||
sudo install -o root -g root -m 0755 /tmp/pre-backup.sh /etc/restic/pre-backup.sh
|
||||
rm -f /tmp/profiles.yaml /tmp/pre-backup.sh
|
||||
"
|
||||
|
||||
# 4. Test the profile before scheduling
|
||||
ssh -t "$HOST" 'sudo resticprofile --config /etc/restic/profiles.yaml --name default backup --dry-run'
|
||||
|
||||
# 5. When dry-run looks clean, wire up systemd timers
|
||||
ssh -t "$HOST" 'sudo resticprofile --config /etc/restic/profiles.yaml schedule'
|
||||
# → installs restic-backup@<profile>.{service,timer} units for backup/forget/check
|
||||
|
||||
# 6. Verify
|
||||
ssh -t "$HOST" 'sudo systemctl list-timers | grep restic'
|
||||
```
|
||||
|
||||
## Prune ceremony (quarterly or as needed)
|
||||
|
||||
Per-repo, when enough forgotten-but-still-on-disk snapshots accumulate:
|
||||
|
||||
```bash
|
||||
# On the rest-server host (ana-docker or the Synology):
|
||||
# 1. Stop the append-only rest-server, start one without the flag
|
||||
# (simplest: edit the stack's .env, remove --append-only from OPTIONS,
|
||||
# docker compose up -d)
|
||||
|
||||
# 2. From the client host, run prune
|
||||
ssh -t <host> 'sudo resticprofile --config /etc/restic/profiles.yaml --name default prune'
|
||||
|
||||
# 3. Put --append-only back on the rest-server and docker compose up -d.
|
||||
```
|
||||
|
||||
Alternatively stand up a second rest-server stack on port 8001 without `--append-only` and point prune runs at that endpoint; keep the append-only one for daily writes.
|
||||
|
||||
## Monitoring
|
||||
|
||||
Each successful run writes `/var/lib/restic/last-success` with the current unix timestamp (this is the `post-backup` hook in `profiles.yaml`). Beszel can be configured to alert when that file's mtime exceeds ~36 hours.
|
||||
|
||||
```bash
|
||||
ssh <host> 'stat -c "%y %n" /var/lib/restic/last-success 2>/dev/null || echo no successful run yet'
|
||||
```
|
||||
Reference in New Issue
Block a user