Initial commit: PFI fleet inventory, stacks, tooling, and backup pipeline

Captures the full workspace state built up to this point: - CLAUDE.md + README.md describing conventions and the four-host fleet (ana-ml2, ana-docker, nh3-docker, esh-docker-vm). - Per-host notes under servers/<host>/ with ssh-target fallback files and latest system-details snapshots (two in-compose credential leaks scrubbed; the upstream compose files still need to move those to .env). - scripts/: server_inspect.sh (read-only remote diagnostic), refresh-server-info.sh (dir-driven discovery + snapshot capture with validation warnings), add-host.sh, sync-stacks.sh (pull compose/conf trees), deploy-stack.sh (push with per-file diff + prompt). - stacks/: canonical compose for backrest, beszel, dozzle, llama-swap, rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired infinity reference. All use the .env-driven + traefik-net + homepage label pattern. - configs/restic/ana-docker/: first resticprofile config + pre-backup hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates for the other three hosts to come. - docs/pfi/: general infrastructure reference carried over. - .gitignore excludes .env, stacks-mirror/, and assorted secret/state filenames to prevent re-leaks on later commits.
2026-04-20 14:29:48 -07:00
commit e376d0aec9
55 changed files with 9101 additions and 0 deletions
@@ -0,0 +1,118 @@
+# configs/restic
+
+Per-host restic backup configs, deployed into `/etc/restic/` on each server and driven by `resticprofile` + `systemd` timers.
+
+## Layout
+
+```
+configs/restic/
+├── README.md                  # this file
+└── <host>/
+    ├── profiles.yaml          # committed, zero secrets
+    ├── pre-backup.sh          # committed, zero secrets
+    └── README.md              # per-host notes (paths, containers, quirks)
+```
+
+On each server, deployed to `/etc/restic/`:
+
+```
+/etc/restic/
+├── profiles.yaml              # scp'd from configs/restic/<host>/profiles.yaml
+├── pre-backup.sh              # scp'd, 0755, root:root
+├── password                   # 0400 root:root — client-side encryption passphrase
+└── restic.env                 # 0600 root:root — RESTIC_REPOSITORY=rest:http://user:pw@host:port/path/
+/var/lib/restic/
+├── stage/                     # temp staging for DB dumps; owned by root, 0700
+└── last-success               # unix timestamp of the last successful run
+```
+
+## Why this shape
+
+- **No secrets in committed config.** `profiles.yaml` references `RESTIC_PASSWORD_FILE=/etc/restic/password` and loads `RESTIC_REPOSITORY` from `restic.env`. Both files live only on the host, 0400/0600 root-owned.
+- **Creds go in the URL, not netrc.** restic's rest backend doesn't consult `~/.netrc` — HTTP basic-auth has to be embedded in the repository URL. Keeping that URL in an env-file (not the committed YAML) means the secret stays on the host.
+- **Pre-hook runs DB dumps into a staging dir**, then `restic backup` includes that dir alongside the regular paths. One snapshot = one point-in-time.
+- **`restic forget` is scheduled; `restic prune` is not.** Rest-server's `--append-only` blocks prune from the client side by design. Prune is a manual ceremony (flip the flag, run prune, flip back).
+
+## Install restic + resticprofile on each host
+
+```bash
+# Current-enough restic. Debian 12 ships 0.14 (too old for some flags);
+# Debian 13 ships 0.18. If you're on 12, grab the .deb from the upstream
+# release page instead.
+sudo apt install -y restic    # or install 0.18+ from github.com/restic/restic/releases
+
+# resticprofile is not in Debian. Download the .deb from its release page.
+v=$(curl -sI https://github.com/creativeprojects/resticprofile/releases/latest \
+     | awk -F'/' '/^location:/{sub(/\r/,""); print $NF}')
+v=${v#v}
+url="https://github.com/creativeprojects/resticprofile/releases/download/v${v}/resticprofile_${v}_linux_amd64.deb"
+curl -sL -o /tmp/resticprofile.deb "$url"
+sudo dpkg -i /tmp/resticprofile.deb
+rm /tmp/resticprofile.deb
+resticprofile version
+```
+
+## Per-host deploy flow (done once per host)
+
+```bash
+HOST=ana-docker     # or ana-ml2, nh3-docker, esh-docker-vm
+
+# 1. Create the target dir (one-time)
+ssh -t "$HOST" 'sudo install -d -o root -g root -m 0755 /etc/restic'
+ssh -t "$HOST" 'sudo install -d -o root -g root -m 0700 /var/lib/restic/stage'
+
+# 2. Seed the two secret files on the host (never in this repo):
+#    - the client-side encryption passphrase (the one used at `restic init`)
+#    - an env-file with the full RESTIC_REPOSITORY URL including HTTP creds
+ssh -t "$HOST" 'sudo install -o root -g root -m 0400 /dev/null /etc/restic/password'
+ssh -t "$HOST" 'sudo install -o root -g root -m 0600 /dev/null /etc/restic/restic.env'
+
+# Seed content (replace <…> with real values from your vault):
+ssh -t "$HOST" "echo '<encryption-passphrase>' | sudo tee /etc/restic/password >/dev/null"
+ssh -t "$HOST" "echo 'RESTIC_REPOSITORY=rest:http://<user>:<http-pw>@<rest-server>:8000/<user>/' | sudo tee /etc/restic/restic.env >/dev/null"
+
+# 3. Push the committed config + pre-hook
+scp "configs/restic/$HOST/profiles.yaml"  "$HOST:/tmp/profiles.yaml"
+scp "configs/restic/$HOST/pre-backup.sh"  "$HOST:/tmp/pre-backup.sh"
+ssh -t "$HOST" "
+  sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml
+  sudo install -o root -g root -m 0755 /tmp/pre-backup.sh /etc/restic/pre-backup.sh
+  rm -f /tmp/profiles.yaml /tmp/pre-backup.sh
+"
+
+# 4. Test the profile before scheduling
+ssh -t "$HOST" 'sudo resticprofile --config /etc/restic/profiles.yaml --name default backup --dry-run'
+
+# 5. When dry-run looks clean, wire up systemd timers
+ssh -t "$HOST" 'sudo resticprofile --config /etc/restic/profiles.yaml schedule'
+#    → installs restic-backup@<profile>.{service,timer} units for backup/forget/check
+
+# 6. Verify
+ssh -t "$HOST" 'sudo systemctl list-timers | grep restic'
+```
+
+## Prune ceremony (quarterly or as needed)
+
+Per-repo, when enough forgotten-but-still-on-disk snapshots accumulate:
+
+```bash
+# On the rest-server host (ana-docker or the Synology):
+# 1. Stop the append-only rest-server, start one without the flag
+#    (simplest: edit the stack's .env, remove --append-only from OPTIONS,
+#    docker compose up -d)
+
+# 2. From the client host, run prune
+ssh -t <host> 'sudo resticprofile --config /etc/restic/profiles.yaml --name default prune'
+
+# 3. Put --append-only back on the rest-server and docker compose up -d.
+```
+
+Alternatively stand up a second rest-server stack on port 8001 without `--append-only` and point prune runs at that endpoint; keep the append-only one for daily writes.
+
+## Monitoring
+
+Each successful run writes `/var/lib/restic/last-success` with the current unix timestamp (this is the `post-backup` hook in `profiles.yaml`). Beszel can be configured to alert when that file's mtime exceeds ~36 hours.
+
+```bash
+ssh <host> 'stat -c "%y %n" /var/lib/restic/last-success 2>/dev/null || echo no successful run yet'
+```
@@ -0,0 +1,75 @@
+#!/bin/bash
+# pre-backup.sh — ana-docker.
+# Runs as root from resticprofile's `run-before`, before `restic backup`.
+#
+# Produces DB dumps in /var/lib/restic/stage/ so the nightly restic
+# snapshot captures consistent point-in-time data for services whose
+# raw volume files are not safe to back up live.
+#
+# Containers handled here:
+#   - synapse-db    (Postgres 16)
+#   - seafile-mysql (MariaDB 10.6)
+#   - vaultwarden   (SQLite w/ WAL; online .backup via sqlite3 if available)
+#
+# Gitea's DB is external (hosted elsewhere in the LAN) so we only back
+# up its data volume; whoever owns the gitea Postgres backs it up
+# separately.
+#
+# Idempotent: clears and recreates its staging files each run.
+
+set -euo pipefail
+
+STAGE=/var/lib/restic/stage
+install -d -o root -g root -m 0700 "$STAGE"
+
+log() { printf '%s pre-backup(ana-docker): %s\n' "$(date -Is)" "$*"; }
+
+# Purge previous stage so stale dumps don't pile up and end up in the snapshot.
+find "$STAGE" -mindepth 1 -maxdepth 1 -exec rm -rf {} +
+
+# ---------- synapse-db (Postgres) ---------------------------------------------
+if docker inspect synapse-db >/dev/null 2>&1; then
+  log "dumping synapse postgres"
+  # -Fc  custom format, internally compressed + restore-to-subset friendly
+  docker exec synapse-db \
+    pg_dump -U synapse -d synapse -Fc --clean --if-exists \
+    > "$STAGE/synapse.pg_dump"
+else
+  log "skip synapse: container not present"
+fi
+
+# ---------- seafile-mysql (MariaDB) -------------------------------------------
+if docker inspect seafile-mysql >/dev/null 2>&1; then
+  log "dumping seafile mariadb"
+  # The root password lives in the container's own env (MYSQL_ROOT_PASSWORD);
+  # expand it inside the container so it never lands in the host's process list.
+  docker exec seafile-mysql sh -c \
+    'mysqldump -uroot -p"$MYSQL_ROOT_PASSWORD" --all-databases --single-transaction --quick 2>/dev/null' \
+    | gzip -c > "$STAGE/seafile.sql.gz"
+else
+  log "skip seafile: container not present"
+fi
+
+# ---------- vaultwarden (SQLite + WAL) ----------------------------------------
+# Vaultwarden uses SQLite in WAL mode. A live copy of db.sqlite3 + -wal is
+# usually recoverable, but sqlite3's own .backup pragma is the correct way
+# to get a consistent snapshot. If the vaultwarden image has sqlite3
+# available, use it; otherwise rely on restic backing up the raw volume.
+if docker inspect vaultwarden >/dev/null 2>&1; then
+  if docker exec vaultwarden sh -c 'command -v sqlite3 >/dev/null 2>&1'; then
+    log "dumping vaultwarden sqlite via .backup"
+    docker exec vaultwarden sqlite3 /data/db.sqlite3 \
+      ".backup /tmp/vaultwarden.sqlite3"
+    docker cp vaultwarden:/tmp/vaultwarden.sqlite3 "$STAGE/vaultwarden.sqlite3"
+    docker exec vaultwarden rm -f /tmp/vaultwarden.sqlite3
+  else
+    log "skip vaultwarden .backup: sqlite3 not in container (raw volume still included via restic)"
+  fi
+else
+  log "skip vaultwarden: container not present"
+fi
+
+# ---------- summary -----------------------------------------------------------
+size=$(du -sh "$STAGE" 2>/dev/null | awk '{print $1}')
+count=$(find "$STAGE" -type f | wc -l)
+log "stage ready: $count files, $size total"
@@ -0,0 +1,76 @@
+# resticprofile config for ana-docker.
+#
+# Writes to the Anaheim-side rest-server at 10.250.50.70 as user
+# `ana-docker`. The full REST URL (including HTTP basic-auth creds)
+# lives in /etc/restic/restic.env — loaded via env-file so this YAML
+# carries zero secrets and is safe to version-control.
+#
+# The client-side encryption passphrase lives in /etc/restic/password.
+
+version: "1"
+
+global:
+  priority: low
+  ionice: true
+  ionice-class: 2
+  ionice-level: 7
+  min-memory: 100
+
+default:
+  env-file: /etc/restic/restic.env          # provides RESTIC_REPOSITORY=rest:http://user:pw@…
+  env:
+    RESTIC_PASSWORD_FILE: /etc/restic/password
+  initialize: false                         # repo was created by `restic init`
+  lock: /var/lock/restic-ana-docker.lock
+
+  backup:
+    verbose: 1
+    run-before:
+      - /etc/restic/pre-backup.sh
+    run-after:
+      - date +%s > /var/lib/restic/last-success
+    source:
+      - /opt/docker
+      - /var/lib/docker/volumes
+      - /var/lib/restic/stage
+    exclude:
+      # Docker internals we never want in a backup
+      - /var/lib/docker/volumes/backingFsBlockDev
+      - /var/lib/docker/volumes/metadata.db
+      # Raw DB files — we dump them via pre-backup.sh into /var/lib/restic/stage
+      - /var/lib/docker/volumes/synapse-db-data
+      - /var/lib/docker/volumes/synapse_synapse-db-data
+      - /var/lib/docker/volumes/seafile_db
+      # Ephemeral / regenerable junk
+      - /opt/docker/compose/*/logs
+      - /opt/docker/conf/traefik-ana/acme.json   # secret material; excluded everywhere
+      - /opt/docker/conf/crowdsec/hub             # upstream-managed, regenerable
+      - "**/*.log"
+      - "**/*.log.*"
+      - "**/*.pid"
+    tag:
+      - host:ana-docker
+      - site:ana
+      - fleet:pfi
+    schedule: "*-*-* 01:00:00"
+    schedule-permission: system
+    schedule-log: /var/log/restic-backup.log
+
+  forget:
+    keep-daily: 7
+    keep-weekly: 4
+    keep-monthly: 12
+    keep-yearly: 3
+    # NOTE: no `prune: true` — rest-server runs with --append-only, which
+    # blocks the destructive half of prune. See README.md "Prune ceremony".
+    tag:
+      - host:ana-docker
+    schedule: "*-*-* 03:00:00"
+    schedule-permission: system
+    schedule-log: /var/log/restic-forget.log
+
+  check:
+    read-data-subset: 10%
+    schedule: "Sun *-*-* 05:00:00"
+    schedule-permission: system
+    schedule-log: /var/log/restic-check.log