diff --git a/configs/restic/esh-docker-vm/README.md b/configs/restic/esh-docker-vm/README.md new file mode 100644 index 0000000..cae409f --- /dev/null +++ b/configs/restic/esh-docker-vm/README.md @@ -0,0 +1,194 @@ +# restic / esh-docker-vm + +Primary ESH home-lab Docker host. VM on `esh-pve`. Covered by vzdump at +the hypervisor (3/3 on esh-pve after VM 108 retirement); this adds +file-level restic with app-consistent DB dumps. + +Cross-site writes to `rest-server-ana` at `10.250.50.70:8000/esh-docker-vm/`. + +## What's backed up + +| Path | Purpose | +|---|---| +| `/opt/docker` | Compose + bind-mounted conf (~12 MB) | +| `/etc` | Host config | +| `/root` | Root's scripts / history / keys | +| `/home` | User home dirs | +| `/var/lib/docker/volumes` | HA / paperless / pgadmin / CWA / dockge / agents | +| `/var/lib/restic/stage` | DB dumps from pre-backup.sh | + +## Critically excluded + +- **NFS mounts from `10.0.50.50`** — `/mnt/{backup,books,compose,documents}`. + Debian NAS protects these at its own layer. +- **`/mnt/backup/docker/esh-vm-docker/*`** — offen-sidecar tarballs on + NFS; once restic proves itself for a week, retire those sidecars. +- Docker internals (`backingFsBlockDev`, `metadata.db`). +- Logs, pids, user caches. + +## Pre-backup hooks + +Unique to this host: most containers don't bundle sqlite3, so +`pre-backup.sh` runs sqlite3 and pg_dump **from the host** against the +volume bind-mount paths. Simpler than building custom images for HA, +pgadmin, and uptime-kuma. + +| Service | DB | Approach | +|---|---|---| +| paperless-ngx | external Postgres `10.0.50.60` / `paperless-ng` | host pg_dump | +| home-assistant | `/var/lib/docker/.../homeassistant_v2.db` | host sqlite3 | +| pgadmin | `/var/lib/docker/.../pgadmin4.db` | host sqlite3 | +| uptime-kuma | `/var/lib/docker/.../kuma.db` | host sqlite3 | +| calibre-web-automated | `/config/app.db` inside container | in-container sqlite3 (it has the binary) | + +## Host prerequisites + +Needs `sqlite3` and `postgresql-client` installed on esh-docker-vm: + +```bash +ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 postgresql-client restic' +``` + +(restic too — not installed on this host yet.) + +## Deploy (one-time setup) + +### 1. Install restic + db-client tooling + +```bash +ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y restic sqlite3 postgresql-client' +``` + +### 2. Add `esh-docker-vm` entry on ana-docker rest-server + +```bash +ssh ana-docker ' + docker run --rm httpd:2.4-alpine htpasswd -nbB esh-docker-vm "" \ + >> /mnt/backup/restic/repo/ana/.htpasswd +' +# no sudo — /mnt/backup is NFS with root_squash; dir is lkraven-owned +``` + +### 3. Install restic creds on esh-docker-vm + +```bash +ssh -t esh-docker-vm 'sudo install -d -o root -g root -m 0700 /etc/restic /var/lib/restic' + +# REST URL +ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/restic.env && chmod 600 /etc/restic/restic.env"' +# paste: RESTIC_REPOSITORY=rest:http://esh-docker-vm:@10.250.50.70:8000/esh-docker-vm/ +# Enter, Ctrl-D + +# Repo passphrase +ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/password && chmod 600 /etc/restic/password"' +# paste: +# Enter, Ctrl-D + +# Install DB creds (from dbcreds.env.example — edit in a temp file first) +cp configs/restic/esh-docker-vm/dbcreds.env.example /tmp/dbcreds.env +${EDITOR:-vi} /tmp/dbcreds.env # set PAPERLESS_PGPASS to real value +scp /tmp/dbcreds.env esh-docker-vm:/tmp/ +ssh -t esh-docker-vm ' + sudo install -o root -g root -m 0600 /tmp/dbcreds.env /etc/restic/dbcreds.env && + rm /tmp/dbcreds.env +' +shred -u /tmp/dbcreds.env 2>/dev/null || rm -f /tmp/dbcreds.env +``` + +### 4. Init the repo + +```bash +ssh -t esh-docker-vm ' + sudo bash -c " + set -a; . /etc/restic/restic.env; set +a + restic init + " +' +# Type passphrase twice at the prompt — same as installed in /etc/restic/password. +``` + +### 5. Deploy pre-backup.sh + +```bash +scp configs/restic/esh-docker-vm/pre-backup.sh esh-docker-vm:/tmp/pre-backup.sh +ssh -t esh-docker-vm ' + sudo install -o root -g root -m 0700 /tmp/pre-backup.sh /etc/restic/pre-backup.sh && + rm /tmp/pre-backup.sh +' +``` + +### 6. Test the hook by itself (without running the full backup) + +```bash +ssh -t esh-docker-vm 'sudo /etc/restic/pre-backup.sh' +ssh -t esh-docker-vm 'sudo ls -lh /var/lib/restic/stage/' +``` + +Expected files: +- `paperless.pg_dump` — should be 1–50 MB depending on doc count +- `home-assistant.sqlite3` — ~50 MB (matches live DB size) +- `calibre-web-automated.app.db` — ~250 KB +- `pgadmin4.db` — ~200 KB +- `uptime-kuma.kuma.db` — varies (history retention) + +Any `WARN:` lines in the hook output indicate a block that was skipped — +read them, debug one at a time. + +### 7. Install resticprofile + +```bash +ssh -t esh-docker-vm ' + curl -sfL https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh \ + | sudo sh -s -- -b /usr/local/bin && + /usr/local/bin/resticprofile version +' +``` + +### 8. Deploy profile + +```bash +scp configs/restic/esh-docker-vm/profiles.yaml esh-docker-vm:/tmp/profiles.yaml +ssh -t esh-docker-vm ' + sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml && + rm /tmp/profiles.yaml && + sudo resticprofile --config /etc/restic/profiles.yaml show +' +``` + +### 9. Enable timers + first backup + +```bash +ssh -t esh-docker-vm ' + sudo resticprofile --config /etc/restic/profiles.yaml schedule --all && + sudo resticprofile --config /etc/restic/profiles.yaml backup --verbose +' +``` + +Expected first snapshot: **~1–2 GB** (HA DB is the dominant file, plus +some Docker volumes). If you see much more, inspect `/home` and `/var/lib/docker/volumes` +for surprise content. + +## Restoring + +```bash +ssh -t esh-docker-vm ' + sudo bash -c " + set -a; . /etc/restic/restic.env; set +a + RESTIC_PASSWORD_FILE=/etc/restic/password \ + restic restore --target /tmp/restore latest --path /opt/docker + " +' +``` + +## Follow-ups after restic is proven (separate session) + +- **Rotate paperless-ngx Postgres password.** Currently `paperless-ng` — + trivially weak. Update the DB, the compose, and `dbcreds.env`. +- **Retire offen/docker-volume-backup sidecars** on paperless-ngx + pgadmin + stacks after ~1 week of clean restic runs. Delete the compose's + sidecar services + the tarballs under `/mnt/backup/docker/esh-vm-docker/`. + +## Recreating the repo + +Same pattern as ana-ml2 — see `configs/restic/ana-ml2/README.md`. Repo +path on ana-docker is `/mnt/backup/restic/repo/ana/esh-docker-vm/`. diff --git a/configs/restic/esh-docker-vm/dbcreds.env.example b/configs/restic/esh-docker-vm/dbcreds.env.example new file mode 100644 index 0000000..5581727 --- /dev/null +++ b/configs/restic/esh-docker-vm/dbcreds.env.example @@ -0,0 +1,22 @@ +# /etc/restic/dbcreds.env on esh-docker-vm — consumed by pre-backup.sh. +# +# Deploy to the host as: +# sudo install -o root -g root -m 0600 dbcreds.env /etc/restic/dbcreds.env +# +# Never commit the real file — it carries production DB passwords. This +# template is the only thing tracked in the repo. + +# --- Paperless-ngx → external Postgres on PFI-Postgres VM (10.0.50.60) ------ +# NOTE (2026-04-21): the current password in the paperless-ngx compose is +# literally "paperless-ng" — trivially weak. Rotate at the DB side before +# this backup is considered secure. +PAPERLESS_PGHOST=10.0.50.60 +PAPERLESS_PGPORT=5432 +PAPERLESS_PGUSER=paperless-ng +PAPERLESS_PGDB=paperless-ng +PAPERLESS_PGPASS=replace-with-postgres-password + +# --- All other services on this host use local SQLite ------------------------ +# (home-assistant, pgadmin, uptime-kuma, calibre-web-automated) +# No external creds required — sqlite3 runs as root either on the host or +# inside the container depending on image. diff --git a/configs/restic/esh-docker-vm/pre-backup.sh b/configs/restic/esh-docker-vm/pre-backup.sh new file mode 100644 index 0000000..ced29ad --- /dev/null +++ b/configs/restic/esh-docker-vm/pre-backup.sh @@ -0,0 +1,129 @@ +#!/bin/bash +# pre-backup.sh — esh-docker-vm. +# Runs as root from resticprofile's `run-before`. +# +# Produces consistent DB dumps in /var/lib/restic/stage/ for services +# whose raw volume files risk inconsistency during live restic capture. +# +# Unique approach for this host: most containers don't bundle sqlite3, +# so we run sqlite3 and pg_dump from the HOST against the volume +# bind-mount paths. Requires sqlite3 + postgresql-client installed +# on esh-docker-vm (apt install sqlite3 postgresql-client). +# +# Services handled: +# - paperless-ngx (external Postgres on 10.0.50.60 — pg_dump from host) +# - home-assistant (local SQLite in volume — sqlite3 .backup from host) +# - calibre-web-automated (local SQLite — sqlite3 .backup inside container, has sqlite3) +# - pgadmin (local SQLite in volume — sqlite3 .backup from host) +# - uptime-kuma (local SQLite in volume — sqlite3 .backup from host) +# +# External DB credentials live in /etc/restic/dbcreds.env (root:600). +# Template: configs/restic/esh-docker-vm/dbcreds.env.example. +# +# Errors in individual blocks log a WARN; whole script doesn't abort. + +set -euo pipefail + +STAGE=/var/lib/restic/stage +install -d -o root -g root -m 0700 "$STAGE" + +log() { printf '%s pre-backup(esh-docker-vm): %s\n' "$(date -Is)" "$*"; } +warn() { log "WARN: $*" >&2; } + +# Purge previous stage so stale dumps don't pile up in the snapshot. +find "$STAGE" -mindepth 1 -maxdepth 1 -exec rm -rf {} + + +# Load external-DB creds +if [ -r /etc/restic/dbcreds.env ]; then + set -a; . /etc/restic/dbcreds.env; set +a +fi + +# Helper: host-side sqlite .backup against a volume-bind path. +# $1 = source .db path (host absolute, typically under /var/lib/docker/volumes/.../_data/) +# $2 = stage filename (just the leaf name) +host_sqlite_backup() { + local src="$1" dst="$STAGE/$2" + if ! command -v sqlite3 >/dev/null 2>&1; then + warn "sqlite3 not on host (apt install sqlite3) — skipping $2" + return 1 + fi + if [ ! -f "$src" ]; then + warn "source db missing: $src — skipping $2" + return 1 + fi + if sqlite3 "$src" ".backup '$dst'" 2>/dev/null; then + log "dumped $2 ($(du -h "$dst" 2>/dev/null | cut -f1))" + else + warn "sqlite3 .backup failed for $src" + rm -f "$dst" + return 1 + fi +} + +# ---------- paperless-ngx (external Postgres on 10.0.50.60) ------------------- +if docker inspect paperless-ngx-webserver-1 >/dev/null 2>&1; then + if [ -z "${PAPERLESS_PGPASS:-}" ]; then + warn "paperless-ngx: PAPERLESS_PGPASS unset in /etc/restic/dbcreds.env — skipping" + elif ! command -v pg_dump >/dev/null 2>&1; then + warn "paperless-ngx: pg_dump not installed — apt install postgresql-client" + else + log "dumping paperless postgres (${PAPERLESS_PGHOST}:${PAPERLESS_PGPORT:-5432})" + PGPASSWORD="$PAPERLESS_PGPASS" pg_dump \ + -h "$PAPERLESS_PGHOST" -p "${PAPERLESS_PGPORT:-5432}" \ + -U "$PAPERLESS_PGUSER" -d "$PAPERLESS_PGDB" \ + -Fc --clean --if-exists \ + > "$STAGE/paperless.pg_dump" \ + || warn "paperless pg_dump failed" + fi +else + log "skip paperless: container not present" +fi + +# ---------- home-assistant (SQLite in named volume, host-side .backup) -------- +# HA's DB is ~50MB and actively written. SQLite .backup is the proper way +# to grab a consistent snapshot while HA is running. +if docker inspect homeassistant >/dev/null 2>&1; then + HA_DB="/var/lib/docker/volumes/homeassistant_homeassistant_data/_data/home-assistant_v2.db" + host_sqlite_backup "$HA_DB" "home-assistant.sqlite3" || true +else + log "skip home-assistant: container not present" +fi + +# ---------- calibre-web-automated (SQLite, sqlite3 inside container) --------- +if docker inspect calibre-web-automated >/dev/null 2>&1; then + log "dumping CWA sqlite via in-container .backup" + if docker exec calibre-web-automated sqlite3 /config/app.db ".backup /tmp/cwa-app.db" 2>/dev/null; then + docker cp calibre-web-automated:/tmp/cwa-app.db "$STAGE/calibre-web-automated.app.db" \ + && docker exec calibre-web-automated rm -f /tmp/cwa-app.db \ + || warn "CWA copy/cleanup failed" + else + warn "CWA sqlite .backup failed" + fi +else + log "skip calibre-web-automated: container not present" +fi + +# ---------- pgadmin (SQLite in named volume, host-side .backup) -------------- +if docker inspect pgadmin4_container >/dev/null 2>&1; then + PG_DB="/var/lib/docker/volumes/pgadmin_pgadmin-data/_data/pgadmin4.db" + host_sqlite_backup "$PG_DB" "pgadmin4.db" || true +else + log "skip pgadmin: container not present" +fi + +# ---------- uptime-kuma (SQLite in named volume, host-side .backup) ---------- +# Container name may vary after force-recreates (e.g. _uptime-kuma). +# Detect by label rather than hardcoded name. +UK_CONTAINER=$(docker ps --filter "label=com.docker.compose.project=uptimekuma" --format "{{.Names}}" | head -1) +[ -z "$UK_CONTAINER" ] && UK_CONTAINER=$(docker ps --format "{{.Names}}" | grep -E "uptime.kuma" | head -1) +if [ -n "$UK_CONTAINER" ]; then + UK_DB="/var/lib/docker/volumes/uptimekuma_uptime-kuma/_data/kuma.db" + host_sqlite_backup "$UK_DB" "uptime-kuma.kuma.db" || true +else + log "skip uptime-kuma: no container matching" +fi + +# ---------- summary ----------------------------------------------------------- +size=$(du -sh "$STAGE" 2>/dev/null | awk '{print $1}') +count=$(find "$STAGE" -type f | wc -l) +log "stage ready: $count files, $size total" diff --git a/configs/restic/esh-docker-vm/profiles.yaml b/configs/restic/esh-docker-vm/profiles.yaml new file mode 100644 index 0000000..edeacad --- /dev/null +++ b/configs/restic/esh-docker-vm/profiles.yaml @@ -0,0 +1,93 @@ +# resticprofile config for esh-docker-vm. +# +# ESH home-lab Docker host — VM on esh-pve. Covered by vzdump at the +# hypervisor (3/3 on esh-pve after the VM 108 retirement), and by this +# file-level restic for fast per-file restore + app-consistent DB dumps. +# +# Writes cross-site to rest-server-ana (10.250.50.70:8000/esh-docker-vm/) +# because the ESH site has no local rest-server. +# +# Multiple DB-bearing services live here (see pre-backup.sh). Several +# containers don't bundle sqlite3, so the pre-backup script runs +# sqlite3 and pg_dump from the HOST against volume bind-mount paths +# (simpler than building custom container images). + +version: "1" + +global: + priority: low + ionice: true + ionice-class: 2 + ionice-level: 7 + min-memory: 100 + +default: + env-file: /etc/restic/restic.env + env: + RESTIC_PASSWORD_FILE: /etc/restic/password + initialize: false + lock: /var/lock/restic-esh-docker-vm.lock + + backup: + verbose: 1 + run-before: + - /etc/restic/pre-backup.sh + run-after: + - date +%s > /var/lib/restic/last-success + source: + - /opt/docker # compose files + bind-mount conf (~12 MB) + - /etc # host config + - /root # root's ad-hoc scripts, ssh keys, history + - /home # user home dirs + - /var/lib/docker/volumes # HA, paperless, pgadmin, CWA, dockge, etc. + - /var/lib/restic/stage # DB dumps produced by pre-backup.sh + exclude: + # CRITICAL: NFS mounts from 10.0.50.50 — hundreds of GB / TB at least. + # /mnt/backup specifically holds offen-sidecar tarballs which are + # redundant once restic is authoritative. + - /mnt/backup + - /mnt/books + - /mnt/compose + - /mnt/documents + # Docker internals + - /var/lib/docker/volumes/backingFsBlockDev + - /var/lib/docker/volumes/metadata.db + # Offen sidecar's buffer / tmp (if any left behind) + - /var/lib/docker/volumes/*_offen_backup_data + # Ephemeral / regenerable + - /opt/docker/compose/*/logs + - "**/*.log" + - "**/*.log.*" + - "**/*.pid" + # Per-user shell / app noise + - /root/.cache + - /root/.local/share/Trash + - /root/.python_history + - /home/*/.cache + - /home/*/.local/share/Trash + - /home/*/.npm + - /home/*/.mozilla/firefox/*/Cache + tag: + - host:esh-docker-vm + - site:esh + - fleet:home-lab + schedule: "*-*-* 01:00:00" + schedule-permission: system + schedule-log: /var/log/restic-backup.log + + forget: + keep-daily: 7 + keep-weekly: 4 + keep-monthly: 12 + keep-yearly: 3 + tag: + - host:esh-docker-vm + schedule: "*-*-* 03:00:00" + schedule-permission: system + schedule-log: /var/log/restic-forget.log + + check: + read-data-subset: 10% + schedule: "Sun *-*-* 05:00:00" + schedule-permission: system + schedule-log: /var/log/restic-check.log