restic/esh-docker-vm: profile + DB hooks + deploy guide

Closes the last file-level backup gap. Primary ESH home-lab Docker host
— five services with state worth consistent dumps:

  - paperless-ngx → external Postgres on 10.0.50.60 (host pg_dump)
  - home-assistant → local SQLite ~50MB (host sqlite3 .backup)
  - calibre-web-automated → local SQLite (in-container sqlite3)
  - pgadmin → local SQLite (host sqlite3)
  - uptime-kuma → local SQLite (host sqlite3; container name may vary
    after force-recreate, detect by label)

Unique to this host: HA/pgadmin/uptime-kuma images don't bundle sqlite3.
Rather than maintaining custom images, pre-backup.sh runs sqlite3 from
the HOST against the volume bind-mount paths. Requires sqlite3 +
postgresql-client installed on esh-docker-vm.

Cross-site writes to rest-server-ana since ESH has no local rest-server.

NFS mounts (/mnt/{backup,books,compose,documents}) explicitly excluded
— hundreds of GB of NAS-side content backed up at the NAS layer. Also
excludes offen-sidecar buffer volumes (paperless + pgadmin currently
run offen/docker-volume-backup alongside — retire once restic has a
week of clean runs).

Found in audit (non-blocking but noted in README follow-ups):
  - paperless-ngx Postgres password is literally "paperless-ng" —
    trivially weak, rotate at next opportunity.

configs/restic/esh-docker-vm/README.md

@@ -0,0 +1,194 @@
+# restic / esh-docker-vm
+
+Primary ESH home-lab Docker host. VM on `esh-pve`. Covered by vzdump at
+the hypervisor (3/3 on esh-pve after VM 108 retirement); this adds
+file-level restic with app-consistent DB dumps.
+
+Cross-site writes to `rest-server-ana` at `10.250.50.70:8000/esh-docker-vm/`.
+
+## What's backed up
+
+| Path | Purpose |
+|---|---|
+| `/opt/docker` | Compose + bind-mounted conf (~12 MB) |
+| `/etc` | Host config |
+| `/root` | Root's scripts / history / keys |
+| `/home` | User home dirs |
+| `/var/lib/docker/volumes` | HA / paperless / pgadmin / CWA / dockge / agents |
+| `/var/lib/restic/stage` | DB dumps from pre-backup.sh |
+
+## Critically excluded
+
+- **NFS mounts from `10.0.50.50`** — `/mnt/{backup,books,compose,documents}`.
+  Debian NAS protects these at its own layer.
+- **`/mnt/backup/docker/esh-vm-docker/*`** — offen-sidecar tarballs on
+  NFS; once restic proves itself for a week, retire those sidecars.
+- Docker internals (`backingFsBlockDev`, `metadata.db`).
+- Logs, pids, user caches.
+
+## Pre-backup hooks
+
+Unique to this host: most containers don't bundle sqlite3, so
+`pre-backup.sh` runs sqlite3 and pg_dump **from the host** against the
+volume bind-mount paths. Simpler than building custom images for HA,
+pgadmin, and uptime-kuma.
+
+| Service | DB | Approach |
+|---|---|---|
+| paperless-ngx | external Postgres `10.0.50.60` / `paperless-ng` | host pg_dump |
+| home-assistant | `/var/lib/docker/.../homeassistant_v2.db` | host sqlite3 |
+| pgadmin | `/var/lib/docker/.../pgadmin4.db` | host sqlite3 |
+| uptime-kuma | `/var/lib/docker/.../kuma.db` | host sqlite3 |
+| calibre-web-automated | `/config/app.db` inside container | in-container sqlite3 (it has the binary) |
+
+## Host prerequisites
+
+Needs `sqlite3` and `postgresql-client` installed on esh-docker-vm:
+
+```bash
+ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 postgresql-client restic'
+```
+
+(restic too — not installed on this host yet.)
+
+## Deploy (one-time setup)
+
+### 1. Install restic + db-client tooling
+
+```bash
+ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y restic sqlite3 postgresql-client'
+```
+
+### 2. Add `esh-docker-vm` entry on ana-docker rest-server
+
+```bash
+ssh ana-docker '
+  docker run --rm httpd:2.4-alpine htpasswd -nbB esh-docker-vm "<NEW-PW>" \
+    >> /mnt/backup/restic/repo/ana/.htpasswd
+'
+# no sudo — /mnt/backup is NFS with root_squash; dir is lkraven-owned
+```
+
+### 3. Install restic creds on esh-docker-vm
+
+```bash
+ssh -t esh-docker-vm 'sudo install -d -o root -g root -m 0700 /etc/restic /var/lib/restic'
+
+# REST URL
+ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/restic.env && chmod 600 /etc/restic/restic.env"'
+# paste: RESTIC_REPOSITORY=rest:http://esh-docker-vm:<HTPASSWD-PW>@10.250.50.70:8000/esh-docker-vm/
+# Enter, Ctrl-D
+
+# Repo passphrase
+ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/password && chmod 600 /etc/restic/password"'
+# paste: <REPO-PASSPHRASE>
+# Enter, Ctrl-D
+
+# Install DB creds (from dbcreds.env.example — edit in a temp file first)
+cp configs/restic/esh-docker-vm/dbcreds.env.example /tmp/dbcreds.env
+${EDITOR:-vi} /tmp/dbcreds.env   # set PAPERLESS_PGPASS to real value
+scp /tmp/dbcreds.env esh-docker-vm:/tmp/
+ssh -t esh-docker-vm '
+  sudo install -o root -g root -m 0600 /tmp/dbcreds.env /etc/restic/dbcreds.env &&
+  rm /tmp/dbcreds.env
+'
+shred -u /tmp/dbcreds.env 2>/dev/null || rm -f /tmp/dbcreds.env
+```
+
+### 4. Init the repo
+
+```bash
+ssh -t esh-docker-vm '
+  sudo bash -c "
+    set -a; . /etc/restic/restic.env; set +a
+    restic init
+  "
+'
+# Type passphrase twice at the prompt — same as installed in /etc/restic/password.
+```
+
+### 5. Deploy pre-backup.sh
+
+```bash
+scp configs/restic/esh-docker-vm/pre-backup.sh esh-docker-vm:/tmp/pre-backup.sh
+ssh -t esh-docker-vm '
+  sudo install -o root -g root -m 0700 /tmp/pre-backup.sh /etc/restic/pre-backup.sh &&
+  rm /tmp/pre-backup.sh
+'
+```
+
+### 6. Test the hook by itself (without running the full backup)
+
+```bash
+ssh -t esh-docker-vm 'sudo /etc/restic/pre-backup.sh'
+ssh -t esh-docker-vm 'sudo ls -lh /var/lib/restic/stage/'
+```
+
+Expected files:
+- `paperless.pg_dump` — should be 1–50 MB depending on doc count
+- `home-assistant.sqlite3` — ~50 MB (matches live DB size)
+- `calibre-web-automated.app.db` — ~250 KB
+- `pgadmin4.db` — ~200 KB
+- `uptime-kuma.kuma.db` — varies (history retention)
+
+Any `WARN:` lines in the hook output indicate a block that was skipped —
+read them, debug one at a time.
+
+### 7. Install resticprofile
+
+```bash
+ssh -t esh-docker-vm '
+  curl -sfL https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh \
+    | sudo sh -s -- -b /usr/local/bin &&
+  /usr/local/bin/resticprofile version
+'
+```
+
+### 8. Deploy profile
+
+```bash
+scp configs/restic/esh-docker-vm/profiles.yaml esh-docker-vm:/tmp/profiles.yaml
+ssh -t esh-docker-vm '
+  sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml &&
+  rm /tmp/profiles.yaml &&
+  sudo resticprofile --config /etc/restic/profiles.yaml show
+'
+```
+
+### 9. Enable timers + first backup
+
+```bash
+ssh -t esh-docker-vm '
+  sudo resticprofile --config /etc/restic/profiles.yaml schedule --all &&
+  sudo resticprofile --config /etc/restic/profiles.yaml backup --verbose
+'
+```
+
+Expected first snapshot: **~1–2 GB** (HA DB is the dominant file, plus
+some Docker volumes). If you see much more, inspect `/home` and `/var/lib/docker/volumes`
+for surprise content.
+
+## Restoring
+
+```bash
+ssh -t esh-docker-vm '
+  sudo bash -c "
+    set -a; . /etc/restic/restic.env; set +a
+    RESTIC_PASSWORD_FILE=/etc/restic/password \
+      restic restore --target /tmp/restore latest --path /opt/docker
+  "
+'
+```
+
+## Follow-ups after restic is proven (separate session)
+
+- **Rotate paperless-ngx Postgres password.** Currently `paperless-ng` —
+  trivially weak. Update the DB, the compose, and `dbcreds.env`.
+- **Retire offen/docker-volume-backup sidecars** on paperless-ngx + pgadmin
+  stacks after ~1 week of clean restic runs. Delete the compose's
+  sidecar services + the tarballs under `/mnt/backup/docker/esh-vm-docker/`.
+
+## Recreating the repo
+
+Same pattern as ana-ml2 — see `configs/restic/ana-ml2/README.md`. Repo
+path on ana-docker is `/mnt/backup/restic/repo/ana/esh-docker-vm/`.

configs/restic/esh-docker-vm/dbcreds.env.example

@@ -0,0 +1,22 @@
+# /etc/restic/dbcreds.env on esh-docker-vm — consumed by pre-backup.sh.
+#
+# Deploy to the host as:
+#   sudo install -o root -g root -m 0600 dbcreds.env /etc/restic/dbcreds.env
+#
+# Never commit the real file — it carries production DB passwords. This
+# template is the only thing tracked in the repo.
+
+# --- Paperless-ngx → external Postgres on PFI-Postgres VM (10.0.50.60) ------
+# NOTE (2026-04-21): the current password in the paperless-ngx compose is
+# literally "paperless-ng" — trivially weak. Rotate at the DB side before
+# this backup is considered secure.
+PAPERLESS_PGHOST=10.0.50.60
+PAPERLESS_PGPORT=5432
+PAPERLESS_PGUSER=paperless-ng
+PAPERLESS_PGDB=paperless-ng
+PAPERLESS_PGPASS=replace-with-postgres-password
+
+# --- All other services on this host use local SQLite ------------------------
+# (home-assistant, pgadmin, uptime-kuma, calibre-web-automated)
+# No external creds required — sqlite3 runs as root either on the host or
+# inside the container depending on image.

configs/restic/esh-docker-vm/pre-backup.sh

@@ -0,0 +1,129 @@
+#!/bin/bash
+# pre-backup.sh — esh-docker-vm.
+# Runs as root from resticprofile's `run-before`.
+#
+# Produces consistent DB dumps in /var/lib/restic/stage/ for services
+# whose raw volume files risk inconsistency during live restic capture.
+#
+# Unique approach for this host: most containers don't bundle sqlite3,
+# so we run sqlite3 and pg_dump from the HOST against the volume
+# bind-mount paths. Requires sqlite3 + postgresql-client installed
+# on esh-docker-vm (apt install sqlite3 postgresql-client).
+#
+# Services handled:
+#   - paperless-ngx     (external Postgres on 10.0.50.60 — pg_dump from host)
+#   - home-assistant    (local SQLite in volume — sqlite3 .backup from host)
+#   - calibre-web-automated (local SQLite — sqlite3 .backup inside container, has sqlite3)
+#   - pgadmin           (local SQLite in volume — sqlite3 .backup from host)
+#   - uptime-kuma       (local SQLite in volume — sqlite3 .backup from host)
+#
+# External DB credentials live in /etc/restic/dbcreds.env (root:600).
+# Template: configs/restic/esh-docker-vm/dbcreds.env.example.
+#
+# Errors in individual blocks log a WARN; whole script doesn't abort.
+
+set -euo pipefail
+
+STAGE=/var/lib/restic/stage
+install -d -o root -g root -m 0700 "$STAGE"
+
+log()  { printf '%s pre-backup(esh-docker-vm): %s\n' "$(date -Is)" "$*"; }
+warn() { log "WARN: $*" >&2; }
+
+# Purge previous stage so stale dumps don't pile up in the snapshot.
+find "$STAGE" -mindepth 1 -maxdepth 1 -exec rm -rf {} +
+
+# Load external-DB creds
+if [ -r /etc/restic/dbcreds.env ]; then
+  set -a; . /etc/restic/dbcreds.env; set +a
+fi
+
+# Helper: host-side sqlite .backup against a volume-bind path.
+#   $1 = source .db path (host absolute, typically under /var/lib/docker/volumes/.../_data/)
+#   $2 = stage filename (just the leaf name)
+host_sqlite_backup() {
+  local src="$1" dst="$STAGE/$2"
+  if ! command -v sqlite3 >/dev/null 2>&1; then
+    warn "sqlite3 not on host (apt install sqlite3) — skipping $2"
+    return 1
+  fi
+  if [ ! -f "$src" ]; then
+    warn "source db missing: $src — skipping $2"
+    return 1
+  fi
+  if sqlite3 "$src" ".backup '$dst'" 2>/dev/null; then
+    log "dumped $2 ($(du -h "$dst" 2>/dev/null | cut -f1))"
+  else
+    warn "sqlite3 .backup failed for $src"
+    rm -f "$dst"
+    return 1
+  fi
+}
+
+# ---------- paperless-ngx (external Postgres on 10.0.50.60) -------------------
+if docker inspect paperless-ngx-webserver-1 >/dev/null 2>&1; then
+  if [ -z "${PAPERLESS_PGPASS:-}" ]; then
+    warn "paperless-ngx: PAPERLESS_PGPASS unset in /etc/restic/dbcreds.env — skipping"
+  elif ! command -v pg_dump >/dev/null 2>&1; then
+    warn "paperless-ngx: pg_dump not installed — apt install postgresql-client"
+  else
+    log "dumping paperless postgres (${PAPERLESS_PGHOST}:${PAPERLESS_PGPORT:-5432})"
+    PGPASSWORD="$PAPERLESS_PGPASS" pg_dump \
+      -h "$PAPERLESS_PGHOST" -p "${PAPERLESS_PGPORT:-5432}" \
+      -U "$PAPERLESS_PGUSER" -d "$PAPERLESS_PGDB" \
+      -Fc --clean --if-exists \
+      > "$STAGE/paperless.pg_dump" \
+      || warn "paperless pg_dump failed"
+  fi
+else
+  log "skip paperless: container not present"
+fi
+
+# ---------- home-assistant (SQLite in named volume, host-side .backup) --------
+# HA's DB is ~50MB and actively written. SQLite .backup is the proper way
+# to grab a consistent snapshot while HA is running.
+if docker inspect homeassistant >/dev/null 2>&1; then
+  HA_DB="/var/lib/docker/volumes/homeassistant_homeassistant_data/_data/home-assistant_v2.db"
+  host_sqlite_backup "$HA_DB" "home-assistant.sqlite3" || true
+else
+  log "skip home-assistant: container not present"
+fi
+
+# ---------- calibre-web-automated (SQLite, sqlite3 inside container) ---------
+if docker inspect calibre-web-automated >/dev/null 2>&1; then
+  log "dumping CWA sqlite via in-container .backup"
+  if docker exec calibre-web-automated sqlite3 /config/app.db ".backup /tmp/cwa-app.db" 2>/dev/null; then
+    docker cp calibre-web-automated:/tmp/cwa-app.db "$STAGE/calibre-web-automated.app.db" \
+      && docker exec calibre-web-automated rm -f /tmp/cwa-app.db \
+      || warn "CWA copy/cleanup failed"
+  else
+    warn "CWA sqlite .backup failed"
+  fi
+else
+  log "skip calibre-web-automated: container not present"
+fi
+
+# ---------- pgadmin (SQLite in named volume, host-side .backup) --------------
+if docker inspect pgadmin4_container >/dev/null 2>&1; then
+  PG_DB="/var/lib/docker/volumes/pgadmin_pgadmin-data/_data/pgadmin4.db"
+  host_sqlite_backup "$PG_DB" "pgadmin4.db" || true
+else
+  log "skip pgadmin: container not present"
+fi
+
+# ---------- uptime-kuma (SQLite in named volume, host-side .backup) ----------
+# Container name may vary after force-recreates (e.g. <hash>_uptime-kuma).
+# Detect by label rather than hardcoded name.
+UK_CONTAINER=$(docker ps --filter "label=com.docker.compose.project=uptimekuma" --format "{{.Names}}" | head -1)
+[ -z "$UK_CONTAINER" ] && UK_CONTAINER=$(docker ps --format "{{.Names}}" | grep -E "uptime.kuma" | head -1)
+if [ -n "$UK_CONTAINER" ]; then
+  UK_DB="/var/lib/docker/volumes/uptimekuma_uptime-kuma/_data/kuma.db"
+  host_sqlite_backup "$UK_DB" "uptime-kuma.kuma.db" || true
+else
+  log "skip uptime-kuma: no container matching"
+fi
+
+# ---------- summary -----------------------------------------------------------
+size=$(du -sh "$STAGE" 2>/dev/null | awk '{print $1}')
+count=$(find "$STAGE" -type f | wc -l)
+log "stage ready: $count files, $size total"

configs/restic/esh-docker-vm/profiles.yaml

@@ -0,0 +1,93 @@
+# resticprofile config for esh-docker-vm.
+#
+# ESH home-lab Docker host — VM on esh-pve. Covered by vzdump at the
+# hypervisor (3/3 on esh-pve after the VM 108 retirement), and by this
+# file-level restic for fast per-file restore + app-consistent DB dumps.
+#
+# Writes cross-site to rest-server-ana (10.250.50.70:8000/esh-docker-vm/)
+# because the ESH site has no local rest-server.
+#
+# Multiple DB-bearing services live here (see pre-backup.sh). Several
+# containers don't bundle sqlite3, so the pre-backup script runs
+# sqlite3 and pg_dump from the HOST against volume bind-mount paths
+# (simpler than building custom container images).
+
+version: "1"
+
+global:
+  priority: low
+  ionice: true
+  ionice-class: 2
+  ionice-level: 7
+  min-memory: 100
+
+default:
+  env-file: /etc/restic/restic.env
+  env:
+    RESTIC_PASSWORD_FILE: /etc/restic/password
+  initialize: false
+  lock: /var/lock/restic-esh-docker-vm.lock
+
+  backup:
+    verbose: 1
+    run-before:
+      - /etc/restic/pre-backup.sh
+    run-after:
+      - date +%s > /var/lib/restic/last-success
+    source:
+      - /opt/docker              # compose files + bind-mount conf (~12 MB)
+      - /etc                     # host config
+      - /root                    # root's ad-hoc scripts, ssh keys, history
+      - /home                    # user home dirs
+      - /var/lib/docker/volumes  # HA, paperless, pgadmin, CWA, dockge, etc.
+      - /var/lib/restic/stage    # DB dumps produced by pre-backup.sh
+    exclude:
+      # CRITICAL: NFS mounts from 10.0.50.50 — hundreds of GB / TB at least.
+      # /mnt/backup specifically holds offen-sidecar tarballs which are
+      # redundant once restic is authoritative.
+      - /mnt/backup
+      - /mnt/books
+      - /mnt/compose
+      - /mnt/documents
+      # Docker internals
+      - /var/lib/docker/volumes/backingFsBlockDev
+      - /var/lib/docker/volumes/metadata.db
+      # Offen sidecar's buffer / tmp (if any left behind)
+      - /var/lib/docker/volumes/*_offen_backup_data
+      # Ephemeral / regenerable
+      - /opt/docker/compose/*/logs
+      - "**/*.log"
+      - "**/*.log.*"
+      - "**/*.pid"
+      # Per-user shell / app noise
+      - /root/.cache
+      - /root/.local/share/Trash
+      - /root/.python_history
+      - /home/*/.cache
+      - /home/*/.local/share/Trash
+      - /home/*/.npm
+      - /home/*/.mozilla/firefox/*/Cache
+    tag:
+      - host:esh-docker-vm
+      - site:esh
+      - fleet:home-lab
+    schedule: "*-*-* 01:00:00"
+    schedule-permission: system
+    schedule-log: /var/log/restic-backup.log
+
+  forget:
+    keep-daily: 7
+    keep-weekly: 4
+    keep-monthly: 12
+    keep-yearly: 3
+    tag:
+      - host:esh-docker-vm
+    schedule: "*-*-* 03:00:00"
+    schedule-permission: system
+    schedule-log: /var/log/restic-forget.log
+
+  check:
+    read-data-subset: 10%
+    schedule: "Sun *-*-* 05:00:00"
+    schedule-permission: system
+    schedule-log: /var/log/restic-check.log