diff --git a/configs/restic/esh-docker-vm/README.md b/configs/restic/esh-docker-vm/README.md index 066f3ab..5161a1f 100644 --- a/configs/restic/esh-docker-vm/README.md +++ b/configs/restic/esh-docker-vm/README.md @@ -29,25 +29,34 @@ Cross-site writes to `rest-server-ana` at `10.250.50.70:8000/esh-docker-vm/`. ## Pre-backup hooks Unique to this host: most containers don't bundle sqlite3, so -`pre-backup.sh` runs sqlite3 and pg_dump **from the host** against the -volume bind-mount paths. Simpler than building custom images for HA +`pre-backup.sh` runs sqlite3 **from the host** against the volume +bind-mount paths. Simpler than building custom images for HA and pgadmin. (uptime-kuma moved to ana-docker on 2026-09-22 and its block was removed; see the header of `pre-backup.sh` for why a stale block took the whole backup down.) +**paperless-ngx's database is NOT dumped here** (removed 2026-09-23, operator +decision). It lives on esh-vm-db (`10.0.50.60`) and is backed up at the source +by that host's fail-closed `pg_dumpall` — see `configs/restic/esh-vm-db/`. +The host-side `pg_dump` this hook used to run had failed auth every night +since 2026-04-24 behind a WARN, and its `> paperless.pg_dump` redirect left a +**0-byte file in every snapshot** that looked like a dump. It was the only consumer of +`/etc/restic/dbcreds.env`, so the creds file and its template went too. +Paperless's *media* volumes are still captured under `/var/lib/docker/volumes`. + | Service | DB | Approach | |---|---|---| -| paperless-ngx | external Postgres `10.0.50.60` / `paperless-ng` | host pg_dump | | home-assistant | `/var/lib/docker/.../homeassistant_v2.db` | host sqlite3 | | pgadmin | `/var/lib/docker/.../pgadmin4.db` | host sqlite3 | | calibre-web-automated | `/config/app.db` inside container | in-container sqlite3 (it has the binary) | ## Host prerequisites -Needs `sqlite3` and `postgresql-client` installed on esh-docker-vm: +Needs `sqlite3` installed on esh-docker-vm (`postgresql-client` was only for +the removed paperless dump; it is still installed, and harmless): ```bash -ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 postgresql-client restic' +ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 restic' ``` (restic too — not installed on this host yet.) @@ -57,7 +66,7 @@ ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 postgresql- ### 1. Install restic + db-client tooling ```bash -ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y restic sqlite3 postgresql-client' +ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y restic sqlite3' ``` ### 2. Add `esh-docker-vm` entry on ana-docker rest-server @@ -84,16 +93,6 @@ ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/restic.env && chmod 600 /e ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/password && chmod 600 /etc/restic/password"' # paste: # Enter, Ctrl-D - -# Install DB creds (from dbcreds.env.example — edit in a temp file first) -cp configs/restic/esh-docker-vm/dbcreds.env.example /tmp/dbcreds.env -${EDITOR:-vi} /tmp/dbcreds.env # set PAPERLESS_PGPASS to real value -scp /tmp/dbcreds.env esh-docker-vm:/tmp/ -ssh -t esh-docker-vm ' - sudo install -o root -g root -m 0600 /tmp/dbcreds.env /etc/restic/dbcreds.env && - rm /tmp/dbcreds.env -' -shred -u /tmp/dbcreds.env 2>/dev/null || rm -f /tmp/dbcreds.env ``` ### 4. Init the repo @@ -126,7 +125,6 @@ ssh -t esh-docker-vm 'sudo ls -lh /var/lib/restic/stage/' ``` Expected files: -- `paperless.pg_dump` — should be 1–50 MB depending on doc count - `home-assistant.sqlite3` — ~50 MB (matches live DB size) - `calibre-web-automated.app.db` — ~250 KB - `pgadmin4.db` — ~200 KB @@ -182,8 +180,11 @@ ssh -t esh-docker-vm ' ## Follow-ups after restic is proven (separate session) -- **Rotate paperless-ngx Postgres password.** Currently `paperless-ng` — - trivially weak. Update the DB, the compose, and `dbcreds.env`. +- **Rotate paperless-ngx Postgres password.** Was `paperless-ng` — + trivially weak. Update the DB and the compose (no copy lives on this + backup path any more). The removed backup copy failed auth from + 2026-04-24 on, so it may already have changed at the DB side; confirm + before assuming. - **Retire offen/docker-volume-backup sidecars** on paperless-ngx + pgadmin stacks after ~1 week of clean restic runs. Delete the compose's sidecar services + the tarballs under `/mnt/backup/docker/esh-vm-docker/`. diff --git a/configs/restic/esh-docker-vm/dbcreds.env.example b/configs/restic/esh-docker-vm/dbcreds.env.example deleted file mode 100644 index 5581727..0000000 --- a/configs/restic/esh-docker-vm/dbcreds.env.example +++ /dev/null @@ -1,22 +0,0 @@ -# /etc/restic/dbcreds.env on esh-docker-vm — consumed by pre-backup.sh. -# -# Deploy to the host as: -# sudo install -o root -g root -m 0600 dbcreds.env /etc/restic/dbcreds.env -# -# Never commit the real file — it carries production DB passwords. This -# template is the only thing tracked in the repo. - -# --- Paperless-ngx → external Postgres on PFI-Postgres VM (10.0.50.60) ------ -# NOTE (2026-04-21): the current password in the paperless-ngx compose is -# literally "paperless-ng" — trivially weak. Rotate at the DB side before -# this backup is considered secure. -PAPERLESS_PGHOST=10.0.50.60 -PAPERLESS_PGPORT=5432 -PAPERLESS_PGUSER=paperless-ng -PAPERLESS_PGDB=paperless-ng -PAPERLESS_PGPASS=replace-with-postgres-password - -# --- All other services on this host use local SQLite ------------------------ -# (home-assistant, pgadmin, uptime-kuma, calibre-web-automated) -# No external creds required — sqlite3 runs as root either on the host or -# inside the container depending on image. diff --git a/configs/restic/esh-docker-vm/pre-backup.sh b/configs/restic/esh-docker-vm/pre-backup.sh index 0a825cd..d680d1c 100644 --- a/configs/restic/esh-docker-vm/pre-backup.sh +++ b/configs/restic/esh-docker-vm/pre-backup.sh @@ -6,12 +6,10 @@ # whose raw volume files risk inconsistency during live restic capture. # # Unique approach for this host: most containers don't bundle sqlite3, -# so we run sqlite3 and pg_dump from the HOST against the volume -# bind-mount paths. Requires sqlite3 + postgresql-client installed -# on esh-docker-vm (apt install sqlite3 postgresql-client). +# so we run sqlite3 from the HOST against the volume bind-mount paths. +# Requires sqlite3 installed on esh-docker-vm (apt install sqlite3). # # Services handled: -# - paperless-ngx (external Postgres on 10.0.50.60 — pg_dump from host) # - home-assistant (local SQLite in volume — sqlite3 .backup from host) # - calibre-web-automated (local SQLite — sqlite3 .backup inside container, has sqlite3) # - pgadmin (local SQLite in volume — sqlite3 .backup from host) @@ -21,8 +19,13 @@ # `docker ps | grep` lookup exited 1 and set -e aborted this script, and # resticprofile then skipped the WHOLE host backup (stale 09-22 → 09-23). # -# External DB credentials live in /etc/restic/dbcreds.env (root:600). -# Template: configs/restic/esh-docker-vm/dbcreds.env.example. +# paperless-ngx's Postgres (on esh-vm-db, 10.0.50.60) was also dumped from +# here until 2026-09-23, when the block was removed by operator decision. It +# is backed up at the source by esh-vm-db's own fail-closed pg_dumpall, and +# this copy had failed auth every night since 2026-04-24 behind a WARN +# nobody read, leaving a 0-byte paperless.pg_dump in every snapshot. That +# block was the only consumer of /etc/restic/dbcreds.env, so the creds +# loader went with it. # # Errors in individual blocks log a WARN; whole script doesn't abort. # That only holds if every lookup for an optional service sits inside an @@ -41,11 +44,6 @@ warn() { log "WARN: $*" >&2; } # Purge previous stage so stale dumps don't pile up in the snapshot. find "$STAGE" -mindepth 1 -maxdepth 1 -exec rm -rf {} + -# Load external-DB creds -if [ -r /etc/restic/dbcreds.env ]; then - set -a; . /etc/restic/dbcreds.env; set +a -fi - # Helper: host-side sqlite .backup against a volume-bind path. # $1 = source .db path (host absolute, typically under /var/lib/docker/volumes/.../_data/) # $2 = stage filename (just the leaf name) @@ -68,25 +66,6 @@ host_sqlite_backup() { fi } -# ---------- paperless-ngx (external Postgres on 10.0.50.60) ------------------- -if docker inspect paperless-ngx-webserver-1 >/dev/null 2>&1; then - if [ -z "${PAPERLESS_PGPASS:-}" ]; then - warn "paperless-ngx: PAPERLESS_PGPASS unset in /etc/restic/dbcreds.env — skipping" - elif ! command -v pg_dump >/dev/null 2>&1; then - warn "paperless-ngx: pg_dump not installed — apt install postgresql-client" - else - log "dumping paperless postgres (${PAPERLESS_PGHOST}:${PAPERLESS_PGPORT:-5432})" - PGPASSWORD="$PAPERLESS_PGPASS" pg_dump \ - -h "$PAPERLESS_PGHOST" -p "${PAPERLESS_PGPORT:-5432}" \ - -U "$PAPERLESS_PGUSER" -d "$PAPERLESS_PGDB" \ - -Fc --clean --if-exists \ - > "$STAGE/paperless.pg_dump" \ - || warn "paperless pg_dump failed" - fi -else - log "skip paperless: container not present" -fi - # ---------- home-assistant (SQLite in named volume, host-side .backup) -------- # HA's DB is ~50MB and actively written. SQLite .backup is the proper way # to grab a consistent snapshot while HA is running. diff --git a/playbooks/esh-docker-vm-restic-drop-kuma-block.yaml b/playbooks/esh-docker-vm-restic-drop-kuma-block.yaml index fc612d0..2122231 100644 --- a/playbooks/esh-docker-vm-restic-drop-kuma-block.yaml +++ b/playbooks/esh-docker-vm-restic-drop-kuma-block.yaml @@ -24,6 +24,12 @@ steps: dest: /etc/restic/pre-backup.sh mode: '0700' + # elway's sudo upload is scp-as-user then `sudo mv`, so the file lands owned + # by the SSH user. This hook is executed by root: it must be root-owned. + - name: Make the hook root-owned + sudo: true + shell: chown root:root /etc/restic/pre-backup.sh + verify: - name: Hook parses under bash sudo: true @@ -36,3 +42,7 @@ verify: - name: Hook runs to completion (stage summary line reached) sudo: true shell: /etc/restic/pre-backup.sh 2>&1 | grep -q 'stage ready:' + + - name: Hook is root:root 0700 + sudo: true + shell: test "$(stat -c '%U:%G %a' /etc/restic/pre-backup.sh)" = "root:root 700" diff --git a/playbooks/esh-docker-vm-restic-drop-paperless-block.yaml b/playbooks/esh-docker-vm-restic-drop-paperless-block.yaml new file mode 100644 index 0000000..5748b5e --- /dev/null +++ b/playbooks/esh-docker-vm-restic-drop-paperless-block.yaml @@ -0,0 +1,61 @@ +# esh-docker-vm: install the pre-backup hook without the paperless pg_dump +# block (operator decision 2026-09-23). +# +# paperless-ngx's Postgres is backed up at the source by esh-vm-db's +# fail-closed pg_dumpall. This host's second copy had been failing auth on a +# stale password for months behind a WARN. That block was the only consumer of +# /etc/restic/dbcreds.env, so the creds file is MOVED (not deleted) into the +# repair dir, out of the live config path. +# +# Rerunnable: the preserve/move steps are `creates:`-guarded; the chown step +# re-applies root ownership after every upload. + +steps: + - name: Preserve the pre-change hook + sudo: true + shell: | + set -eu + install -d -m 0700 /var/lib/restic/repair-20260923 + cp -p /etc/restic/pre-backup.sh /var/lib/restic/repair-20260923/pre-backup.sh.pre-paperless + creates: /var/lib/restic/repair-20260923/pre-backup.sh.pre-paperless + + - name: Move the now-unused dbcreds.env out of /etc/restic + sudo: true + shell: mv /etc/restic/dbcreds.env /var/lib/restic/repair-20260923/dbcreds.env + creates: /var/lib/restic/repair-20260923/dbcreds.env + + - name: Install the hook without the paperless block + sudo: true + upload: + src: configs/restic/esh-docker-vm/pre-backup.sh + dest: /etc/restic/pre-backup.sh + mode: '0700' + + # elway's sudo upload is scp-as-user then `sudo mv`, so the file lands owned + # by the SSH user. This hook is executed by root: it must be root-owned. + - name: Make the hook root-owned + sudo: true + shell: chown root:root /etc/restic/pre-backup.sh /var/lib/restic/repair-20260923/pre-backup.sh.pre-paperless + +verify: + - name: Hook parses under bash + sudo: true + shell: bash -n /etc/restic/pre-backup.sh + + - name: No paperless dump or creds loader left in the live hook + sudo: true + shell: "! grep -q -E 'PAPERLESS_PG|dbcreds.env\"?$|pg_dump \\\\' /etc/restic/pre-backup.sh" + + - name: dbcreds.env is gone from /etc/restic + sudo: true + shell: test ! -e /etc/restic/dbcreds.env + + - name: Hook runs to completion with no WARN lines + sudo: true + shell: | + out=$(/etc/restic/pre-backup.sh 2>&1) + printf '%s\n' "$out" | grep -q 'stage ready:' && ! printf '%s\n' "$out" | grep -q 'WARN' + + - name: Hook is root:root 0700 + sudo: true + shell: test "$(stat -c '%U:%G %a' /etc/restic/pre-backup.sh)" = "root:root 700"