fix(restic/esh-docker-vm): stop dumping paperless's Postgres from this host

Operator decision. paperless-ngx's database lives on esh-vm-db and is backed
up at the source by that host's fail-closed pg_dumpall. esh-docker-vm's
second copy had failed auth every night since 2026-04-24 behind a WARN. Its
`> paperless.pg_dump` redirect left a 0-byte file in every snapshot
(confirmed in snapshot 6ec9f74f), which looked like a dump but held nothing.

The block was the only consumer of /etc/restic/dbcreds.env, so the creds
loader is gone and the template dbcreds.env.example is deleted. The host
file was moved (not deleted) to /var/lib/restic/repair-20260923/.
Paperless's media volumes are still captured under /var/lib/docker/volumes.

Also fixes ownership. elway's sudo upload does scp-as-user then `sudo mv`,
so the hook deployed at 08:04 landed infra-ops:infra-ops even though root
executes it. Both esh-docker-vm playbooks now chown it back to root and
verify root:root 700.

Verified: the live hook hash matches canonical (e0d3ddcef1bddf43), and
the manual backup saved snapshot decfae71 with 3 staged dumps and no WARN
lines.
This commit is contained in:
vh
2026-09-23 09:18:03 -07:00
parent 25e41d2ab5
commit 6e8da46a28
5 changed files with 100 additions and 71 deletions
+9 -30
View File
@@ -6,12 +6,10 @@
# whose raw volume files risk inconsistency during live restic capture.
#
# Unique approach for this host: most containers don't bundle sqlite3,
# so we run sqlite3 and pg_dump from the HOST against the volume
# bind-mount paths. Requires sqlite3 + postgresql-client installed
# on esh-docker-vm (apt install sqlite3 postgresql-client).
# so we run sqlite3 from the HOST against the volume bind-mount paths.
# Requires sqlite3 installed on esh-docker-vm (apt install sqlite3).
#
# Services handled:
# - paperless-ngx (external Postgres on 10.0.50.60 — pg_dump from host)
# - home-assistant (local SQLite in volume — sqlite3 .backup from host)
# - calibre-web-automated (local SQLite — sqlite3 .backup inside container, has sqlite3)
# - pgadmin (local SQLite in volume — sqlite3 .backup from host)
@@ -21,8 +19,13 @@
# `docker ps | grep` lookup exited 1 and set -e aborted this script, and
# resticprofile then skipped the WHOLE host backup (stale 09-22 → 09-23).
#
# External DB credentials live in /etc/restic/dbcreds.env (root:600).
# Template: configs/restic/esh-docker-vm/dbcreds.env.example.
# paperless-ngx's Postgres (on esh-vm-db, 10.0.50.60) was also dumped from
# here until 2026-09-23, when the block was removed by operator decision. It
# is backed up at the source by esh-vm-db's own fail-closed pg_dumpall, and
# this copy had failed auth every night since 2026-04-24 behind a WARN
# nobody read, leaving a 0-byte paperless.pg_dump in every snapshot. That
# block was the only consumer of /etc/restic/dbcreds.env, so the creds
# loader went with it.
#
# Errors in individual blocks log a WARN; whole script doesn't abort.
# That only holds if every lookup for an optional service sits inside an
@@ -41,11 +44,6 @@ warn() { log "WARN: $*" >&2; }
# Purge previous stage so stale dumps don't pile up in the snapshot.
find "$STAGE" -mindepth 1 -maxdepth 1 -exec rm -rf {} +
# Load external-DB creds
if [ -r /etc/restic/dbcreds.env ]; then
set -a; . /etc/restic/dbcreds.env; set +a
fi
# Helper: host-side sqlite .backup against a volume-bind path.
# $1 = source .db path (host absolute, typically under /var/lib/docker/volumes/.../_data/)
# $2 = stage filename (just the leaf name)
@@ -68,25 +66,6 @@ host_sqlite_backup() {
fi
}
# ---------- paperless-ngx (external Postgres on 10.0.50.60) -------------------
if docker inspect paperless-ngx-webserver-1 >/dev/null 2>&1; then
if [ -z "${PAPERLESS_PGPASS:-}" ]; then
warn "paperless-ngx: PAPERLESS_PGPASS unset in /etc/restic/dbcreds.env — skipping"
elif ! command -v pg_dump >/dev/null 2>&1; then
warn "paperless-ngx: pg_dump not installed — apt install postgresql-client"
else
log "dumping paperless postgres (${PAPERLESS_PGHOST}:${PAPERLESS_PGPORT:-5432})"
PGPASSWORD="$PAPERLESS_PGPASS" pg_dump \
-h "$PAPERLESS_PGHOST" -p "${PAPERLESS_PGPORT:-5432}" \
-U "$PAPERLESS_PGUSER" -d "$PAPERLESS_PGDB" \
-Fc --clean --if-exists \
> "$STAGE/paperless.pg_dump" \
|| warn "paperless pg_dump failed"
fi
else
log "skip paperless: container not present"
fi
# ---------- home-assistant (SQLite in named volume, host-side .backup) --------
# HA's DB is ~50MB and actively written. SQLite .backup is the proper way
# to grab a consistent snapshot while HA is running.