fix(restic/esh-docker-vm): stop dumping paperless's Postgres from this host
Operator decision. paperless-ngx's database lives on esh-vm-db and is backed up at the source by that host's fail-closed pg_dumpall. esh-docker-vm's second copy had failed auth every night since 2026-04-24 behind a WARN. Its `> paperless.pg_dump` redirect left a 0-byte file in every snapshot (confirmed in snapshot 6ec9f74f), which looked like a dump but held nothing. The block was the only consumer of /etc/restic/dbcreds.env, so the creds loader is gone and the template dbcreds.env.example is deleted. The host file was moved (not deleted) to /var/lib/restic/repair-20260923/. Paperless's media volumes are still captured under /var/lib/docker/volumes. Also fixes ownership. elway's sudo upload does scp-as-user then `sudo mv`, so the hook deployed at 08:04 landed infra-ops:infra-ops even though root executes it. Both esh-docker-vm playbooks now chown it back to root and verify root:root 700. Verified: the live hook hash matches canonical (e0d3ddcef1bddf43), and the manual backup saved snapshot decfae71 with 3 staged dumps and no WARN lines.
This commit is contained in:
@@ -29,25 +29,34 @@ Cross-site writes to `rest-server-ana` at `10.250.50.70:8000/esh-docker-vm/`.
|
||||
## Pre-backup hooks
|
||||
|
||||
Unique to this host: most containers don't bundle sqlite3, so
|
||||
`pre-backup.sh` runs sqlite3 and pg_dump **from the host** against the
|
||||
volume bind-mount paths. Simpler than building custom images for HA
|
||||
`pre-backup.sh` runs sqlite3 **from the host** against the volume
|
||||
bind-mount paths. Simpler than building custom images for HA
|
||||
and pgadmin. (uptime-kuma moved to ana-docker on 2026-09-22 and its block
|
||||
was removed; see the header of `pre-backup.sh` for why a stale block took
|
||||
the whole backup down.)
|
||||
|
||||
**paperless-ngx's database is NOT dumped here** (removed 2026-09-23, operator
|
||||
decision). It lives on esh-vm-db (`10.0.50.60`) and is backed up at the source
|
||||
by that host's fail-closed `pg_dumpall` — see `configs/restic/esh-vm-db/`.
|
||||
The host-side `pg_dump` this hook used to run had failed auth every night
|
||||
since 2026-04-24 behind a WARN, and its `> paperless.pg_dump` redirect left a
|
||||
**0-byte file in every snapshot** that looked like a dump. It was the only consumer of
|
||||
`/etc/restic/dbcreds.env`, so the creds file and its template went too.
|
||||
Paperless's *media* volumes are still captured under `/var/lib/docker/volumes`.
|
||||
|
||||
| Service | DB | Approach |
|
||||
|---|---|---|
|
||||
| paperless-ngx | external Postgres `10.0.50.60` / `paperless-ng` | host pg_dump |
|
||||
| home-assistant | `/var/lib/docker/.../homeassistant_v2.db` | host sqlite3 |
|
||||
| pgadmin | `/var/lib/docker/.../pgadmin4.db` | host sqlite3 |
|
||||
| calibre-web-automated | `/config/app.db` inside container | in-container sqlite3 (it has the binary) |
|
||||
|
||||
## Host prerequisites
|
||||
|
||||
Needs `sqlite3` and `postgresql-client` installed on esh-docker-vm:
|
||||
Needs `sqlite3` installed on esh-docker-vm (`postgresql-client` was only for
|
||||
the removed paperless dump; it is still installed, and harmless):
|
||||
|
||||
```bash
|
||||
ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 postgresql-client restic'
|
||||
ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 restic'
|
||||
```
|
||||
|
||||
(restic too — not installed on this host yet.)
|
||||
@@ -57,7 +66,7 @@ ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 postgresql-
|
||||
### 1. Install restic + db-client tooling
|
||||
|
||||
```bash
|
||||
ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y restic sqlite3 postgresql-client'
|
||||
ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y restic sqlite3'
|
||||
```
|
||||
|
||||
### 2. Add `esh-docker-vm` entry on ana-docker rest-server
|
||||
@@ -84,16 +93,6 @@ ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/restic.env && chmod 600 /e
|
||||
ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/password && chmod 600 /etc/restic/password"'
|
||||
# paste: <REPO-PASSPHRASE>
|
||||
# Enter, Ctrl-D
|
||||
|
||||
# Install DB creds (from dbcreds.env.example — edit in a temp file first)
|
||||
cp configs/restic/esh-docker-vm/dbcreds.env.example /tmp/dbcreds.env
|
||||
${EDITOR:-vi} /tmp/dbcreds.env # set PAPERLESS_PGPASS to real value
|
||||
scp /tmp/dbcreds.env esh-docker-vm:/tmp/
|
||||
ssh -t esh-docker-vm '
|
||||
sudo install -o root -g root -m 0600 /tmp/dbcreds.env /etc/restic/dbcreds.env &&
|
||||
rm /tmp/dbcreds.env
|
||||
'
|
||||
shred -u /tmp/dbcreds.env 2>/dev/null || rm -f /tmp/dbcreds.env
|
||||
```
|
||||
|
||||
### 4. Init the repo
|
||||
@@ -126,7 +125,6 @@ ssh -t esh-docker-vm 'sudo ls -lh /var/lib/restic/stage/'
|
||||
```
|
||||
|
||||
Expected files:
|
||||
- `paperless.pg_dump` — should be 1–50 MB depending on doc count
|
||||
- `home-assistant.sqlite3` — ~50 MB (matches live DB size)
|
||||
- `calibre-web-automated.app.db` — ~250 KB
|
||||
- `pgadmin4.db` — ~200 KB
|
||||
@@ -182,8 +180,11 @@ ssh -t esh-docker-vm '
|
||||
|
||||
## Follow-ups after restic is proven (separate session)
|
||||
|
||||
- **Rotate paperless-ngx Postgres password.** Currently `paperless-ng` —
|
||||
trivially weak. Update the DB, the compose, and `dbcreds.env`.
|
||||
- **Rotate paperless-ngx Postgres password.** Was `paperless-ng` —
|
||||
trivially weak. Update the DB and the compose (no copy lives on this
|
||||
backup path any more). The removed backup copy failed auth from
|
||||
2026-04-24 on, so it may already have changed at the DB side; confirm
|
||||
before assuming.
|
||||
- **Retire offen/docker-volume-backup sidecars** on paperless-ngx + pgadmin
|
||||
stacks after ~1 week of clean restic runs. Delete the compose's
|
||||
sidecar services + the tarballs under `/mnt/backup/docker/esh-vm-docker/`.
|
||||
|
||||
@@ -1,22 +0,0 @@
|
||||
# /etc/restic/dbcreds.env on esh-docker-vm — consumed by pre-backup.sh.
|
||||
#
|
||||
# Deploy to the host as:
|
||||
# sudo install -o root -g root -m 0600 dbcreds.env /etc/restic/dbcreds.env
|
||||
#
|
||||
# Never commit the real file — it carries production DB passwords. This
|
||||
# template is the only thing tracked in the repo.
|
||||
|
||||
# --- Paperless-ngx → external Postgres on PFI-Postgres VM (10.0.50.60) ------
|
||||
# NOTE (2026-04-21): the current password in the paperless-ngx compose is
|
||||
# literally "paperless-ng" — trivially weak. Rotate at the DB side before
|
||||
# this backup is considered secure.
|
||||
PAPERLESS_PGHOST=10.0.50.60
|
||||
PAPERLESS_PGPORT=5432
|
||||
PAPERLESS_PGUSER=paperless-ng
|
||||
PAPERLESS_PGDB=paperless-ng
|
||||
PAPERLESS_PGPASS=replace-with-postgres-password
|
||||
|
||||
# --- All other services on this host use local SQLite ------------------------
|
||||
# (home-assistant, pgadmin, uptime-kuma, calibre-web-automated)
|
||||
# No external creds required — sqlite3 runs as root either on the host or
|
||||
# inside the container depending on image.
|
||||
@@ -6,12 +6,10 @@
|
||||
# whose raw volume files risk inconsistency during live restic capture.
|
||||
#
|
||||
# Unique approach for this host: most containers don't bundle sqlite3,
|
||||
# so we run sqlite3 and pg_dump from the HOST against the volume
|
||||
# bind-mount paths. Requires sqlite3 + postgresql-client installed
|
||||
# on esh-docker-vm (apt install sqlite3 postgresql-client).
|
||||
# so we run sqlite3 from the HOST against the volume bind-mount paths.
|
||||
# Requires sqlite3 installed on esh-docker-vm (apt install sqlite3).
|
||||
#
|
||||
# Services handled:
|
||||
# - paperless-ngx (external Postgres on 10.0.50.60 — pg_dump from host)
|
||||
# - home-assistant (local SQLite in volume — sqlite3 .backup from host)
|
||||
# - calibre-web-automated (local SQLite — sqlite3 .backup inside container, has sqlite3)
|
||||
# - pgadmin (local SQLite in volume — sqlite3 .backup from host)
|
||||
@@ -21,8 +19,13 @@
|
||||
# `docker ps | grep` lookup exited 1 and set -e aborted this script, and
|
||||
# resticprofile then skipped the WHOLE host backup (stale 09-22 → 09-23).
|
||||
#
|
||||
# External DB credentials live in /etc/restic/dbcreds.env (root:600).
|
||||
# Template: configs/restic/esh-docker-vm/dbcreds.env.example.
|
||||
# paperless-ngx's Postgres (on esh-vm-db, 10.0.50.60) was also dumped from
|
||||
# here until 2026-09-23, when the block was removed by operator decision. It
|
||||
# is backed up at the source by esh-vm-db's own fail-closed pg_dumpall, and
|
||||
# this copy had failed auth every night since 2026-04-24 behind a WARN
|
||||
# nobody read, leaving a 0-byte paperless.pg_dump in every snapshot. That
|
||||
# block was the only consumer of /etc/restic/dbcreds.env, so the creds
|
||||
# loader went with it.
|
||||
#
|
||||
# Errors in individual blocks log a WARN; whole script doesn't abort.
|
||||
# That only holds if every lookup for an optional service sits inside an
|
||||
@@ -41,11 +44,6 @@ warn() { log "WARN: $*" >&2; }
|
||||
# Purge previous stage so stale dumps don't pile up in the snapshot.
|
||||
find "$STAGE" -mindepth 1 -maxdepth 1 -exec rm -rf {} +
|
||||
|
||||
# Load external-DB creds
|
||||
if [ -r /etc/restic/dbcreds.env ]; then
|
||||
set -a; . /etc/restic/dbcreds.env; set +a
|
||||
fi
|
||||
|
||||
# Helper: host-side sqlite .backup against a volume-bind path.
|
||||
# $1 = source .db path (host absolute, typically under /var/lib/docker/volumes/.../_data/)
|
||||
# $2 = stage filename (just the leaf name)
|
||||
@@ -68,25 +66,6 @@ host_sqlite_backup() {
|
||||
fi
|
||||
}
|
||||
|
||||
# ---------- paperless-ngx (external Postgres on 10.0.50.60) -------------------
|
||||
if docker inspect paperless-ngx-webserver-1 >/dev/null 2>&1; then
|
||||
if [ -z "${PAPERLESS_PGPASS:-}" ]; then
|
||||
warn "paperless-ngx: PAPERLESS_PGPASS unset in /etc/restic/dbcreds.env — skipping"
|
||||
elif ! command -v pg_dump >/dev/null 2>&1; then
|
||||
warn "paperless-ngx: pg_dump not installed — apt install postgresql-client"
|
||||
else
|
||||
log "dumping paperless postgres (${PAPERLESS_PGHOST}:${PAPERLESS_PGPORT:-5432})"
|
||||
PGPASSWORD="$PAPERLESS_PGPASS" pg_dump \
|
||||
-h "$PAPERLESS_PGHOST" -p "${PAPERLESS_PGPORT:-5432}" \
|
||||
-U "$PAPERLESS_PGUSER" -d "$PAPERLESS_PGDB" \
|
||||
-Fc --clean --if-exists \
|
||||
> "$STAGE/paperless.pg_dump" \
|
||||
|| warn "paperless pg_dump failed"
|
||||
fi
|
||||
else
|
||||
log "skip paperless: container not present"
|
||||
fi
|
||||
|
||||
# ---------- home-assistant (SQLite in named volume, host-side .backup) --------
|
||||
# HA's DB is ~50MB and actively written. SQLite .backup is the proper way
|
||||
# to grab a consistent snapshot while HA is running.
|
||||
|
||||
@@ -24,6 +24,12 @@ steps:
|
||||
dest: /etc/restic/pre-backup.sh
|
||||
mode: '0700'
|
||||
|
||||
# elway's sudo upload is scp-as-user then `sudo mv`, so the file lands owned
|
||||
# by the SSH user. This hook is executed by root: it must be root-owned.
|
||||
- name: Make the hook root-owned
|
||||
sudo: true
|
||||
shell: chown root:root /etc/restic/pre-backup.sh
|
||||
|
||||
verify:
|
||||
- name: Hook parses under bash
|
||||
sudo: true
|
||||
@@ -36,3 +42,7 @@ verify:
|
||||
- name: Hook runs to completion (stage summary line reached)
|
||||
sudo: true
|
||||
shell: /etc/restic/pre-backup.sh 2>&1 | grep -q 'stage ready:'
|
||||
|
||||
- name: Hook is root:root 0700
|
||||
sudo: true
|
||||
shell: test "$(stat -c '%U:%G %a' /etc/restic/pre-backup.sh)" = "root:root 700"
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
# esh-docker-vm: install the pre-backup hook without the paperless pg_dump
|
||||
# block (operator decision 2026-09-23).
|
||||
#
|
||||
# paperless-ngx's Postgres is backed up at the source by esh-vm-db's
|
||||
# fail-closed pg_dumpall. This host's second copy had been failing auth on a
|
||||
# stale password for months behind a WARN. That block was the only consumer of
|
||||
# /etc/restic/dbcreds.env, so the creds file is MOVED (not deleted) into the
|
||||
# repair dir, out of the live config path.
|
||||
#
|
||||
# Rerunnable: the preserve/move steps are `creates:`-guarded; the chown step
|
||||
# re-applies root ownership after every upload.
|
||||
|
||||
steps:
|
||||
- name: Preserve the pre-change hook
|
||||
sudo: true
|
||||
shell: |
|
||||
set -eu
|
||||
install -d -m 0700 /var/lib/restic/repair-20260923
|
||||
cp -p /etc/restic/pre-backup.sh /var/lib/restic/repair-20260923/pre-backup.sh.pre-paperless
|
||||
creates: /var/lib/restic/repair-20260923/pre-backup.sh.pre-paperless
|
||||
|
||||
- name: Move the now-unused dbcreds.env out of /etc/restic
|
||||
sudo: true
|
||||
shell: mv /etc/restic/dbcreds.env /var/lib/restic/repair-20260923/dbcreds.env
|
||||
creates: /var/lib/restic/repair-20260923/dbcreds.env
|
||||
|
||||
- name: Install the hook without the paperless block
|
||||
sudo: true
|
||||
upload:
|
||||
src: configs/restic/esh-docker-vm/pre-backup.sh
|
||||
dest: /etc/restic/pre-backup.sh
|
||||
mode: '0700'
|
||||
|
||||
# elway's sudo upload is scp-as-user then `sudo mv`, so the file lands owned
|
||||
# by the SSH user. This hook is executed by root: it must be root-owned.
|
||||
- name: Make the hook root-owned
|
||||
sudo: true
|
||||
shell: chown root:root /etc/restic/pre-backup.sh /var/lib/restic/repair-20260923/pre-backup.sh.pre-paperless
|
||||
|
||||
verify:
|
||||
- name: Hook parses under bash
|
||||
sudo: true
|
||||
shell: bash -n /etc/restic/pre-backup.sh
|
||||
|
||||
- name: No paperless dump or creds loader left in the live hook
|
||||
sudo: true
|
||||
shell: "! grep -q -E 'PAPERLESS_PG|dbcreds.env\"?$|pg_dump \\\\' /etc/restic/pre-backup.sh"
|
||||
|
||||
- name: dbcreds.env is gone from /etc/restic
|
||||
sudo: true
|
||||
shell: test ! -e /etc/restic/dbcreds.env
|
||||
|
||||
- name: Hook runs to completion with no WARN lines
|
||||
sudo: true
|
||||
shell: |
|
||||
out=$(/etc/restic/pre-backup.sh 2>&1)
|
||||
printf '%s\n' "$out" | grep -q 'stage ready:' && ! printf '%s\n' "$out" | grep -q 'WARN'
|
||||
|
||||
- name: Hook is root:root 0700
|
||||
sudo: true
|
||||
shell: test "$(stat -c '%U:%G %a' /etc/restic/pre-backup.sh)" = "root:root 700"
|
||||
Reference in New Issue
Block a user