fix(restic/esh-docker-vm): stop dumping paperless's Postgres from this host

Operator decision. paperless-ngx's database lives on esh-vm-db and is backed
up at the source by that host's fail-closed pg_dumpall. esh-docker-vm's
second copy had failed auth every night since 2026-04-24 behind a WARN. Its
`> paperless.pg_dump` redirect left a 0-byte file in every snapshot
(confirmed in snapshot 6ec9f74f), which looked like a dump but held nothing.

The block was the only consumer of /etc/restic/dbcreds.env, so the creds
loader is gone and the template dbcreds.env.example is deleted. The host
file was moved (not deleted) to /var/lib/restic/repair-20260923/.
Paperless's media volumes are still captured under /var/lib/docker/volumes.

Also fixes ownership. elway's sudo upload does scp-as-user then `sudo mv`,
so the hook deployed at 08:04 landed infra-ops:infra-ops even though root
executes it. Both esh-docker-vm playbooks now chown it back to root and
verify root:root 700.

Verified: the live hook hash matches canonical (e0d3ddcef1bddf43), and
the manual backup saved snapshot decfae71 with 3 staged dumps and no WARN
lines.
This commit is contained in:
vh
2026-09-23 09:18:03 -07:00
parent 25e41d2ab5
commit 6e8da46a28
5 changed files with 100 additions and 71 deletions
@@ -24,6 +24,12 @@ steps:
dest: /etc/restic/pre-backup.sh
mode: '0700'
# elway's sudo upload is scp-as-user then `sudo mv`, so the file lands owned
# by the SSH user. This hook is executed by root: it must be root-owned.
- name: Make the hook root-owned
sudo: true
shell: chown root:root /etc/restic/pre-backup.sh
verify:
- name: Hook parses under bash
sudo: true
@@ -36,3 +42,7 @@ verify:
- name: Hook runs to completion (stage summary line reached)
sudo: true
shell: /etc/restic/pre-backup.sh 2>&1 | grep -q 'stage ready:'
- name: Hook is root:root 0700
sudo: true
shell: test "$(stat -c '%U:%G %a' /etc/restic/pre-backup.sh)" = "root:root 700"
@@ -0,0 +1,61 @@
# esh-docker-vm: install the pre-backup hook without the paperless pg_dump
# block (operator decision 2026-09-23).
#
# paperless-ngx's Postgres is backed up at the source by esh-vm-db's
# fail-closed pg_dumpall. This host's second copy had been failing auth on a
# stale password for months behind a WARN. That block was the only consumer of
# /etc/restic/dbcreds.env, so the creds file is MOVED (not deleted) into the
# repair dir, out of the live config path.
#
# Rerunnable: the preserve/move steps are `creates:`-guarded; the chown step
# re-applies root ownership after every upload.
steps:
- name: Preserve the pre-change hook
sudo: true
shell: |
set -eu
install -d -m 0700 /var/lib/restic/repair-20260923
cp -p /etc/restic/pre-backup.sh /var/lib/restic/repair-20260923/pre-backup.sh.pre-paperless
creates: /var/lib/restic/repair-20260923/pre-backup.sh.pre-paperless
- name: Move the now-unused dbcreds.env out of /etc/restic
sudo: true
shell: mv /etc/restic/dbcreds.env /var/lib/restic/repair-20260923/dbcreds.env
creates: /var/lib/restic/repair-20260923/dbcreds.env
- name: Install the hook without the paperless block
sudo: true
upload:
src: configs/restic/esh-docker-vm/pre-backup.sh
dest: /etc/restic/pre-backup.sh
mode: '0700'
# elway's sudo upload is scp-as-user then `sudo mv`, so the file lands owned
# by the SSH user. This hook is executed by root: it must be root-owned.
- name: Make the hook root-owned
sudo: true
shell: chown root:root /etc/restic/pre-backup.sh /var/lib/restic/repair-20260923/pre-backup.sh.pre-paperless
verify:
- name: Hook parses under bash
sudo: true
shell: bash -n /etc/restic/pre-backup.sh
- name: No paperless dump or creds loader left in the live hook
sudo: true
shell: "! grep -q -E 'PAPERLESS_PG|dbcreds.env\"?$|pg_dump \\\\' /etc/restic/pre-backup.sh"
- name: dbcreds.env is gone from /etc/restic
sudo: true
shell: test ! -e /etc/restic/dbcreds.env
- name: Hook runs to completion with no WARN lines
sudo: true
shell: |
out=$(/etc/restic/pre-backup.sh 2>&1)
printf '%s\n' "$out" | grep -q 'stage ready:' && ! printf '%s\n' "$out" | grep -q 'WARN'
- name: Hook is root:root 0700
sudo: true
shell: test "$(stat -c '%U:%G %a' /etc/restic/pre-backup.sh)" = "root:root 700"