fix(restic/esh-docker-vm): stop dumping paperless's Postgres from this host
Operator decision. paperless-ngx's database lives on esh-vm-db and is backed up at the source by that host's fail-closed pg_dumpall. esh-docker-vm's second copy had failed auth every night since 2026-04-24 behind a WARN. Its `> paperless.pg_dump` redirect left a 0-byte file in every snapshot (confirmed in snapshot 6ec9f74f), which looked like a dump but held nothing. The block was the only consumer of /etc/restic/dbcreds.env, so the creds loader is gone and the template dbcreds.env.example is deleted. The host file was moved (not deleted) to /var/lib/restic/repair-20260923/. Paperless's media volumes are still captured under /var/lib/docker/volumes. Also fixes ownership. elway's sudo upload does scp-as-user then `sudo mv`, so the hook deployed at 08:04 landed infra-ops:infra-ops even though root executes it. Both esh-docker-vm playbooks now chown it back to root and verify root:root 700. Verified: the live hook hash matches canonical (e0d3ddcef1bddf43), and the manual backup saved snapshot decfae71 with 3 staged dumps and no WARN lines.
This commit is contained in:
@@ -24,6 +24,12 @@ steps:
|
||||
dest: /etc/restic/pre-backup.sh
|
||||
mode: '0700'
|
||||
|
||||
# elway's sudo upload is scp-as-user then `sudo mv`, so the file lands owned
|
||||
# by the SSH user. This hook is executed by root: it must be root-owned.
|
||||
- name: Make the hook root-owned
|
||||
sudo: true
|
||||
shell: chown root:root /etc/restic/pre-backup.sh
|
||||
|
||||
verify:
|
||||
- name: Hook parses under bash
|
||||
sudo: true
|
||||
@@ -36,3 +42,7 @@ verify:
|
||||
- name: Hook runs to completion (stage summary line reached)
|
||||
sudo: true
|
||||
shell: /etc/restic/pre-backup.sh 2>&1 | grep -q 'stage ready:'
|
||||
|
||||
- name: Hook is root:root 0700
|
||||
sudo: true
|
||||
shell: test "$(stat -c '%U:%G %a' /etc/restic/pre-backup.sh)" = "root:root 700"
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
# esh-docker-vm: install the pre-backup hook without the paperless pg_dump
|
||||
# block (operator decision 2026-09-23).
|
||||
#
|
||||
# paperless-ngx's Postgres is backed up at the source by esh-vm-db's
|
||||
# fail-closed pg_dumpall. This host's second copy had been failing auth on a
|
||||
# stale password for months behind a WARN. That block was the only consumer of
|
||||
# /etc/restic/dbcreds.env, so the creds file is MOVED (not deleted) into the
|
||||
# repair dir, out of the live config path.
|
||||
#
|
||||
# Rerunnable: the preserve/move steps are `creates:`-guarded; the chown step
|
||||
# re-applies root ownership after every upload.
|
||||
|
||||
steps:
|
||||
- name: Preserve the pre-change hook
|
||||
sudo: true
|
||||
shell: |
|
||||
set -eu
|
||||
install -d -m 0700 /var/lib/restic/repair-20260923
|
||||
cp -p /etc/restic/pre-backup.sh /var/lib/restic/repair-20260923/pre-backup.sh.pre-paperless
|
||||
creates: /var/lib/restic/repair-20260923/pre-backup.sh.pre-paperless
|
||||
|
||||
- name: Move the now-unused dbcreds.env out of /etc/restic
|
||||
sudo: true
|
||||
shell: mv /etc/restic/dbcreds.env /var/lib/restic/repair-20260923/dbcreds.env
|
||||
creates: /var/lib/restic/repair-20260923/dbcreds.env
|
||||
|
||||
- name: Install the hook without the paperless block
|
||||
sudo: true
|
||||
upload:
|
||||
src: configs/restic/esh-docker-vm/pre-backup.sh
|
||||
dest: /etc/restic/pre-backup.sh
|
||||
mode: '0700'
|
||||
|
||||
# elway's sudo upload is scp-as-user then `sudo mv`, so the file lands owned
|
||||
# by the SSH user. This hook is executed by root: it must be root-owned.
|
||||
- name: Make the hook root-owned
|
||||
sudo: true
|
||||
shell: chown root:root /etc/restic/pre-backup.sh /var/lib/restic/repair-20260923/pre-backup.sh.pre-paperless
|
||||
|
||||
verify:
|
||||
- name: Hook parses under bash
|
||||
sudo: true
|
||||
shell: bash -n /etc/restic/pre-backup.sh
|
||||
|
||||
- name: No paperless dump or creds loader left in the live hook
|
||||
sudo: true
|
||||
shell: "! grep -q -E 'PAPERLESS_PG|dbcreds.env\"?$|pg_dump \\\\' /etc/restic/pre-backup.sh"
|
||||
|
||||
- name: dbcreds.env is gone from /etc/restic
|
||||
sudo: true
|
||||
shell: test ! -e /etc/restic/dbcreds.env
|
||||
|
||||
- name: Hook runs to completion with no WARN lines
|
||||
sudo: true
|
||||
shell: |
|
||||
out=$(/etc/restic/pre-backup.sh 2>&1)
|
||||
printf '%s\n' "$out" | grep -q 'stage ready:' && ! printf '%s\n' "$out" | grep -q 'WARN'
|
||||
|
||||
- name: Hook is root:root 0700
|
||||
sudo: true
|
||||
shell: test "$(stat -c '%U:%G %a' /etc/restic/pre-backup.sh)" = "root:root 700"
|
||||
Reference in New Issue
Block a user