fix(restic/esh-docker-vm): stop dumping paperless's Postgres from this host

Operator decision. paperless-ngx's database lives on esh-vm-db and is backed
up at the source by that host's fail-closed pg_dumpall. esh-docker-vm's
second copy had failed auth every night since 2026-04-24 behind a WARN. Its
`> paperless.pg_dump` redirect left a 0-byte file in every snapshot
(confirmed in snapshot 6ec9f74f), which looked like a dump but held nothing.

The block was the only consumer of /etc/restic/dbcreds.env, so the creds
loader is gone and the template dbcreds.env.example is deleted. The host
file was moved (not deleted) to /var/lib/restic/repair-20260923/.
Paperless's media volumes are still captured under /var/lib/docker/volumes.

Also fixes ownership. elway's sudo upload does scp-as-user then `sudo mv`,
so the hook deployed at 08:04 landed infra-ops:infra-ops even though root
executes it. Both esh-docker-vm playbooks now chown it back to root and
verify root:root 700.

Verified: the live hook hash matches canonical (e0d3ddcef1bddf43), and
the manual backup saved snapshot decfae71 with 3 staged dumps and no WARN
lines.
This commit is contained in:
vh
2026-09-23 09:18:03 -07:00
parent 25e41d2ab5
commit 6e8da46a28
5 changed files with 100 additions and 71 deletions
+20 -19
View File
@@ -29,25 +29,34 @@ Cross-site writes to `rest-server-ana` at `10.250.50.70:8000/esh-docker-vm/`.
## Pre-backup hooks
Unique to this host: most containers don't bundle sqlite3, so
`pre-backup.sh` runs sqlite3 and pg_dump **from the host** against the
volume bind-mount paths. Simpler than building custom images for HA
`pre-backup.sh` runs sqlite3 **from the host** against the volume
bind-mount paths. Simpler than building custom images for HA
and pgadmin. (uptime-kuma moved to ana-docker on 2026-09-22 and its block
was removed; see the header of `pre-backup.sh` for why a stale block took
the whole backup down.)
**paperless-ngx's database is NOT dumped here** (removed 2026-09-23, operator
decision). It lives on esh-vm-db (`10.0.50.60`) and is backed up at the source
by that host's fail-closed `pg_dumpall` — see `configs/restic/esh-vm-db/`.
The host-side `pg_dump` this hook used to run had failed auth every night
since 2026-04-24 behind a WARN, and its `> paperless.pg_dump` redirect left a
**0-byte file in every snapshot** that looked like a dump. It was the only consumer of
`/etc/restic/dbcreds.env`, so the creds file and its template went too.
Paperless's *media* volumes are still captured under `/var/lib/docker/volumes`.
| Service | DB | Approach |
|---|---|---|
| paperless-ngx | external Postgres `10.0.50.60` / `paperless-ng` | host pg_dump |
| home-assistant | `/var/lib/docker/.../homeassistant_v2.db` | host sqlite3 |
| pgadmin | `/var/lib/docker/.../pgadmin4.db` | host sqlite3 |
| calibre-web-automated | `/config/app.db` inside container | in-container sqlite3 (it has the binary) |
## Host prerequisites
Needs `sqlite3` and `postgresql-client` installed on esh-docker-vm:
Needs `sqlite3` installed on esh-docker-vm (`postgresql-client` was only for
the removed paperless dump; it is still installed, and harmless):
```bash
ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 postgresql-client restic'
ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 restic'
```
(restic too — not installed on this host yet.)
@@ -57,7 +66,7 @@ ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 postgresql-
### 1. Install restic + db-client tooling
```bash
ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y restic sqlite3 postgresql-client'
ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y restic sqlite3'
```
### 2. Add `esh-docker-vm` entry on ana-docker rest-server
@@ -84,16 +93,6 @@ ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/restic.env && chmod 600 /e
ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/password && chmod 600 /etc/restic/password"'
# paste: <REPO-PASSPHRASE>
# Enter, Ctrl-D
# Install DB creds (from dbcreds.env.example — edit in a temp file first)
cp configs/restic/esh-docker-vm/dbcreds.env.example /tmp/dbcreds.env
${EDITOR:-vi} /tmp/dbcreds.env # set PAPERLESS_PGPASS to real value
scp /tmp/dbcreds.env esh-docker-vm:/tmp/
ssh -t esh-docker-vm '
sudo install -o root -g root -m 0600 /tmp/dbcreds.env /etc/restic/dbcreds.env &&
rm /tmp/dbcreds.env
'
shred -u /tmp/dbcreds.env 2>/dev/null || rm -f /tmp/dbcreds.env
```
### 4. Init the repo
@@ -126,7 +125,6 @@ ssh -t esh-docker-vm 'sudo ls -lh /var/lib/restic/stage/'
```
Expected files:
- `paperless.pg_dump` — should be 1–50 MB depending on doc count
- `home-assistant.sqlite3` — ~50 MB (matches live DB size)
- `calibre-web-automated.app.db` — ~250 KB
- `pgadmin4.db` — ~200 KB
@@ -182,8 +180,11 @@ ssh -t esh-docker-vm '
## Follow-ups after restic is proven (separate session)
- **Rotate paperless-ngx Postgres password.** Currently `paperless-ng` —
trivially weak. Update the DB, the compose, and `dbcreds.env`.
- **Rotate paperless-ngx Postgres password.** Was `paperless-ng` —
trivially weak. Update the DB and the compose (no copy lives on this
backup path any more). The removed backup copy failed auth from
2026-04-24 on, so it may already have changed at the DB side; confirm
before assuming.
- **Retire offen/docker-volume-backup sidecars** on paperless-ngx + pgadmin
stacks after ~1 week of clean restic runs. Delete the compose's
sidecar services + the tarballs under `/mnt/backup/docker/esh-vm-docker/`.