fix(restic/esh-docker-vm): stop dumping paperless's Postgres from this host
Operator decision. paperless-ngx's database lives on esh-vm-db and is backed up at the source by that host's fail-closed pg_dumpall. esh-docker-vm's second copy had failed auth every night since 2026-04-24 behind a WARN. Its `> paperless.pg_dump` redirect left a 0-byte file in every snapshot (confirmed in snapshot 6ec9f74f), which looked like a dump but held nothing. The block was the only consumer of /etc/restic/dbcreds.env, so the creds loader is gone and the template dbcreds.env.example is deleted. The host file was moved (not deleted) to /var/lib/restic/repair-20260923/. Paperless's media volumes are still captured under /var/lib/docker/volumes. Also fixes ownership. elway's sudo upload does scp-as-user then `sudo mv`, so the hook deployed at 08:04 landed infra-ops:infra-ops even though root executes it. Both esh-docker-vm playbooks now chown it back to root and verify root:root 700. Verified: the live hook hash matches canonical (e0d3ddcef1bddf43), and the manual backup saved snapshot decfae71 with 3 staged dumps and no WARN lines.
This commit is contained in:
@@ -29,25 +29,34 @@ Cross-site writes to `rest-server-ana` at `10.250.50.70:8000/esh-docker-vm/`.
|
||||
## Pre-backup hooks
|
||||
|
||||
Unique to this host: most containers don't bundle sqlite3, so
|
||||
`pre-backup.sh` runs sqlite3 and pg_dump **from the host** against the
|
||||
volume bind-mount paths. Simpler than building custom images for HA
|
||||
`pre-backup.sh` runs sqlite3 **from the host** against the volume
|
||||
bind-mount paths. Simpler than building custom images for HA
|
||||
and pgadmin. (uptime-kuma moved to ana-docker on 2026-09-22 and its block
|
||||
was removed; see the header of `pre-backup.sh` for why a stale block took
|
||||
the whole backup down.)
|
||||
|
||||
**paperless-ngx's database is NOT dumped here** (removed 2026-09-23, operator
|
||||
decision). It lives on esh-vm-db (`10.0.50.60`) and is backed up at the source
|
||||
by that host's fail-closed `pg_dumpall` — see `configs/restic/esh-vm-db/`.
|
||||
The host-side `pg_dump` this hook used to run had failed auth every night
|
||||
since 2026-04-24 behind a WARN, and its `> paperless.pg_dump` redirect left a
|
||||
**0-byte file in every snapshot** that looked like a dump. It was the only consumer of
|
||||
`/etc/restic/dbcreds.env`, so the creds file and its template went too.
|
||||
Paperless's *media* volumes are still captured under `/var/lib/docker/volumes`.
|
||||
|
||||
| Service | DB | Approach |
|
||||
|---|---|---|
|
||||
| paperless-ngx | external Postgres `10.0.50.60` / `paperless-ng` | host pg_dump |
|
||||
| home-assistant | `/var/lib/docker/.../homeassistant_v2.db` | host sqlite3 |
|
||||
| pgadmin | `/var/lib/docker/.../pgadmin4.db` | host sqlite3 |
|
||||
| calibre-web-automated | `/config/app.db` inside container | in-container sqlite3 (it has the binary) |
|
||||
|
||||
## Host prerequisites
|
||||
|
||||
Needs `sqlite3` and `postgresql-client` installed on esh-docker-vm:
|
||||
Needs `sqlite3` installed on esh-docker-vm (`postgresql-client` was only for
|
||||
the removed paperless dump; it is still installed, and harmless):
|
||||
|
||||
```bash
|
||||
ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 postgresql-client restic'
|
||||
ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 restic'
|
||||
```
|
||||
|
||||
(restic too — not installed on this host yet.)
|
||||
@@ -57,7 +66,7 @@ ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 postgresql-
|
||||
### 1. Install restic + db-client tooling
|
||||
|
||||
```bash
|
||||
ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y restic sqlite3 postgresql-client'
|
||||
ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y restic sqlite3'
|
||||
```
|
||||
|
||||
### 2. Add `esh-docker-vm` entry on ana-docker rest-server
|
||||
@@ -84,16 +93,6 @@ ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/restic.env && chmod 600 /e
|
||||
ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/password && chmod 600 /etc/restic/password"'
|
||||
# paste: <REPO-PASSPHRASE>
|
||||
# Enter, Ctrl-D
|
||||
|
||||
# Install DB creds (from dbcreds.env.example — edit in a temp file first)
|
||||
cp configs/restic/esh-docker-vm/dbcreds.env.example /tmp/dbcreds.env
|
||||
${EDITOR:-vi} /tmp/dbcreds.env # set PAPERLESS_PGPASS to real value
|
||||
scp /tmp/dbcreds.env esh-docker-vm:/tmp/
|
||||
ssh -t esh-docker-vm '
|
||||
sudo install -o root -g root -m 0600 /tmp/dbcreds.env /etc/restic/dbcreds.env &&
|
||||
rm /tmp/dbcreds.env
|
||||
'
|
||||
shred -u /tmp/dbcreds.env 2>/dev/null || rm -f /tmp/dbcreds.env
|
||||
```
|
||||
|
||||
### 4. Init the repo
|
||||
@@ -126,7 +125,6 @@ ssh -t esh-docker-vm 'sudo ls -lh /var/lib/restic/stage/'
|
||||
```
|
||||
|
||||
Expected files:
|
||||
- `paperless.pg_dump` — should be 1–50 MB depending on doc count
|
||||
- `home-assistant.sqlite3` — ~50 MB (matches live DB size)
|
||||
- `calibre-web-automated.app.db` — ~250 KB
|
||||
- `pgadmin4.db` — ~200 KB
|
||||
@@ -182,8 +180,11 @@ ssh -t esh-docker-vm '
|
||||
|
||||
## Follow-ups after restic is proven (separate session)
|
||||
|
||||
- **Rotate paperless-ngx Postgres password.** Currently `paperless-ng` —
|
||||
trivially weak. Update the DB, the compose, and `dbcreds.env`.
|
||||
- **Rotate paperless-ngx Postgres password.** Was `paperless-ng` —
|
||||
trivially weak. Update the DB and the compose (no copy lives on this
|
||||
backup path any more). The removed backup copy failed auth from
|
||||
2026-04-24 on, so it may already have changed at the DB side; confirm
|
||||
before assuming.
|
||||
- **Retire offen/docker-volume-backup sidecars** on paperless-ngx + pgadmin
|
||||
stacks after ~1 week of clean restic runs. Delete the compose's
|
||||
sidecar services + the tarballs under `/mnt/backup/docker/esh-vm-docker/`.
|
||||
|
||||
Reference in New Issue
Block a user