Operator decision. paperless-ngx's database lives on esh-vm-db and is backed
up at the source by that host's fail-closed pg_dumpall. esh-docker-vm's
second copy had failed auth every night since 2026-04-24 behind a WARN. Its
`> paperless.pg_dump` redirect left a 0-byte file in every snapshot
(confirmed in snapshot 6ec9f74f), which looked like a dump but held nothing.
The block was the only consumer of /etc/restic/dbcreds.env, so the creds
loader is gone and the template dbcreds.env.example is deleted. The host
file was moved (not deleted) to /var/lib/restic/repair-20260923/.
Paperless's media volumes are still captured under /var/lib/docker/volumes.
Also fixes ownership. elway's sudo upload does scp-as-user then `sudo mv`,
so the hook deployed at 08:04 landed infra-ops:infra-ops even though root
executes it. Both esh-docker-vm playbooks now chown it back to root and
verify root:root 700.
Verified: the live hook hash matches canonical (e0d3ddcef1bddf43), and
the manual backup saved snapshot decfae71 with 3 staged dumps and no WARN
lines.
Uptime Kuma moved from esh-docker-vm to ana-docker on 2026-09-22. The
pre-backup hook's fallback lookup, `docker ps | grep -E "uptime.kuma"`,
then matched nothing and exited 1. Under set -euo pipefail that aborted the
hook, and resticprofile treats a failed run-before as fatal, so no snapshot
was taken from 2026-09-22 01:00 until this fix (backup-freshness: 54h stale).
The block is removed rather than guarded because there is nothing on this
host left for it to back up. The header now records the invariant the
"blocks only WARN" promise depends on: every optional-service lookup must sit
inside an `if` test or end in `|| true`. The remaining blocks were checked
and all do.
Deployed with playbooks/esh-docker-vm-restic-drop-kuma-block.yaml (the
pre-fix hook is kept in /var/lib/restic/repair-20260923/). The live hook
hash matches the canonical copy (43e6bea8b8569602). The manual backup saved
snapshot 6ec9f74f, and backup-freshness now reports all backups fresh.
Closes the last file-level backup gap. Primary ESH home-lab Docker host
— five services with state worth consistent dumps:
- paperless-ngx → external Postgres on 10.0.50.60 (host pg_dump)
- home-assistant → local SQLite ~50MB (host sqlite3 .backup)
- calibre-web-automated → local SQLite (in-container sqlite3)
- pgadmin → local SQLite (host sqlite3)
- uptime-kuma → local SQLite (host sqlite3; container name may vary
after force-recreate, detect by label)
Unique to this host: HA/pgadmin/uptime-kuma images don't bundle sqlite3.
Rather than maintaining custom images, pre-backup.sh runs sqlite3 from
the HOST against the volume bind-mount paths. Requires sqlite3 +
postgresql-client installed on esh-docker-vm.
Cross-site writes to rest-server-ana since ESH has no local rest-server.
NFS mounts (/mnt/{backup,books,compose,documents}) explicitly excluded
— hundreds of GB of NAS-side content backed up at the NAS layer. Also
excludes offen-sidecar buffer volumes (paperless + pgadmin currently
run offen/docker-volume-backup alongside — retire once restic has a
week of clean runs).
Found in audit (non-blocking but noted in README follow-ups):
- paperless-ngx Postgres password is literally "paperless-ng" —
trivially weak, rotate at next opportunity.
vh