esh-pfi-infrastructure/configs/restic/esh-docker-vm

restic / esh-docker-vm

Primary ESH home-lab Docker host. VM on esh-pve. Covered by vzdump at the hypervisor (3/3 on esh-pve after VM 108 retirement); this adds file-level restic with app-consistent DB dumps.

Cross-site writes to rest-server-ana at 10.250.50.70:8000/esh-docker-vm/.

What's backed up

Path	Purpose
/opt/docker	Compose + bind-mounted conf (~12 MB)
/etc	Host config
/root	Root's scripts / history / keys
/home	User home dirs
/var/lib/docker/volumes	HA / paperless / pgadmin / CWA / dockge / agents
/var/lib/restic/stage	DB dumps from pre-backup.sh

Critically excluded

• NFS mounts from 10.0.50.50 — /mnt/{backup,books,compose,documents}. Debian NAS protects these at its own layer.
• /mnt/backup/docker/esh-vm-docker/* — offen-sidecar tarballs on NFS; once restic proves itself for a week, retire those sidecars.
• Docker internals (backingFsBlockDev, metadata.db).
• Logs, pids, user caches.

Pre-backup hooks

Unique to this host: most containers don't bundle sqlite3, so pre-backup.sh runs sqlite3 from the host against the volume bind-mount paths. Simpler than building custom images for HA and pgadmin. (uptime-kuma moved to ana-docker on 2026-09-22 and its block was removed; see the header of pre-backup.sh for why a stale block took the whole backup down.)

paperless-ngx's database is NOT dumped here (removed 2026-09-23, operator decision). It lives on esh-vm-db (10.0.50.60) and is backed up at the source by that host's fail-closed pg_dumpall — see configs/restic/esh-vm-db/. The host-side pg_dump this hook used to run had failed auth every night since 2026-04-24 behind a WARN, and its > paperless.pg_dump redirect left a 0-byte file in every snapshot that looked like a dump. It was the only consumer of /etc/restic/dbcreds.env, so the creds file and its template went too. Paperless's media volumes are still captured under /var/lib/docker/volumes.

Service	DB	Approach
home-assistant	/var/lib/docker/.../homeassistant_v2.db	host sqlite3
pgadmin	/var/lib/docker/.../pgadmin4.db	host sqlite3
calibre-web-automated	/config/app.db inside container	in-container sqlite3 (it has the binary)

Host prerequisites

Needs sqlite3 installed on esh-docker-vm (postgresql-client was only for the removed paperless dump; it is still installed, and harmless):

ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 restic'

(restic too — not installed on this host yet.)

Deploy (one-time setup)

1. Install restic + db-client tooling

ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y restic sqlite3'

2. Add esh-docker-vm entry on ana-docker rest-server

ssh ana-docker '
  docker run --rm httpd:2.4-alpine htpasswd -nbB esh-docker-vm "<NEW-PW>" \
    >> /mnt/backup/restic/repo/ana/.htpasswd
'
# no sudo — /mnt/backup is NFS with root_squash; dir is lkraven-owned

3. Install restic creds on esh-docker-vm

ssh -t esh-docker-vm 'sudo install -d -o root -g root -m 0700 /etc/restic /var/lib/restic'

# REST URL
ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/restic.env && chmod 600 /etc/restic/restic.env"'
# paste: RESTIC_REPOSITORY=rest:http://esh-docker-vm:<HTPASSWD-PW>@10.250.50.70:8000/esh-docker-vm/
# Enter, Ctrl-D

# Repo passphrase
ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/password && chmod 600 /etc/restic/password"'
# paste: <REPO-PASSPHRASE>
# Enter, Ctrl-D

4. Init the repo

ssh -t esh-docker-vm '
  sudo bash -c "
    set -a; . /etc/restic/restic.env; set +a
    restic init
  "
'
# Type passphrase twice at the prompt — same as installed in /etc/restic/password.

5. Deploy pre-backup.sh

scp configs/restic/esh-docker-vm/pre-backup.sh esh-docker-vm:/tmp/pre-backup.sh
ssh -t esh-docker-vm '
  sudo install -o root -g root -m 0700 /tmp/pre-backup.sh /etc/restic/pre-backup.sh &&
  rm /tmp/pre-backup.sh
'

6. Test the hook by itself (without running the full backup)

ssh -t esh-docker-vm 'sudo /etc/restic/pre-backup.sh'
ssh -t esh-docker-vm 'sudo ls -lh /var/lib/restic/stage/'

Expected files:

• home-assistant.sqlite3 — ~50 MB (matches live DB size)
• calibre-web-automated.app.db — ~250 KB
• pgadmin4.db — ~200 KB

Any WARN: lines in the hook output indicate a block that was skipped — read them, debug one at a time.

7. Install resticprofile

ssh -t esh-docker-vm '
  curl -sfL https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh \
    | sudo sh -s -- -b /usr/local/bin &&
  /usr/local/bin/resticprofile version
'

8. Deploy profile

scp configs/restic/esh-docker-vm/profiles.yaml esh-docker-vm:/tmp/profiles.yaml
ssh -t esh-docker-vm '
  sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml &&
  rm /tmp/profiles.yaml &&
  sudo resticprofile --config /etc/restic/profiles.yaml show
'

9. Enable timers + first backup

ssh -t esh-docker-vm '
  sudo resticprofile --config /etc/restic/profiles.yaml schedule --all &&
  sudo resticprofile --config /etc/restic/profiles.yaml backup --verbose
'

Expected first snapshot: ~1–2 GB (HA DB is the dominant file, plus some Docker volumes). If you see much more, inspect /home and /var/lib/docker/volumes for surprise content.

Restoring

ssh -t esh-docker-vm '
  sudo bash -c "
    set -a; . /etc/restic/restic.env; set +a
    RESTIC_PASSWORD_FILE=/etc/restic/password \
      restic restore --target /tmp/restore latest --path /opt/docker
  "
'

Follow-ups after restic is proven (separate session)

• Rotate paperless-ngx Postgres password. Was paperless-ng — trivially weak. Update the DB and the compose (no copy lives on this backup path any more). The removed backup copy failed auth from 2026-04-24 on, so it may already have changed at the DB side; confirm before assuming.
• Retire offen/docker-volume-backup sidecars on paperless-ngx + pgadmin stacks after ~1 week of clean restic runs. Delete the compose's sidecar services + the tarballs under /mnt/backup/docker/esh-vm-docker/.

Recreating the repo

Same pattern as ana-ml2 — see configs/restic/ana-ml2/README.md. Repo path on ana-docker is /mnt/backup/restic/repo/ana/esh-docker-vm/.