vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

restic/esh-docker-vm: profile + DB hooks + deploy guide

Browse Source 
Closes the last file-level backup gap. Primary ESH home-lab Docker host
— five services with state worth consistent dumps:

  - paperless-ngx → external Postgres on 10.0.50.60 (host pg_dump)
  - home-assistant → local SQLite ~50MB (host sqlite3 .backup)
  - calibre-web-automated → local SQLite (in-container sqlite3)
  - pgadmin → local SQLite (host sqlite3)
  - uptime-kuma → local SQLite (host sqlite3; container name may vary
    after force-recreate, detect by label)

Unique to this host: HA/pgadmin/uptime-kuma images don't bundle sqlite3.
Rather than maintaining custom images, pre-backup.sh runs sqlite3 from
the HOST against the volume bind-mount paths. Requires sqlite3 +
postgresql-client installed on esh-docker-vm.

Cross-site writes to rest-server-ana since ESH has no local rest-server.

NFS mounts (/mnt/{backup,books,compose,documents}) explicitly excluded
— hundreds of GB of NAS-side content backed up at the NAS layer. Also
excludes offen-sidecar buffer volumes (paperless + pgadmin currently
run offen/docker-volume-backup alongside — retire once restic has a
week of clean runs).

Found in audit (non-blocking but noted in README follow-ups):
  - paperless-ngx Postgres password is literally "paperless-ng" —
    trivially weak, rotate at next opportunity.
This commit is contained in:
vh
2026-04-21 11:04:37 -07:00
parent 8c70372cc4
commit 1e24e0f799
4 changed files with 438 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
configs/restic/esh-docker-vm/README.md
+194
View File
@@ -0,0 +1,194 @@
# restic / esh-docker-vm
Primary ESH home-lab Docker host. VM on `esh-pve`. Covered by vzdump at
the hypervisor (3/3 on esh-pve after VM 108 retirement); this adds
file-level restic with app-consistent DB dumps.
Cross-site writes to `rest-server-ana` at `10.250.50.70:8000/esh-docker-vm/`.
## What's backed up
| Path | Purpose |
|---|---|
| `/opt/docker` | Compose + bind-mounted conf (~12 MB) |
| `/etc` | Host config |
| `/root` | Root's scripts / history / keys |
| `/home` | User home dirs |
| `/var/lib/docker/volumes` | HA / paperless / pgadmin / CWA / dockge / agents |
| `/var/lib/restic/stage` | DB dumps from pre-backup.sh |
## Critically excluded
- **NFS mounts from `10.0.50.50`** — `/mnt/{backup,books,compose,documents}`.
Debian NAS protects these at its own layer.
- **`/mnt/backup/docker/esh-vm-docker/*`** — offen-sidecar tarballs on
NFS; once restic proves itself for a week, retire those sidecars.
- Docker internals (`backingFsBlockDev`, `metadata.db`).
- Logs, pids, user caches.
## Pre-backup hooks
Unique to this host: most containers don't bundle sqlite3, so
`pre-backup.sh` runs sqlite3 and pg_dump **from the host** against the
volume bind-mount paths. Simpler than building custom images for HA,
pgadmin, and uptime-kuma.
| Service | DB | Approach |
|---|---|---|
| paperless-ngx | external Postgres `10.0.50.60` / `paperless-ng` | host pg_dump |
| home-assistant | `/var/lib/docker/.../homeassistant_v2.db` | host sqlite3 |
| pgadmin | `/var/lib/docker/.../pgadmin4.db` | host sqlite3 |
| uptime-kuma | `/var/lib/docker/.../kuma.db` | host sqlite3 |
| calibre-web-automated | `/config/app.db` inside container | in-container sqlite3 (it has the binary) |
## Host prerequisites
Needs `sqlite3` and `postgresql-client` installed on esh-docker-vm:
```bash
ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 postgresql-client restic'
```
(restic too — not installed on this host yet.)
## Deploy (one-time setup)
### 1. Install restic + db-client tooling
```bash
ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y restic sqlite3 postgresql-client'
```
### 2. Add `esh-docker-vm` entry on ana-docker rest-server
```bash
ssh ana-docker '
docker run --rm httpd:2.4-alpine htpasswd -nbB esh-docker-vm "<NEW-PW>" \
>> /mnt/backup/restic/repo/ana/.htpasswd
'
# no sudo — /mnt/backup is NFS with root_squash; dir is lkraven-owned
```
### 3. Install restic creds on esh-docker-vm
```bash
ssh -t esh-docker-vm 'sudo install -d -o root -g root -m 0700 /etc/restic /var/lib/restic'
# REST URL
ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/restic.env && chmod 600 /etc/restic/restic.env"'
# paste: RESTIC_REPOSITORY=rest:http://esh-docker-vm:<HTPASSWD-PW>@10.250.50.70:8000/esh-docker-vm/
# Enter, Ctrl-D
# Repo passphrase
ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/password && chmod 600 /etc/restic/password"'
# paste: <REPO-PASSPHRASE>
# Enter, Ctrl-D
# Install DB creds (from dbcreds.env.example — edit in a temp file first)
cp configs/restic/esh-docker-vm/dbcreds.env.example /tmp/dbcreds.env
${EDITOR:-vi} /tmp/dbcreds.env # set PAPERLESS_PGPASS to real value
scp /tmp/dbcreds.env esh-docker-vm:/tmp/
ssh -t esh-docker-vm '
sudo install -o root -g root -m 0600 /tmp/dbcreds.env /etc/restic/dbcreds.env &&
rm /tmp/dbcreds.env
'
shred -u /tmp/dbcreds.env 2>/dev/null || rm -f /tmp/dbcreds.env
```
### 4. Init the repo
```bash
ssh -t esh-docker-vm '
sudo bash -c "
set -a; . /etc/restic/restic.env; set +a
restic init
"
'
# Type passphrase twice at the prompt — same as installed in /etc/restic/password.
```
### 5. Deploy pre-backup.sh
```bash
scp configs/restic/esh-docker-vm/pre-backup.sh esh-docker-vm:/tmp/pre-backup.sh
ssh -t esh-docker-vm '
sudo install -o root -g root -m 0700 /tmp/pre-backup.sh /etc/restic/pre-backup.sh &&
rm /tmp/pre-backup.sh
'
```
### 6. Test the hook by itself (without running the full backup)
```bash
ssh -t esh-docker-vm 'sudo /etc/restic/pre-backup.sh'
ssh -t esh-docker-vm 'sudo ls -lh /var/lib/restic/stage/'
```
Expected files:
- `paperless.pg_dump` — should be 1–50 MB depending on doc count
- `home-assistant.sqlite3` — ~50 MB (matches live DB size)
- `calibre-web-automated.app.db` — ~250 KB
- `pgadmin4.db` — ~200 KB
- `uptime-kuma.kuma.db` — varies (history retention)
Any `WARN:` lines in the hook output indicate a block that was skipped —
read them, debug one at a time.
### 7. Install resticprofile
```bash
ssh -t esh-docker-vm '
curl -sfL https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh \
| sudo sh -s -- -b /usr/local/bin &&
/usr/local/bin/resticprofile version
'
```
### 8. Deploy profile
```bash
scp configs/restic/esh-docker-vm/profiles.yaml esh-docker-vm:/tmp/profiles.yaml
ssh -t esh-docker-vm '
sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml &&
rm /tmp/profiles.yaml &&
sudo resticprofile --config /etc/restic/profiles.yaml show
'
```
### 9. Enable timers + first backup
```bash
ssh -t esh-docker-vm '
sudo resticprofile --config /etc/restic/profiles.yaml schedule --all &&
sudo resticprofile --config /etc/restic/profiles.yaml backup --verbose
'
```
Expected first snapshot: **~1–2 GB** (HA DB is the dominant file, plus
some Docker volumes). If you see much more, inspect `/home` and `/var/lib/docker/volumes`
for surprise content.
## Restoring
```bash
ssh -t esh-docker-vm '
sudo bash -c "
set -a; . /etc/restic/restic.env; set +a
RESTIC_PASSWORD_FILE=/etc/restic/password \
restic restore --target /tmp/restore latest --path /opt/docker
"
'
```
## Follow-ups after restic is proven (separate session)
- **Rotate paperless-ngx Postgres password.** Currently `paperless-ng` —
trivially weak. Update the DB, the compose, and `dbcreds.env`.
- **Retire offen/docker-volume-backup sidecars** on paperless-ngx + pgadmin
stacks after ~1 week of clean restic runs. Delete the compose's
sidecar services + the tarballs under `/mnt/backup/docker/esh-vm-docker/`.
## Recreating the repo
Same pattern as ana-ml2 — see `configs/restic/ana-ml2/README.md`. Repo
path on ana-docker is `/mnt/backup/restic/repo/ana/esh-docker-vm/`.