esh-pfi-infrastructure/configs/restic/esh-docker-vm/README.md

# restic / esh-docker-vm

Primary ESH home-lab Docker host. VM on `esh-pve`. Covered by vzdump at
the hypervisor (3/3 on esh-pve after VM 108 retirement); this adds
file-level restic with app-consistent DB dumps.

Cross-site writes to `rest-server-ana` at `10.250.50.70:8000/esh-docker-vm/`.

## What's backed up

| Path | Purpose |
|---|---|
| `/opt/docker` | Compose + bind-mounted conf (~12 MB) |
| `/etc` | Host config |
| `/root` | Root's scripts / history / keys |
| `/home` | User home dirs |
| `/var/lib/docker/volumes` | HA / paperless / pgadmin / CWA / dockge / agents |
| `/var/lib/restic/stage` | DB dumps from pre-backup.sh |

## Critically excluded

- **NFS mounts from `10.0.50.50`** — `/mnt/{backup,books,compose,documents}`.
  Debian NAS protects these at its own layer.
- **`/mnt/backup/docker/esh-vm-docker/*`** — offen-sidecar tarballs on
  NFS; once restic proves itself for a week, retire those sidecars.
- Docker internals (`backingFsBlockDev`, `metadata.db`).
- Logs, pids, user caches.

## Pre-backup hooks

Unique to this host: most containers don't bundle sqlite3, so
`pre-backup.sh` runs sqlite3 **from the host** against the volume
bind-mount paths. Simpler than building custom images for HA
and pgadmin. (uptime-kuma moved to ana-docker on 2026-09-22 and its block
was removed; see the header of `pre-backup.sh` for why a stale block took
the whole backup down.)

**paperless-ngx's database is NOT dumped here** (removed 2026-09-23, operator
decision). It lives on esh-vm-db (`10.0.50.60`) and is backed up at the source
by that host's fail-closed `pg_dumpall` — see `configs/restic/esh-vm-db/`.
The host-side `pg_dump` this hook used to run had failed auth every night
since 2026-04-24 behind a WARN, and its `> paperless.pg_dump` redirect left a
**0-byte file in every snapshot** that looked like a dump. It was the only consumer of
`/etc/restic/dbcreds.env`, so the creds file and its template went too.
Paperless's *media* volumes are still captured under `/var/lib/docker/volumes`.

| Service | DB | Approach |
|---|---|---|
| home-assistant | `/var/lib/docker/.../homeassistant_v2.db` | host sqlite3 |
| pgadmin | `/var/lib/docker/.../pgadmin4.db` | host sqlite3 |
| calibre-web-automated | `/config/app.db` inside container | in-container sqlite3 (it has the binary) |

## Host prerequisites

Needs `sqlite3` installed on esh-docker-vm (`postgresql-client` was only for
the removed paperless dump; it is still installed, and harmless):

```bash
ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 restic'
```

(restic too — not installed on this host yet.)

## Deploy (one-time setup)

### 1. Install restic + db-client tooling

```bash
ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y restic sqlite3'
```

### 2. Add `esh-docker-vm` entry on ana-docker rest-server

```bash
ssh ana-docker '
  docker run --rm httpd:2.4-alpine htpasswd -nbB esh-docker-vm "<NEW-PW>" \
    >> /mnt/backup/restic/repo/ana/.htpasswd
'
# no sudo — /mnt/backup is NFS with root_squash; dir is lkraven-owned
```

### 3. Install restic creds on esh-docker-vm

```bash
ssh -t esh-docker-vm 'sudo install -d -o root -g root -m 0700 /etc/restic /var/lib/restic'

# REST URL
ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/restic.env && chmod 600 /etc/restic/restic.env"'
# paste: RESTIC_REPOSITORY=rest:http://esh-docker-vm:<HTPASSWD-PW>@10.250.50.70:8000/esh-docker-vm/
# Enter, Ctrl-D

# Repo passphrase
ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/password && chmod 600 /etc/restic/password"'
# paste: <REPO-PASSPHRASE>
# Enter, Ctrl-D
```

### 4. Init the repo

```bash
ssh -t esh-docker-vm '
  sudo bash -c "
    set -a; . /etc/restic/restic.env; set +a
    restic init
  "
'
# Type passphrase twice at the prompt — same as installed in /etc/restic/password.
```

### 5. Deploy pre-backup.sh

```bash
scp configs/restic/esh-docker-vm/pre-backup.sh esh-docker-vm:/tmp/pre-backup.sh
ssh -t esh-docker-vm '
  sudo install -o root -g root -m 0700 /tmp/pre-backup.sh /etc/restic/pre-backup.sh &&
  rm /tmp/pre-backup.sh
'
```

### 6. Test the hook by itself (without running the full backup)

```bash
ssh -t esh-docker-vm 'sudo /etc/restic/pre-backup.sh'
ssh -t esh-docker-vm 'sudo ls -lh /var/lib/restic/stage/'
```

Expected files:
- `home-assistant.sqlite3` — ~50 MB (matches live DB size)
- `calibre-web-automated.app.db` — ~250 KB
- `pgadmin4.db` — ~200 KB

Any `WARN:` lines in the hook output indicate a block that was skipped —
read them, debug one at a time.

### 7. Install resticprofile

```bash
ssh -t esh-docker-vm '
  curl -sfL https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh \
    | sudo sh -s -- -b /usr/local/bin &&
  /usr/local/bin/resticprofile version
'
```

### 8. Deploy profile

```bash
scp configs/restic/esh-docker-vm/profiles.yaml esh-docker-vm:/tmp/profiles.yaml
ssh -t esh-docker-vm '
  sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml &&
  rm /tmp/profiles.yaml &&
  sudo resticprofile --config /etc/restic/profiles.yaml show
'
```

### 9. Enable timers + first backup

```bash
ssh -t esh-docker-vm '
  sudo resticprofile --config /etc/restic/profiles.yaml schedule --all &&
  sudo resticprofile --config /etc/restic/profiles.yaml backup --verbose
'
```

Expected first snapshot: **~1–2 GB** (HA DB is the dominant file, plus
some Docker volumes). If you see much more, inspect `/home` and `/var/lib/docker/volumes`
for surprise content.

## Restoring

```bash
ssh -t esh-docker-vm '
  sudo bash -c "
    set -a; . /etc/restic/restic.env; set +a
    RESTIC_PASSWORD_FILE=/etc/restic/password \
      restic restore --target /tmp/restore latest --path /opt/docker
  "
'
```

## Follow-ups after restic is proven (separate session)

- **Rotate paperless-ngx Postgres password.** Was `paperless-ng` —
  trivially weak. Update the DB and the compose (no copy lives on this
  backup path any more). The removed backup copy failed auth from
  2026-04-24 on, so it may already have changed at the DB side; confirm
  before assuming.
- **Retire offen/docker-volume-backup sidecars** on paperless-ngx + pgadmin
  stacks after ~1 week of clean restic runs. Delete the compose's
  sidecar services + the tarballs under `/mnt/backup/docker/esh-vm-docker/`.

## Recreating the repo

Same pattern as ana-ml2 — see `configs/restic/ana-ml2/README.md`. Repo
path on ana-docker is `/mnt/backup/restic/repo/ana/esh-docker-vm/`.