esh-pfi-infrastructure/configs/restic/README.md

# configs/restic

Per-host restic backup configs, deployed into `/etc/restic/` on each server and driven by `resticprofile` + `systemd` timers.

Writes flow to two rest-servers: the Anaheim one on ana-docker stores data to an NFS mount backed by the Debian file server at `10.250.50.50`; the NH3 one on the Synology at `10.100.50.50` stores to local Btrfs. Cross-site rsync keeps each side holding a mirror of the other.

## Layout

```
configs/restic/
├── README.md                  # this file
└── <host>/
    ├── profiles.yaml          # committed, zero secrets
    ├── pre-backup.sh          # committed, zero secrets
    └── README.md              # per-host notes (paths, containers, quirks)
```

On each server, deployed to `/etc/restic/`:

```
/etc/restic/
├── profiles.yaml              # scp'd from configs/restic/<host>/profiles.yaml
├── pre-backup.sh              # scp'd, 0755, root:root
├── password                   # 0400 root:root — client-side encryption passphrase
└── restic.env                 # 0600 root:root — RESTIC_REPOSITORY=rest:http://user:pw@host:port/path/
/var/lib/restic/
├── stage/                     # temp staging for DB dumps; owned by root, 0700
└── last-success               # unix timestamp of the last successful run
```

## Why this shape

- **No secrets in committed config.** `profiles.yaml` references `RESTIC_PASSWORD_FILE=/etc/restic/password` and loads `RESTIC_REPOSITORY` from `restic.env`. Both files live only on the host, 0400/0600 root-owned.
- **Creds go in the URL, not netrc.** restic's rest backend doesn't consult `~/.netrc` — HTTP basic-auth has to be embedded in the repository URL. Keeping that URL in an env-file (not the committed YAML) means the secret stays on the host.
- **Pre-hook runs DB dumps into a staging dir**, then `restic backup` includes that dir alongside the regular paths. One snapshot = one point-in-time.
- **`restic forget` is scheduled; `restic prune` is not.** Rest-server's `--append-only` blocks prune from the client side by design. Prune is a manual ceremony (flip the flag, run prune, flip back).

## Install restic + resticprofile on each host

```bash
# Current-enough restic. Debian 12 ships 0.14 (too old for some flags);
# Debian 13 ships 0.18. If you're on 12, grab the .deb from the upstream
# release page instead.
sudo apt install -y restic    # or install 0.18+ from github.com/restic/restic/releases

# resticprofile is not in Debian. Download the .deb from its release page.
v=$(curl -sI https://github.com/creativeprojects/resticprofile/releases/latest \
     | awk -F'/' '/^location:/{sub(/\r/,""); print $NF}')
v=${v#v}
url="https://github.com/creativeprojects/resticprofile/releases/download/v${v}/resticprofile_${v}_linux_amd64.deb"
curl -sL -o /tmp/resticprofile.deb "$url"
sudo dpkg -i /tmp/resticprofile.deb
rm /tmp/resticprofile.deb
resticprofile version
```

## Per-host deploy flow (done once per host)

```bash
HOST=ana-docker     # or ana-ml2, nh3-docker, esh-docker-vm

# 1. Create the target dir (one-time)
ssh -t "$HOST" 'sudo install -d -o root -g root -m 0755 /etc/restic'
ssh -t "$HOST" 'sudo install -d -o root -g root -m 0700 /var/lib/restic/stage'

# 2. Seed the two secret files on the host (never in this repo):
#    - the client-side encryption passphrase (the one used at `restic init`)
#    - an env-file with the full RESTIC_REPOSITORY URL including HTTP creds
ssh -t "$HOST" 'sudo install -o root -g root -m 0400 /dev/null /etc/restic/password'
ssh -t "$HOST" 'sudo install -o root -g root -m 0600 /dev/null /etc/restic/restic.env'

# Seed content (replace <…> with real values from your vault):
ssh -t "$HOST" "echo '<encryption-passphrase>' | sudo tee /etc/restic/password >/dev/null"
ssh -t "$HOST" "echo 'RESTIC_REPOSITORY=rest:http://<user>:<http-pw>@<rest-server>:8000/<user>/' | sudo tee /etc/restic/restic.env >/dev/null"

# 3. Push the committed config + pre-hook
scp "configs/restic/$HOST/profiles.yaml"  "$HOST:/tmp/profiles.yaml"
scp "configs/restic/$HOST/pre-backup.sh"  "$HOST:/tmp/pre-backup.sh"
ssh -t "$HOST" "
  sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml
  sudo install -o root -g root -m 0755 /tmp/pre-backup.sh /etc/restic/pre-backup.sh
  rm -f /tmp/profiles.yaml /tmp/pre-backup.sh
"

# 4. Test the profile before scheduling
ssh -t "$HOST" 'sudo resticprofile --config /etc/restic/profiles.yaml --name default backup --dry-run'

# 5. When dry-run looks clean, wire up systemd timers
ssh -t "$HOST" 'sudo resticprofile --config /etc/restic/profiles.yaml schedule'
#    → installs restic-backup@<profile>.{service,timer} units for backup/forget/check

# 6. Verify
ssh -t "$HOST" 'sudo systemctl list-timers | grep restic'
```

## Prune ceremony (quarterly or as needed)

Per-repo, when enough forgotten-but-still-on-disk snapshots accumulate:

```bash
# On the rest-server host (ana-docker or the Synology):
# 1. Stop the append-only rest-server, start one without the flag
#    (simplest: edit the stack's .env, remove --append-only from OPTIONS,
#    docker compose up -d)

# 2. From the client host, run prune
ssh -t <host> 'sudo resticprofile --config /etc/restic/profiles.yaml --name default prune'

# 3. Put --append-only back on the rest-server and docker compose up -d.
```

Alternatively stand up a second rest-server stack on port 8001 without `--append-only` and point prune runs at that endpoint; keep the append-only one for daily writes.

## Monitoring

Each successful run writes `/var/lib/restic/last-success` with the current unix timestamp (this is the `post-backup` hook in `profiles.yaml`). Beszel can be configured to alert when that file's mtime exceeds ~36 hours.

```bash
ssh <host> 'stat -c "%y %n" /var/lib/restic/last-success 2>/dev/null || echo no successful run yet'
```