esh-pfi-infrastructure/stacks/rest-server-nh3/README.md

# rest-server-nh3

NH3-site restic backup endpoint. Runs in Synology Container Manager on `10.100.50.50` and stores pack files on a Btrfs share so Synology snapshots protect against local corruption.

**Server:** Synology RS2418+ at `10.100.50.50`
**Port:** `http://10.100.50.50:8000` (configurable via `.env`)
**Data:** `/volume1/Backup/restic/` (configurable)

Paired with `rest-server-ana` on **ana-docker** (`http://10.250.50.70:8000`, data on an NFS mount backed by the Debian file server at `10.250.50.50`) as the Anaheim-side endpoint. Each fleet host backs up to the rest-server closest to it; an rsync job on ana-docker mirrors the two trees against each other for off-site redundancy.

## Auth model

`--private-repos` + `--append-only`, enforced via htpasswd:

- One HTTP basic-auth user **per host** (`ana-docker`, `ana-ml2`, `nh3-docker`, `esh-docker-vm`).
- Each user can only write under `/<username>/…` — a compromised host can't see or delete another host's data.
- Append-only means a compromised client can add to its own repo but can't rewrite or delete existing packs, so ransomware on a backed-up host doesn't destroy history.
- **Trade-off:** `restic forget --prune` can't run against an append-only endpoint. Prune ceremony described at the bottom of this file.

## Pre-deploy: create the data path and htpasswd

On the Synology (SSH in as an admin-capable user, or DSM *File Station*):

```bash
# 1. Create the restic data share on a Btrfs volume
ssh admin@10.100.50.50 'sudo mkdir -p /volume1/Backup/restic && \
                       sudo chown 1000:1000 /volume1/Backup/restic && \
                       sudo chmod 700 /volume1/Backup/restic'

# 2. Generate htpasswd entries. The Synology doesn't ship apache2-utils,
#    so use a throwaway container:
ssh admin@10.100.50.50 'cd /volume1/Backup/restic && \
  sudo touch .htpasswd && sudo chown 1000:1000 .htpasswd && sudo chmod 600 .htpasswd'

for user in ana-docker ana-ml2 nh3-docker esh-docker-vm; do
  read -s -p "password for $user: " pw; echo
  ssh admin@10.100.50.50 \
    "docker run --rm httpd:2.4-alpine htpasswd -nbB $user '$pw'" \
    | ssh admin@10.100.50.50 "sudo tee -a /volume1/Backup/restic/.htpasswd >/dev/null"
done
```

Record every password in your off-host password manager (1Password / Vaultwarden etc.) — you'll paste them into Backrest and into the systemd timer configs later.

## Deploy

In Synology **Container Manager**:

1. *Project* → **Create** → Name `rest-server`, Path `/volume1/docker/compose/rest-server/`.
2. Copy `compose.yaml` into the project path; copy `.env.example` → `.env` and edit if needed (default `REST_PORT=8000` and `DATA_DIR=/volume1/Backup/restic` should be fine).
3. Start the project.

CLI equivalent (if you have SSH + a shell account that can run Docker on the NAS):

```bash
ssh admin@10.100.50.50
sudo mkdir -p /volume1/docker/compose/rest-server
sudo chown $USER /volume1/docker/compose/rest-server
cd /volume1/docker/compose/rest-server

# scp the files from this workspace, then:
cp .env.example .env
docker compose config
docker compose up -d
docker compose logs -f
```

## Verify

From this workstation:

```bash
# Should return "200 OK" or redirect to /metrics; anything 5xx is a problem.
curl -u ana-docker:<password> -sv http://10.100.50.50:8000/ana-docker/ -o /dev/null

# Once restic is wired up, init the repo (one-time, per host):
RESTIC_REPOSITORY='rest:http://ana-docker:<password>@10.100.50.50:8000/ana-docker/' \
RESTIC_PASSWORD='<client-side-encryption-passphrase>' \
  restic init
```

Restic URI shape for each host (paste into Backrest when adding the repo):

```
rest:http://<user>:<pass>@10.100.50.50:8000/<user>/
```

## Prune ceremony (because of --append-only)

Because `--append-only` blocks deletes, `restic forget --prune` will fail against the live endpoint. Two options, pick one and stick with it:

### Option A — temporary flag flip (simplest, requires a maintenance window)

1. On the Synology, edit the stack's `.env` and set `EXTRA_OPTIONS=--no-auth` — **only kidding, don't.** Set `EXTRA_OPTIONS= ` and comment out `--append-only` in the compose `OPTIONS=` line (or parameterize if you prefer).
2. `docker compose up -d` to restart with deletes allowed.
3. Run `restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-yearly 3 --prune` from the origin host.
4. Restore `--append-only` and `docker compose up -d`.

Treat this as a quarterly change, not a cron job. Schedule it so you're present if restic hits anything weird.

### Option B — second endpoint on a different port (automation-friendly)

Stand up a second `rest-server` container against the same `DATA_DIR` without `--append-only`, listening on e.g. `8001`, reachable only from within the Synology / over VPN. A scheduled prune job hits that endpoint; day-to-day backup traffic continues to hit `:8000` in append-only mode.

If you end up wanting this, copy this stack to `stacks/rest-server-nh3-prune/` with `REST_PORT=8001` and `--append-only` removed.

## Off-site replication

Scheduled on ana-docker:

```bash
# Example — not the final script, just illustrating the shape.
rsync -avz --delete \
  admin@10.100.50.50:/volume1/Backup/restic/ \
  /mnt/backup/restic-mirror-nh3/
```

`rsync` is safe because restic packs are immutable once written — nothing under `/volume1/Backup/restic/<user>/data/` gets rewritten, only added or (during prune) removed. A raw `rsync --delete` with prune running only on the origin side is enough; no filesystem-level locks required.

## What doesn't live here

- No `.env` in the committed copy — only `.env.example`.
- `.htpasswd` is never checked in, never synced via `sync-stacks.sh` (its `*.ht*` isn't in the global exclude but the data dir is outside the stack path).
- Client-side restic passwords (the encryption passphrase for each repo) are separate from the HTTP auth passwords and never stored on the Synology.