esh-pfi-infrastructure/stacks/rest-server-ana/README.md

# rest-server-ana

Anaheim-site restic backup endpoint. Replaces the older `restic` stack on ana-docker with the same auth model as `rest-server-nh3` on the Synology, so every client host uses identical URL shapes against either endpoint.

**Server:** ana-docker (`10.250.50.70`)
**Port:** `http://10.250.50.70:8000`
**Data root:** `/mnt/backup/restic/repo/` (NFS mount on the host, served by the Debian file server at `10.250.50.50`)

## Data layout

The NFS export at `/mnt/backup/restic/repo/` is partitioned by site into three sibling subdirs:

| Path | Role |
|---|---|
| `/mnt/backup/restic/repo/ana/` | **Live data served by this rest-server.** Each Anaheim-side host writes its repo under here (e.g. `ana/ana-docker/`, `ana/ana-ml2/`). `.htpasswd` also lives here. |
| `/mnt/backup/restic/repo/esh/` | Destination for mirroring ESH-site backups to Anaheim (cross-site redundancy; not yet wired up). |
| `/mnt/backup/restic/repo/nh3/` | Destination for mirroring NH3-site backups (from the Synology `rest-server-nh3`) into Anaheim. Same — pending. |

The rest-server container is configured with `/mnt/backup/restic/repo/ana` as its data root (see compose's `DATA_DIR` env), so clients use URLs of the form `rest:http://user:pw@10.250.50.70:8000/<host>/` — the `<host>` path is relative to `ana/`.

The two mirror dirs (`esh/`, `nh3/`) aren't served by restic at all; they're rsync destinations.

Paired with:
- **`rest-server-nh3`** on the Synology (`10.100.50.50:8000`, data on Btrfs).
- A cross-site rsync job (TBD, on ana-docker) that mirrors each site's data tree to the other so either NAS can fully restore either site's hosts.

## What changed from the old `restic` stack

| | old `restic` on ana-docker | this stack |
|---|---|---|
| `--private-repos` | no | **yes** |
| `--append-only` | no | **yes** |
| `--prometheus` | no | **yes** |
| healthcheck | no | yes |
| `.env`-driven | no | yes |
| restart policy | none | `unless-stopped` |
| image version | floating `latest` | `${REST_SERVER_VERSION}` |
| stack dir on server | `/opt/docker/compose/restic/` | `/opt/docker/compose/rest-server-ana/` |

Data path is unchanged (`/mnt/backup/restic/repo/ana/`) so nothing new needs to be allocated on the NAS.

## Pre-deploy: clean the data dir and create htpasswd

Since there's nothing in the existing path we want to keep, start fresh so the on-disk layout matches `--private-repos`:

```bash
ssh ana-docker '
  # Stop the old stack so port 8000 and the data dir are free
  cd /opt/docker/compose/restic
  docker compose down

  # Wipe the old non-private-repos layout
  sudo rm -rf /mnt/backup/restic/repo/ana/*
  sudo rm -rf /mnt/backup/restic/repo/ana/.htpasswd  # if present

  # Create the htpasswd file. Use the same passwords here as on the NH3
  # Synology so each host has one credential that works at either endpoint.
  sudo touch /mnt/backup/restic/repo/ana/.htpasswd
  sudo chmod 600 /mnt/backup/restic/repo/ana/.htpasswd
'

# Generate htpasswd entries locally (one per host) and append. Using the
# `httpd:2.4-alpine` throwaway container so we do not depend on
# apache2-utils being installed on ana-docker.
for user in ana-docker ana-ml2 nh3-docker esh-docker-vm; do
  read -rs -p "password for $user (must match the NH3 Synology): " pw; echo
  docker run --rm httpd:2.4-alpine htpasswd -nbB "$user" "$pw" \
    | ssh ana-docker 'sudo tee -a /mnt/backup/restic/repo/ana/.htpasswd >/dev/null'
done
```

If you run that locally and don't have Docker here, equivalent on the server:

```bash
ssh ana-docker "docker run --rm httpd:2.4-alpine htpasswd -nbB <user> '<pw>'" \
  | ssh ana-docker 'sudo tee -a /mnt/backup/restic/repo/ana/.htpasswd >/dev/null'
```

## Deploy

Stage the new stack and push it:

```bash
# Stage the stack into the mirror (if not already done via sync-stacks.sh)
mkdir -p stacks-mirror/ana-docker/rest-server-ana
cp stacks/rest-server-ana/compose.yaml stacks/rest-server-ana/.env.example \
   stacks-mirror/ana-docker/rest-server-ana/

scripts/deploy-stack.sh ana-docker rest-server-ana
```

Confirm at the prompt. Then on the server:

```bash
ssh ana-docker '
  cd /opt/docker/compose/rest-server-ana
  cp -n .env.example .env
  docker compose config
  docker compose up -d
  docker compose logs --tail=30
'
```

## Retire the old stack

Once the new one is healthy and the first repo has initialized successfully from a client:

```bash
ssh ana-docker '
  cd /opt/docker/compose/restic
  docker compose down
  # Optionally remove the old stack dir (keep it for a release or two
  # in case you need to roll back):
  # rm -rf /opt/docker/compose/restic
'
```

## Verify

```bash
# 401 from the root — service up, auth enforced
curl -sS -o /dev/null -w 'unauth status=%{http_code}\n' \
  http://10.250.50.70:8000/

# 200 / 404 from a real user+password — auth valid, --private-repos path OK
curl -sS -o /dev/null -w 'auth   status=%{http_code}\n' \
  -u ana-docker:<password> http://10.250.50.70:8000/ana-docker/

# Init a repo from a client host (one-time per host)
ssh ana-docker '
  export RESTIC_REPOSITORY="rest:http://ana-docker:<rest-pw>@10.250.50.70:8000/ana-docker/"
  export RESTIC_PASSWORD="<client-side-encryption-passphrase>"
  restic init
'
```

## Prune ceremony

Same as `rest-server-nh3` — prune is blocked by `--append-only`. Two options, pick one per endpoint:

- **Temporary flag flip:** edit compose, remove `--append-only` from `OPTIONS`, `docker compose up -d`, run `restic forget --prune` from origin hosts, put the flag back, `docker compose up -d`. Quarterly change.
- **Second endpoint on a different port:** stand up a sibling container (e.g. port `8001`) against the same data dir without `--append-only`, reachable only from a trusted host. Everyday backups still hit `:8000`.

If you go the second-endpoint route, copy this stack to `stacks/rest-server-ana-prune/` with `REST_PORT=8001` and `--append-only` removed from the compose.

## Off-site replication

Scheduled on ana-docker (to be written). Mirror destinations live in the
sibling subdirs described under "Data layout" above:

```bash
# Pull NH3's tree into ana-docker's /mnt/backup/restic/repo/nh3/
rsync -avz --delete admin@10.100.50.50:/volume1/Backup/restic/ /mnt/backup/restic/repo/nh3/

# Pull the ESH rest-server's tree into /mnt/backup/restic/repo/esh/
# (once an ESH rest-server is set up — currently ESH hosts write directly
# to this Anaheim rest-server at ana/esh-*)

# Push our tree (ana/) to NH3 for redundancy in the other direction
rsync -avz --delete /mnt/backup/restic/repo/ana/ admin@10.100.50.50:/volume1/Backup/restic-mirror-ana/
```

Each sync unidirectional, running in the direction its data flows. Prune
runs only at the origin so the mirror shrinks correctly.