esh-pfi-infrastructure/stacks/rest-server-ana/README.md

rest-server-ana

Anaheim-site restic backup endpoint. Replaces the older restic stack on ana-docker with the same auth model as rest-server-nh3 on the Synology, so every client host uses identical URL shapes against either endpoint.

Server: ana-docker (10.250.50.70) Port: http://10.250.50.70:8000 Data root: /mnt/backup/restic/repo/ (NFS mount on the host, served by the Debian file server at 10.250.50.50)

Data layout

The NFS export at /mnt/backup/restic/repo/ is partitioned by site into three sibling subdirs:

Path	Role
/mnt/backup/restic/repo/ana/	Live data served by this rest-server. Each Anaheim-side host writes its repo under here (e.g. ana/ana-docker/, ana/ana-ml2/). .htpasswd also lives here.
/mnt/backup/restic/repo/esh/	Destination for mirroring ESH-site backups to Anaheim (cross-site redundancy; not yet wired up).
/mnt/backup/restic/repo/nh3/	Destination for mirroring NH3-site backups (from the Synology rest-server-nh3) into Anaheim. Same — pending.

The rest-server container is configured with /mnt/backup/restic/repo/ana as its data root (see compose's DATA_DIR env), so clients use URLs of the form rest:http://user:pw@10.250.50.70:8000/<host>/ — the <host> path is relative to ana/.

The two mirror dirs (esh/, nh3/) aren't served by restic at all; they're rsync destinations.

Paired with:

• rest-server-nh3 on the Synology (10.100.50.50:8000, data on Btrfs).
• A cross-site rsync job (TBD, on ana-docker) that mirrors each site's data tree to the other so either NAS can fully restore either site's hosts.

What changed from the old restic stack

	old restic on ana-docker	this stack
--private-repos	no	yes
--append-only	no	yes
--prometheus	no	yes
healthcheck	no	yes
.env-driven	no	yes
restart policy	none	unless-stopped
image version	floating latest	${REST_SERVER_VERSION}
stack dir on server	/opt/docker/compose/restic/	/opt/docker/compose/rest-server-ana/

Data path is unchanged (/mnt/backup/restic/repo/ana/) so nothing new needs to be allocated on the NAS.

Pre-deploy: clean the data dir and create htpasswd

Since there's nothing in the existing path we want to keep, start fresh so the on-disk layout matches --private-repos:

ssh ana-docker '
  # Stop the old stack so port 8000 and the data dir are free
  cd /opt/docker/compose/restic
  docker compose down

  # Wipe the old non-private-repos layout
  sudo rm -rf /mnt/backup/restic/repo/ana/*
  sudo rm -rf /mnt/backup/restic/repo/ana/.htpasswd  # if present

  # Create the htpasswd file. Use the same passwords here as on the NH3
  # Synology so each host has one credential that works at either endpoint.
  sudo touch /mnt/backup/restic/repo/ana/.htpasswd
  sudo chmod 600 /mnt/backup/restic/repo/ana/.htpasswd
'

# Generate htpasswd entries locally (one per host) and append. Using the
# `httpd:2.4-alpine` throwaway container so we do not depend on
# apache2-utils being installed on ana-docker.
for user in ana-docker ana-ml2 nh3-docker esh-docker-vm; do
  read -rs -p "password for $user (must match the NH3 Synology): " pw; echo
  docker run --rm httpd:2.4-alpine htpasswd -nbB "$user" "$pw" \
    | ssh ana-docker 'sudo tee -a /mnt/backup/restic/repo/ana/.htpasswd >/dev/null'
done

If you run that locally and don't have Docker here, equivalent on the server:

ssh ana-docker "docker run --rm httpd:2.4-alpine htpasswd -nbB <user> '<pw>'" \
  | ssh ana-docker 'sudo tee -a /mnt/backup/restic/repo/ana/.htpasswd >/dev/null'

Deploy

Stage the new stack and push it:

# Stage the stack into the mirror (if not already done via sync-stacks.sh)
mkdir -p stacks-mirror/ana-docker/rest-server-ana
cp stacks/rest-server-ana/compose.yaml stacks/rest-server-ana/.env.example \
   stacks-mirror/ana-docker/rest-server-ana/

scripts/deploy-stack.sh ana-docker rest-server-ana

Confirm at the prompt. Then on the server:

ssh ana-docker '
  cd /opt/docker/compose/rest-server-ana
  cp -n .env.example .env
  docker compose config
  docker compose up -d
  docker compose logs --tail=30
'

Retire the old stack

Once the new one is healthy and the first repo has initialized successfully from a client:

ssh ana-docker '
  cd /opt/docker/compose/restic
  docker compose down
  # Optionally remove the old stack dir (keep it for a release or two
  # in case you need to roll back):
  # rm -rf /opt/docker/compose/restic
'

Verify

# 401 from the root — service up, auth enforced
curl -sS -o /dev/null -w 'unauth status=%{http_code}\n' \
  http://10.250.50.70:8000/

# 200 / 404 from a real user+password — auth valid, --private-repos path OK
curl -sS -o /dev/null -w 'auth   status=%{http_code}\n' \
  -u ana-docker:<password> http://10.250.50.70:8000/ana-docker/

# Init a repo from a client host (one-time per host)
ssh ana-docker '
  export RESTIC_REPOSITORY="rest:http://ana-docker:<rest-pw>@10.250.50.70:8000/ana-docker/"
  export RESTIC_PASSWORD="<client-side-encryption-passphrase>"
  restic init
'

Prune ceremony

Same as rest-server-nh3 — prune is blocked by --append-only. Two options, pick one per endpoint:

• Temporary flag flip: edit compose, remove --append-only from OPTIONS, docker compose up -d, run restic forget --prune from origin hosts, put the flag back, docker compose up -d. Quarterly change.
• Second endpoint on a different port: stand up a sibling container (e.g. port 8001) against the same data dir without --append-only, reachable only from a trusted host. Everyday backups still hit :8000.

If you go the second-endpoint route, copy this stack to stacks/rest-server-ana-prune/ with REST_PORT=8001 and --append-only removed from the compose.

Off-site replication

Scheduled on ana-docker (to be written). Mirror destinations live in the sibling subdirs described under "Data layout" above:

# Pull NH3's tree into ana-docker's /mnt/backup/restic/repo/nh3/
rsync -avz --delete admin@10.100.50.50:/volume1/Backup/restic/ /mnt/backup/restic/repo/nh3/

# Pull the ESH rest-server's tree into /mnt/backup/restic/repo/esh/
# (once an ESH rest-server is set up — currently ESH hosts write directly
# to this Anaheim rest-server at ana/esh-*)

# Push our tree (ana/) to NH3 for redundancy in the other direction
rsync -avz --delete /mnt/backup/restic/repo/ana/ admin@10.100.50.50:/volume1/Backup/restic-mirror-ana/

Each sync unidirectional, running in the direction its data flows. Prune runs only at the origin so the mirror shrinks correctly.