esh-pfi-infrastructure/stacks/rest-server-ana

rest-server-ana

Anaheim-site restic backup endpoint. Replaces the older restic stack on ana-docker with the same auth model as rest-server-nh3 on the Synology, so every client host uses identical URL shapes against either endpoint.

Server: ana-docker (10.250.50.70) Port: http://10.250.50.70:8000 Data: /mnt/backup/restic/repo/ana/ (NFS mount on the host, served by the Debian file server at 10.250.50.50)

Paired with:

• rest-server-nh3 on the Synology (10.100.50.50:8000, data on Btrfs).
• A cross-site rsync job (TBD, on ana-docker) that mirrors each site's data tree to the other so either NAS can fully restore either site's hosts.

What changed from the old restic stack

	old restic on ana-docker	this stack
--private-repos	no	yes
--append-only	no	yes
--prometheus	no	yes
healthcheck	no	yes
.env-driven	no	yes
restart policy	none	unless-stopped
image version	floating latest	${REST_SERVER_VERSION}
stack dir on server	/opt/docker/compose/restic/	/opt/docker/compose/rest-server-ana/

Data path is unchanged (/mnt/backup/restic/repo/ana/) so nothing new needs to be allocated on the NAS.

Pre-deploy: clean the data dir and create htpasswd

Since there's nothing in the existing path we want to keep, start fresh so the on-disk layout matches --private-repos:

ssh ana-docker '
  # Stop the old stack so port 8000 and the data dir are free
  cd /opt/docker/compose/restic
  docker compose down

  # Wipe the old non-private-repos layout
  sudo rm -rf /mnt/backup/restic/repo/ana/*
  sudo rm -rf /mnt/backup/restic/repo/ana/.htpasswd  # if present

  # Create the htpasswd file. Use the same passwords here as on the NH3
  # Synology so each host has one credential that works at either endpoint.
  sudo touch /mnt/backup/restic/repo/ana/.htpasswd
  sudo chmod 600 /mnt/backup/restic/repo/ana/.htpasswd
'

# Generate htpasswd entries locally (one per host) and append. Using the
# `httpd:2.4-alpine` throwaway container so we do not depend on
# apache2-utils being installed on ana-docker.
for user in ana-docker ana-ml2 nh3-docker esh-docker-vm; do
  read -rs -p "password for $user (must match the NH3 Synology): " pw; echo
  docker run --rm httpd:2.4-alpine htpasswd -nbB "$user" "$pw" \
    | ssh ana-docker 'sudo tee -a /mnt/backup/restic/repo/ana/.htpasswd >/dev/null'
done

If you run that locally and don't have Docker here, equivalent on the server:

ssh ana-docker "docker run --rm httpd:2.4-alpine htpasswd -nbB <user> '<pw>'" \
  | ssh ana-docker 'sudo tee -a /mnt/backup/restic/repo/ana/.htpasswd >/dev/null'

Deploy

Stage the new stack and push it:

# Stage the stack into the mirror (if not already done via sync-stacks.sh)
mkdir -p stacks-mirror/ana-docker/rest-server-ana
cp stacks/rest-server-ana/compose.yaml stacks/rest-server-ana/.env.example \
   stacks-mirror/ana-docker/rest-server-ana/

scripts/deploy-stack.sh ana-docker rest-server-ana

Confirm at the prompt. Then on the server:

ssh ana-docker '
  cd /opt/docker/compose/rest-server-ana
  cp -n .env.example .env
  docker compose config
  docker compose up -d
  docker compose logs --tail=30
'

Retire the old stack

Once the new one is healthy and the first repo has initialized successfully from a client:

ssh ana-docker '
  cd /opt/docker/compose/restic
  docker compose down
  # Optionally remove the old stack dir (keep it for a release or two
  # in case you need to roll back):
  # rm -rf /opt/docker/compose/restic
'

Verify

# 401 from the root — service up, auth enforced
curl -sS -o /dev/null -w 'unauth status=%{http_code}\n' \
  http://10.250.50.70:8000/

# 200 / 404 from a real user+password — auth valid, --private-repos path OK
curl -sS -o /dev/null -w 'auth   status=%{http_code}\n' \
  -u ana-docker:<password> http://10.250.50.70:8000/ana-docker/

# Init a repo from a client host (one-time per host)
ssh ana-docker '
  export RESTIC_REPOSITORY="rest:http://ana-docker:<rest-pw>@10.250.50.70:8000/ana-docker/"
  export RESTIC_PASSWORD="<client-side-encryption-passphrase>"
  restic init
'

Prune ceremony

Same as rest-server-nh3 — prune is blocked by --append-only. Two options, pick one per endpoint:

• Temporary flag flip: edit compose, remove --append-only from OPTIONS, docker compose up -d, run restic forget --prune from origin hosts, put the flag back, docker compose up -d. Quarterly change.
• Second endpoint on a different port: stand up a sibling container (e.g. port 8001) against the same data dir without --append-only, reachable only from a trusted host. Everyday backups still hit :8000.

If you go the second-endpoint route, copy this stack to stacks/rest-server-ana-prune/ with REST_PORT=8001 and --append-only removed from the compose.

Off-site replication

Scheduled on ana-docker (to be written):

# Pull NH3's tree down to this side
rsync -avz --delete admin@10.100.50.50:/volume1/Backup/restic/ /mnt/backup/restic-mirror-nh3/
# Push our tree to NH3
rsync -avz --delete /mnt/backup/restic/repo/ana/ admin@10.100.50.50:/volume1/Backup/restic-mirror-ana/

Two unidirectional syncs, each running in the direction its data flows. Prune runs only at the origin so the mirror shrinks correctly.