esh-pfi-infrastructure/stacks/rest-server-nh3

rest-server-nh3

NH3-site restic backup endpoint. Runs in Synology Container Manager on 10.100.50.50 and stores pack files on a Btrfs share so Synology snapshots protect against local corruption.

Server: Synology RS2418+ at 10.100.50.50 Port: http://10.100.50.50:8000 (configurable via .env) Data: /volume1/Backup/restic/ (configurable)

Paired with rest-server-ana on ana-docker (http://10.250.50.70:8000, data on an NFS mount backed by the Debian file server at 10.250.50.50) as the Anaheim-side endpoint. Each fleet host backs up to the rest-server closest to it; an rsync job on ana-docker mirrors the two trees against each other for off-site redundancy.

Auth model

--private-repos + --append-only, enforced via htpasswd:

• One HTTP basic-auth user per host (ana-docker, ana-ml2, nh3-docker, esh-docker-vm).
• Each user can only write under /<username>/… — a compromised host can't see or delete another host's data.
• Append-only means a compromised client can add to its own repo but can't rewrite or delete existing packs, so ransomware on a backed-up host doesn't destroy history.
• Trade-off: restic forget --prune can't run against an append-only endpoint. Prune ceremony described at the bottom of this file.

Pre-deploy: create the data path and htpasswd

On the Synology (SSH in as an admin-capable user, or DSM File Station):

# 1. Create the restic data share on a Btrfs volume
ssh admin@10.100.50.50 'sudo mkdir -p /volume1/Backup/restic && \
                       sudo chown 1000:1000 /volume1/Backup/restic && \
                       sudo chmod 700 /volume1/Backup/restic'

# 2. Generate htpasswd entries. The Synology doesn't ship apache2-utils,
#    so use a throwaway container:
ssh admin@10.100.50.50 'cd /volume1/Backup/restic && \
  sudo touch .htpasswd && sudo chown 1000:1000 .htpasswd && sudo chmod 600 .htpasswd'

for user in ana-docker ana-ml2 nh3-docker esh-docker-vm; do
  read -s -p "password for $user: " pw; echo
  ssh admin@10.100.50.50 \
    "docker run --rm httpd:2.4-alpine htpasswd -nbB $user '$pw'" \
    | ssh admin@10.100.50.50 "sudo tee -a /volume1/Backup/restic/.htpasswd >/dev/null"
done

Record every password in your off-host password manager (1Password / Vaultwarden etc.) — you'll paste them into Backrest and into the systemd timer configs later.

Deploy

In Synology Container Manager:

1. Project → Create → Name rest-server, Path /volume1/docker/compose/rest-server/.
2. Copy compose.yaml into the project path; copy .env.example → .env and edit if needed (default REST_PORT=8000 and DATA_DIR=/volume1/Backup/restic should be fine).
3. Start the project.

CLI equivalent (if you have SSH + a shell account that can run Docker on the NAS):

ssh admin@10.100.50.50
sudo mkdir -p /volume1/docker/compose/rest-server
sudo chown $USER /volume1/docker/compose/rest-server
cd /volume1/docker/compose/rest-server

# scp the files from this workspace, then:
cp .env.example .env
docker compose config
docker compose up -d
docker compose logs -f

Verify

From this workstation:

# Should return "200 OK" or redirect to /metrics; anything 5xx is a problem.
curl -u ana-docker:<password> -sv http://10.100.50.50:8000/ana-docker/ -o /dev/null

# Once restic is wired up, init the repo (one-time, per host):
RESTIC_REPOSITORY='rest:http://ana-docker:<password>@10.100.50.50:8000/ana-docker/' \
RESTIC_PASSWORD='<client-side-encryption-passphrase>' \
  restic init

Restic URI shape for each host (paste into Backrest when adding the repo):

rest:http://<user>:<pass>@10.100.50.50:8000/<user>/

Prune ceremony (because of --append-only)

Because --append-only blocks deletes, restic forget --prune will fail against the live endpoint. Two options, pick one and stick with it:

Option A — temporary flag flip (simplest, requires a maintenance window)

1. On the Synology, edit the stack's .env and set EXTRA_OPTIONS=--no-auth — only kidding, don't. Set EXTRA_OPTIONS= and comment out --append-only in the compose OPTIONS= line (or parameterize if you prefer).
2. docker compose up -d to restart with deletes allowed.
3. Run restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-yearly 3 --prune from the origin host.
4. Restore --append-only and docker compose up -d.

Treat this as a quarterly change, not a cron job. Schedule it so you're present if restic hits anything weird.

Option B — second endpoint on a different port (automation-friendly)

Stand up a second rest-server container against the same DATA_DIR without --append-only, listening on e.g. 8001, reachable only from within the Synology / over VPN. A scheduled prune job hits that endpoint; day-to-day backup traffic continues to hit :8000 in append-only mode.

If you end up wanting this, copy this stack to stacks/rest-server-nh3-prune/ with REST_PORT=8001 and --append-only removed.

Off-site replication

Scheduled on ana-docker:

# Example — not the final script, just illustrating the shape.
rsync -avz --delete \
  admin@10.100.50.50:/volume1/Backup/restic/ \
  /mnt/backup/restic-mirror-nh3/

rsync is safe because restic packs are immutable once written — nothing under /volume1/Backup/restic/<user>/data/ gets rewritten, only added or (during prune) removed. A raw rsync --delete with prune running only on the origin side is enough; no filesystem-level locks required.

What doesn't live here

• No .env in the committed copy — only .env.example.
• .htpasswd is never checked in, never synced via sync-stacks.sh (its *.ht* isn't in the global exclude but the data dir is outside the stack path).
• Client-side restic passwords (the encryption passphrase for each repo) are separate from the HTTP auth passwords and never stored on the Synology.