vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: PFI fleet inventory, stacks, tooling, and backup pipeline

Browse Source 
Captures the full workspace state built up to this point:

  - CLAUDE.md + README.md describing conventions and the four-host fleet
    (ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
  - Per-host notes under servers/<host>/ with ssh-target fallback files
    and latest system-details snapshots (two in-compose credential leaks
    scrubbed; the upstream compose files still need to move those to .env).
  - scripts/: server_inspect.sh (read-only remote diagnostic),
    refresh-server-info.sh (dir-driven discovery + snapshot capture with
    validation warnings), add-host.sh, sync-stacks.sh (pull
    compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
  - stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
    rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
    infinity reference. All use the .env-driven + traefik-net + homepage
    label pattern.
  - configs/restic/ana-docker/: first resticprofile config + pre-backup
    hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
    for the other three hosts to come.
  - docs/pfi/: general infrastructure reference carried over.
  - .gitignore excludes .env, stacks-mirror/, and assorted secret/state
    filenames to prevent re-leaks on later commits.
This commit is contained in:
vh
2026-04-20 14:29:48 -07:00
commit e376d0aec9
55 changed files with 9101 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/rest-server-ana/README.md
+143
View File
@@ -0,0 +1,143 @@
# rest-server-ana
Anaheim-site restic backup endpoint. Replaces the older `restic` stack on ana-docker with the same auth model as `rest-server-nh3` on the Synology, so every client host uses identical URL shapes against either endpoint.
**Server:** ana-docker (`10.250.50.70`)
**Port:** `http://10.250.50.70:8000`
**Data:** `/mnt/backup/restic/repo/ana/` (TrueNAS NFS mount on the host)
Paired with:
- **`rest-server-nh3`** on the Synology (`10.100.50.50:8000`, data on Btrfs).
- A cross-site rsync job (TBD, on ana-docker) that mirrors each site's data tree to the other so either NAS can fully restore either site's hosts.
## What changed from the old `restic` stack
| | old `restic` on ana-docker | this stack |
|---|---|---|
| `--private-repos` | no | **yes** |
| `--append-only` | no | **yes** |
| `--prometheus` | no | **yes** |
| healthcheck | no | yes |
| `.env`-driven | no | yes |
| restart policy | none | `unless-stopped` |
| image version | floating `latest` | `${REST_SERVER_VERSION}` |
| stack dir on server | `/opt/docker/compose/restic/` | `/opt/docker/compose/rest-server-ana/` |
Data path is unchanged (`/mnt/backup/restic/repo/ana/`) so nothing new needs to be allocated on TrueNAS.
## Pre-deploy: clean the data dir and create htpasswd
Since there's nothing in the existing path we want to keep, start fresh so the on-disk layout matches `--private-repos`:
```bash
ssh ana-docker '
# Stop the old stack so port 8000 and the data dir are free
cd /opt/docker/compose/restic
docker compose down
# Wipe the old non-private-repos layout
sudo rm -rf /mnt/backup/restic/repo/ana/*
sudo rm -rf /mnt/backup/restic/repo/ana/.htpasswd # if present
# Create the htpasswd file. Use the same passwords here as on the NH3
# Synology so each host has one credential that works at either endpoint.
sudo touch /mnt/backup/restic/repo/ana/.htpasswd
sudo chmod 600 /mnt/backup/restic/repo/ana/.htpasswd
'
# Generate htpasswd entries locally (one per host) and append. Using the
# `httpd:2.4-alpine` throwaway container so we do not depend on
# apache2-utils being installed on ana-docker.
for user in ana-docker ana-ml2 nh3-docker esh-docker-vm; do
read -rs -p "password for $user (must match the NH3 Synology): " pw; echo
docker run --rm httpd:2.4-alpine htpasswd -nbB "$user" "$pw" \
| ssh ana-docker 'sudo tee -a /mnt/backup/restic/repo/ana/.htpasswd >/dev/null'
done
```
If you run that locally and don't have Docker here, equivalent on the server:
```bash
ssh ana-docker "docker run --rm httpd:2.4-alpine htpasswd -nbB <user> '<pw>'" \
| ssh ana-docker 'sudo tee -a /mnt/backup/restic/repo/ana/.htpasswd >/dev/null'
```
## Deploy
Stage the new stack and push it:
```bash
# Stage the stack into the mirror (if not already done via sync-stacks.sh)
mkdir -p stacks-mirror/ana-docker/rest-server-ana
cp stacks/rest-server-ana/compose.yaml stacks/rest-server-ana/.env.example \
stacks-mirror/ana-docker/rest-server-ana/
scripts/deploy-stack.sh ana-docker rest-server-ana
```
Confirm at the prompt. Then on the server:
```bash
ssh ana-docker '
cd /opt/docker/compose/rest-server-ana
cp -n .env.example .env
docker compose config
docker compose up -d
docker compose logs --tail=30
'
```
## Retire the old stack
Once the new one is healthy and the first repo has initialized successfully from a client:
```bash
ssh ana-docker '
cd /opt/docker/compose/restic
docker compose down
# Optionally remove the old stack dir (keep it for a release or two
# in case you need to roll back):
# rm -rf /opt/docker/compose/restic
'
```
## Verify
```bash
# 401 from the root — service up, auth enforced
curl -sS -o /dev/null -w 'unauth status=%{http_code}\n' \
http://10.250.50.70:8000/
# 200 / 404 from a real user+password — auth valid, --private-repos path OK
curl -sS -o /dev/null -w 'auth status=%{http_code}\n' \
-u ana-docker:<password> http://10.250.50.70:8000/ana-docker/
# Init a repo from a client host (one-time per host)
ssh ana-docker '
export RESTIC_REPOSITORY="rest:http://ana-docker:<rest-pw>@10.250.50.70:8000/ana-docker/"
export RESTIC_PASSWORD="<client-side-encryption-passphrase>"
restic init
'
```
## Prune ceremony
Same as `rest-server-nh3` — prune is blocked by `--append-only`. Two options, pick one per endpoint:
- **Temporary flag flip:** edit compose, remove `--append-only` from `OPTIONS`, `docker compose up -d`, run `restic forget --prune` from origin hosts, put the flag back, `docker compose up -d`. Quarterly change.
- **Second endpoint on a different port:** stand up a sibling container (e.g. port `8001`) against the same data dir without `--append-only`, reachable only from a trusted host. Everyday backups still hit `:8000`.
If you go the second-endpoint route, copy this stack to `stacks/rest-server-ana-prune/` with `REST_PORT=8001` and `--append-only` removed from the compose.
## Off-site replication
Scheduled on ana-docker (to be written):
```bash
# Pull NH3's tree down to this side
rsync -avz --delete admin@10.100.50.50:/volume1/Backup/restic/ /mnt/backup/restic-mirror-nh3/
# Push our tree to NH3
rsync -avz --delete /mnt/backup/restic/repo/ana/ admin@10.100.50.50:/volume1/Backup/restic-mirror-ana/
```
Two unidirectional syncs, each running in the direction its data flows. Prune runs only at the origin so the mirror shrinks correctly.