vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: PFI fleet inventory, stacks, tooling, and backup pipeline

Browse Source 
Captures the full workspace state built up to this point:

  - CLAUDE.md + README.md describing conventions and the four-host fleet
    (ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
  - Per-host notes under servers/<host>/ with ssh-target fallback files
    and latest system-details snapshots (two in-compose credential leaks
    scrubbed; the upstream compose files still need to move those to .env).
  - scripts/: server_inspect.sh (read-only remote diagnostic),
    refresh-server-info.sh (dir-driven discovery + snapshot capture with
    validation warnings), add-host.sh, sync-stacks.sh (pull
    compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
  - stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
    rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
    infinity reference. All use the .env-driven + traefik-net + homepage
    label pattern.
  - configs/restic/ana-docker/: first resticprofile config + pre-backup
    hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
    for the other three hosts to come.
  - docs/pfi/: general infrastructure reference carried over.
  - .gitignore excludes .env, stacks-mirror/, and assorted secret/state
    filenames to prevent re-leaks on later commits.
This commit is contained in:
vh
2026-04-20 14:29:48 -07:00
commit e376d0aec9
55 changed files with 9101 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/rest-server-nh3/.env.example
+35
View File
@@ -0,0 +1,35 @@
# rest-server (NH3 Synology) tunables. Copy to `.env` on the Synology.
#
# cp .env.example .env
# # edit if needed
# docker compose up -d
#
# Synology-specific notes:
# - Put this stack under /volume1/docker/compose/rest-server/
# (Container Manager's default project root pattern on DSM 7.x).
# - Point DATA_DIR at a Btrfs share you own — /volume1/Backup/restic
# is the natural choice since /volume1/Backup is already the
# fleet-facing backup share (exported via NFS).
REST_SERVER_VERSION=latest
# Host port the container listens on (container internal port is 8000)
REST_PORT=8000
# Where restic pack files live on the Synology. Must be a writable
# Btrfs path. Will hold one subdir per user (--private-repos layout):
# ${DATA_DIR}/ana-docker/
# ${DATA_DIR}/ana-ml2/
# ${DATA_DIR}/nh3-docker/
# ${DATA_DIR}/esh-docker-vm/
# Plus the auth file at ${DATA_DIR}/.htpasswd.
DATA_DIR=/volume1/Backup/restic
# Timezone — affects log lines and the /metrics timestamps
TZ=America/Los_Angeles
# Any extra rest-server flags (rare). Some useful ones:
# --no-verify-upload — trust the client's hash; faster writes
# --max-size=<bytes> — cap per-repo size
# Leave blank unless you have a reason.
EXTRA_OPTIONS=
stacks/rest-server-nh3/README.md
+124
View File
@@ -0,0 +1,124 @@
# rest-server-nh3
NH3-site restic backup endpoint. Runs in Synology Container Manager on `10.100.50.50` and stores pack files on a Btrfs share so Synology snapshots protect against local corruption.
**Server:** Synology RS2418+ at `10.100.50.50`
**Port:** `http://10.100.50.50:8000` (configurable via `.env`)
**Data:** `/volume1/Backup/restic/` (configurable)
Paired with the existing `restic rest-server` on **ana-docker** (`http://10.250.50.70:8000`, data on TrueNAS NFS) as the Anaheim-side endpoint. Each fleet host backs up to the rest-server closest to it; an rsync job on ana-docker mirrors the two trees against each other for off-site redundancy.
## Auth model
`--private-repos` + `--append-only`, enforced via htpasswd:
- One HTTP basic-auth user **per host** (`ana-docker`, `ana-ml2`, `nh3-docker`, `esh-docker-vm`).
- Each user can only write under `/<username>/…` — a compromised host can't see or delete another host's data.
- Append-only means a compromised client can add to its own repo but can't rewrite or delete existing packs, so ransomware on a backed-up host doesn't destroy history.
- **Trade-off:** `restic forget --prune` can't run against an append-only endpoint. Prune ceremony described at the bottom of this file.
## Pre-deploy: create the data path and htpasswd
On the Synology (SSH in as an admin-capable user, or DSM *File Station*):
```bash
# 1. Create the restic data share on a Btrfs volume
ssh admin@10.100.50.50 'sudo mkdir -p /volume1/Backup/restic && \
sudo chown 1000:1000 /volume1/Backup/restic && \
sudo chmod 700 /volume1/Backup/restic'
# 2. Generate htpasswd entries. The Synology doesn't ship apache2-utils,
# so use a throwaway container:
ssh admin@10.100.50.50 'cd /volume1/Backup/restic && \
sudo touch .htpasswd && sudo chown 1000:1000 .htpasswd && sudo chmod 600 .htpasswd'
for user in ana-docker ana-ml2 nh3-docker esh-docker-vm; do
read -s -p "password for $user: " pw; echo
ssh admin@10.100.50.50 \
"docker run --rm httpd:2.4-alpine htpasswd -nbB $user '$pw'" \
| ssh admin@10.100.50.50 "sudo tee -a /volume1/Backup/restic/.htpasswd >/dev/null"
done
```
Record every password in your off-host password manager (1Password / Vaultwarden etc.) — you'll paste them into Backrest and into the systemd timer configs later.
## Deploy
In Synology **Container Manager**:
1. *Project* → **Create** → Name `rest-server`, Path `/volume1/docker/compose/rest-server/`.
2. Copy `compose.yaml` into the project path; copy `.env.example` → `.env` and edit if needed (default `REST_PORT=8000` and `DATA_DIR=/volume1/Backup/restic` should be fine).
3. Start the project.
CLI equivalent (if you have SSH + a shell account that can run Docker on the NAS):
```bash
ssh admin@10.100.50.50
sudo mkdir -p /volume1/docker/compose/rest-server
sudo chown $USER /volume1/docker/compose/rest-server
cd /volume1/docker/compose/rest-server
# scp the files from this workspace, then:
cp .env.example .env
docker compose config
docker compose up -d
docker compose logs -f
```
## Verify
From this workstation:
```bash
# Should return "200 OK" or redirect to /metrics; anything 5xx is a problem.
curl -u ana-docker:<password> -sv http://10.100.50.50:8000/ana-docker/ -o /dev/null
# Once restic is wired up, init the repo (one-time, per host):
RESTIC_REPOSITORY='rest:http://ana-docker:<password>@10.100.50.50:8000/ana-docker/' \
RESTIC_PASSWORD='<client-side-encryption-passphrase>' \
restic init
```
Restic URI shape for each host (paste into Backrest when adding the repo):
```
rest:http://<user>:<pass>@10.100.50.50:8000/<user>/
```
## Prune ceremony (because of --append-only)
Because `--append-only` blocks deletes, `restic forget --prune` will fail against the live endpoint. Two options, pick one and stick with it:
### Option A — temporary flag flip (simplest, requires a maintenance window)
1. On the Synology, edit the stack's `.env` and set `EXTRA_OPTIONS=--no-auth` — **only kidding, don't.** Set `EXTRA_OPTIONS= ` and comment out `--append-only` in the compose `OPTIONS=` line (or parameterize if you prefer).
2. `docker compose up -d` to restart with deletes allowed.
3. Run `restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-yearly 3 --prune` from the origin host.
4. Restore `--append-only` and `docker compose up -d`.
Treat this as a quarterly change, not a cron job. Schedule it so you're present if restic hits anything weird.
### Option B — second endpoint on a different port (automation-friendly)
Stand up a second `rest-server` container against the same `DATA_DIR` without `--append-only`, listening on e.g. `8001`, reachable only from within the Synology / over VPN. A scheduled prune job hits that endpoint; day-to-day backup traffic continues to hit `:8000` in append-only mode.
If you end up wanting this, copy this stack to `stacks/rest-server-nh3-prune/` with `REST_PORT=8001` and `--append-only` removed.
## Off-site replication
Scheduled on ana-docker:
```bash
# Example — not the final script, just illustrating the shape.
rsync -avz --delete \
admin@10.100.50.50:/volume1/Backup/restic/ \
/mnt/backup/restic-mirror-nh3/
```
`rsync` is safe because restic packs are immutable once written — nothing under `/volume1/Backup/restic/<user>/data/` gets rewritten, only added or (during prune) removed. A raw `rsync --delete` with prune running only on the origin side is enough; no filesystem-level locks required.
## What doesn't live here
- No `.env` in the committed copy — only `.env.example`.
- `.htpasswd` is never checked in, never synced via `sync-stacks.sh` (its `*.ht*` isn't in the global exclude but the data dir is outside the stack path).
- Client-side restic passwords (the encryption passphrase for each repo) are separate from the HTTP auth passwords and never stored on the Synology.
stacks/rest-server-nh3/compose.yaml
+40
View File
@@ -0,0 +1,40 @@
# rest-server (NH3 Synology) — restic backup target for the fleet.
#
# Deploys into Synology Container Manager on 10.100.50.50. Data lives on
# a Btrfs shared folder so it gets Synology snapshots + optional
# replication to a sibling share if you configure one later.
#
# Auth model:
# --private-repos : every URL path must start with /<username>/ and the
# HTTP basic-auth user must match. One user per host.
# Per-host repos are strictly isolated.
# --append-only : on-disk data can be ADDED but not REMOVED or REWRITTEN.
# A compromised host can't delete its own history.
# Prune requires disabling this (see README).
#
# Credentials come from /data/.htpasswd — see README for how to populate
# it. That file is mounted read-only into the container.
#
# All tunables live in .env — edit that, not this file.
services:
rest-server:
image: restic/rest-server:${REST_SERVER_VERSION}
container_name: rest-server
restart: unless-stopped
ports:
- "${REST_PORT}:8000"
volumes:
- ${DATA_DIR}:/data
environment:
- OPTIONS=--private-repos --append-only --prometheus ${EXTRA_OPTIONS:-}
- TZ=${TZ:-America/Los_Angeles}
# rest-server stores repos under /data and looks for /data/.htpasswd
# automatically — no extra bind mount needed as long as the htpasswd
# file is created inside DATA_DIR before startup.
healthcheck:
test: ["CMD-SHELL", "wget -qO- http://localhost:8000/metrics >/dev/null || exit 1"]
interval: 30s
timeout: 10s
retries: 3
start_period: 15s