Initial commit: PFI fleet inventory, stacks, tooling, and backup pipeline
Captures the full workspace state built up to this point:
- CLAUDE.md + README.md describing conventions and the four-host fleet
(ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
- Per-host notes under servers/<host>/ with ssh-target fallback files
and latest system-details snapshots (two in-compose credential leaks
scrubbed; the upstream compose files still need to move those to .env).
- scripts/: server_inspect.sh (read-only remote diagnostic),
refresh-server-info.sh (dir-driven discovery + snapshot capture with
validation warnings), add-host.sh, sync-stacks.sh (pull
compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
- stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
infinity reference. All use the .env-driven + traefik-net + homepage
label pattern.
- configs/restic/ana-docker/: first resticprofile config + pre-backup
hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
for the other three hosts to come.
- docs/pfi/: general infrastructure reference carried over.
- .gitignore excludes .env, stacks-mirror/, and assorted secret/state
filenames to prevent re-leaks on later commits.
This commit is contained in:
@@ -0,0 +1,41 @@
|
||||
# rest-server-ana stack tunables. Copy to `.env` on ana-docker.
|
||||
#
|
||||
# cp .env.example .env
|
||||
# # edit if needed
|
||||
# docker compose up -d
|
||||
#
|
||||
# Matches stacks/rest-server-nh3/.env.example — keep them aligned so
|
||||
# restic clients see the same URL shape against either endpoint.
|
||||
|
||||
REST_SERVER_VERSION=latest
|
||||
|
||||
# Host port the container listens on (container internal is 8000)
|
||||
REST_PORT=8000
|
||||
|
||||
# Where restic pack files live. TrueNAS NFS share is already mounted at
|
||||
# /mnt/backup on ana-docker; the "repo/ana" subdir is the historical
|
||||
# location used by the prior (non-private-repos) rest-server.
|
||||
#
|
||||
# With --private-repos, layout becomes:
|
||||
# ${DATA_DIR}/ana-docker/
|
||||
# ${DATA_DIR}/ana-ml2/
|
||||
# ${DATA_DIR}/nh3-docker/
|
||||
# ${DATA_DIR}/esh-docker-vm/
|
||||
# Plus the auth file at ${DATA_DIR}/.htpasswd.
|
||||
DATA_DIR=/mnt/backup/restic/repo/ana
|
||||
|
||||
# UID/GID the container process runs as. Must match the owner of
|
||||
# DATA_DIR so NFS root_squash doesn't bite. On ana-docker this is the
|
||||
# `lkraven` user (1000:1000).
|
||||
REST_UID=1000
|
||||
REST_GID=1000
|
||||
|
||||
# Timezone — affects log lines and /metrics timestamps
|
||||
TZ=America/Los_Angeles
|
||||
|
||||
# Extra rest-server flags. Examples:
|
||||
# --prometheus-no-auth — make /metrics public (needed if Beszel or
|
||||
# Prometheus scrapes without creds)
|
||||
# --no-verify-upload — trust the client's hash; faster writes
|
||||
# Leave blank unless you have a reason.
|
||||
EXTRA_OPTIONS=
|
||||
@@ -0,0 +1,143 @@
|
||||
# rest-server-ana
|
||||
|
||||
Anaheim-site restic backup endpoint. Replaces the older `restic` stack on ana-docker with the same auth model as `rest-server-nh3` on the Synology, so every client host uses identical URL shapes against either endpoint.
|
||||
|
||||
**Server:** ana-docker (`10.250.50.70`)
|
||||
**Port:** `http://10.250.50.70:8000`
|
||||
**Data:** `/mnt/backup/restic/repo/ana/` (TrueNAS NFS mount on the host)
|
||||
|
||||
Paired with:
|
||||
- **`rest-server-nh3`** on the Synology (`10.100.50.50:8000`, data on Btrfs).
|
||||
- A cross-site rsync job (TBD, on ana-docker) that mirrors each site's data tree to the other so either NAS can fully restore either site's hosts.
|
||||
|
||||
## What changed from the old `restic` stack
|
||||
|
||||
| | old `restic` on ana-docker | this stack |
|
||||
|---|---|---|
|
||||
| `--private-repos` | no | **yes** |
|
||||
| `--append-only` | no | **yes** |
|
||||
| `--prometheus` | no | **yes** |
|
||||
| healthcheck | no | yes |
|
||||
| `.env`-driven | no | yes |
|
||||
| restart policy | none | `unless-stopped` |
|
||||
| image version | floating `latest` | `${REST_SERVER_VERSION}` |
|
||||
| stack dir on server | `/opt/docker/compose/restic/` | `/opt/docker/compose/rest-server-ana/` |
|
||||
|
||||
Data path is unchanged (`/mnt/backup/restic/repo/ana/`) so nothing new needs to be allocated on TrueNAS.
|
||||
|
||||
## Pre-deploy: clean the data dir and create htpasswd
|
||||
|
||||
Since there's nothing in the existing path we want to keep, start fresh so the on-disk layout matches `--private-repos`:
|
||||
|
||||
```bash
|
||||
ssh ana-docker '
|
||||
# Stop the old stack so port 8000 and the data dir are free
|
||||
cd /opt/docker/compose/restic
|
||||
docker compose down
|
||||
|
||||
# Wipe the old non-private-repos layout
|
||||
sudo rm -rf /mnt/backup/restic/repo/ana/*
|
||||
sudo rm -rf /mnt/backup/restic/repo/ana/.htpasswd # if present
|
||||
|
||||
# Create the htpasswd file. Use the same passwords here as on the NH3
|
||||
# Synology so each host has one credential that works at either endpoint.
|
||||
sudo touch /mnt/backup/restic/repo/ana/.htpasswd
|
||||
sudo chmod 600 /mnt/backup/restic/repo/ana/.htpasswd
|
||||
'
|
||||
|
||||
# Generate htpasswd entries locally (one per host) and append. Using the
|
||||
# `httpd:2.4-alpine` throwaway container so we do not depend on
|
||||
# apache2-utils being installed on ana-docker.
|
||||
for user in ana-docker ana-ml2 nh3-docker esh-docker-vm; do
|
||||
read -rs -p "password for $user (must match the NH3 Synology): " pw; echo
|
||||
docker run --rm httpd:2.4-alpine htpasswd -nbB "$user" "$pw" \
|
||||
| ssh ana-docker 'sudo tee -a /mnt/backup/restic/repo/ana/.htpasswd >/dev/null'
|
||||
done
|
||||
```
|
||||
|
||||
If you run that locally and don't have Docker here, equivalent on the server:
|
||||
|
||||
```bash
|
||||
ssh ana-docker "docker run --rm httpd:2.4-alpine htpasswd -nbB <user> '<pw>'" \
|
||||
| ssh ana-docker 'sudo tee -a /mnt/backup/restic/repo/ana/.htpasswd >/dev/null'
|
||||
```
|
||||
|
||||
## Deploy
|
||||
|
||||
Stage the new stack and push it:
|
||||
|
||||
```bash
|
||||
# Stage the stack into the mirror (if not already done via sync-stacks.sh)
|
||||
mkdir -p stacks-mirror/ana-docker/rest-server-ana
|
||||
cp stacks/rest-server-ana/compose.yaml stacks/rest-server-ana/.env.example \
|
||||
stacks-mirror/ana-docker/rest-server-ana/
|
||||
|
||||
scripts/deploy-stack.sh ana-docker rest-server-ana
|
||||
```
|
||||
|
||||
Confirm at the prompt. Then on the server:
|
||||
|
||||
```bash
|
||||
ssh ana-docker '
|
||||
cd /opt/docker/compose/rest-server-ana
|
||||
cp -n .env.example .env
|
||||
docker compose config
|
||||
docker compose up -d
|
||||
docker compose logs --tail=30
|
||||
'
|
||||
```
|
||||
|
||||
## Retire the old stack
|
||||
|
||||
Once the new one is healthy and the first repo has initialized successfully from a client:
|
||||
|
||||
```bash
|
||||
ssh ana-docker '
|
||||
cd /opt/docker/compose/restic
|
||||
docker compose down
|
||||
# Optionally remove the old stack dir (keep it for a release or two
|
||||
# in case you need to roll back):
|
||||
# rm -rf /opt/docker/compose/restic
|
||||
'
|
||||
```
|
||||
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
# 401 from the root — service up, auth enforced
|
||||
curl -sS -o /dev/null -w 'unauth status=%{http_code}\n' \
|
||||
http://10.250.50.70:8000/
|
||||
|
||||
# 200 / 404 from a real user+password — auth valid, --private-repos path OK
|
||||
curl -sS -o /dev/null -w 'auth status=%{http_code}\n' \
|
||||
-u ana-docker:<password> http://10.250.50.70:8000/ana-docker/
|
||||
|
||||
# Init a repo from a client host (one-time per host)
|
||||
ssh ana-docker '
|
||||
export RESTIC_REPOSITORY="rest:http://ana-docker:<rest-pw>@10.250.50.70:8000/ana-docker/"
|
||||
export RESTIC_PASSWORD="<client-side-encryption-passphrase>"
|
||||
restic init
|
||||
'
|
||||
```
|
||||
|
||||
## Prune ceremony
|
||||
|
||||
Same as `rest-server-nh3` — prune is blocked by `--append-only`. Two options, pick one per endpoint:
|
||||
|
||||
- **Temporary flag flip:** edit compose, remove `--append-only` from `OPTIONS`, `docker compose up -d`, run `restic forget --prune` from origin hosts, put the flag back, `docker compose up -d`. Quarterly change.
|
||||
- **Second endpoint on a different port:** stand up a sibling container (e.g. port `8001`) against the same data dir without `--append-only`, reachable only from a trusted host. Everyday backups still hit `:8000`.
|
||||
|
||||
If you go the second-endpoint route, copy this stack to `stacks/rest-server-ana-prune/` with `REST_PORT=8001` and `--append-only` removed from the compose.
|
||||
|
||||
## Off-site replication
|
||||
|
||||
Scheduled on ana-docker (to be written):
|
||||
|
||||
```bash
|
||||
# Pull NH3's tree down to this side
|
||||
rsync -avz --delete admin@10.100.50.50:/volume1/Backup/restic/ /mnt/backup/restic-mirror-nh3/
|
||||
# Push our tree to NH3
|
||||
rsync -avz --delete /mnt/backup/restic/repo/ana/ admin@10.100.50.50:/volume1/Backup/restic-mirror-ana/
|
||||
```
|
||||
|
||||
Two unidirectional syncs, each running in the direction its data flows. Prune runs only at the origin so the mirror shrinks correctly.
|
||||
@@ -0,0 +1,59 @@
|
||||
# rest-server (Anaheim) — restic backup target for the fleet.
|
||||
#
|
||||
# Deploys to ana-docker. Data dir is on the TrueNAS NFS mount
|
||||
# (/mnt/backup/restic/repo/ana) so snapshots on the NAS side protect the
|
||||
# backup blobs themselves.
|
||||
#
|
||||
# Mirrors stacks/rest-server-nh3/ in every meaningful way — same auth
|
||||
# model, same on-disk layout, same operational semantics — so each client
|
||||
# host uses an identical URL shape against either endpoint:
|
||||
#
|
||||
# rest:http://<user>:<pw>@10.100.50.50:8000/<user>/ (NH3 Synology)
|
||||
# rest:http://<user>:<pw>@10.250.50.70:8000/<user>/ (this stack)
|
||||
#
|
||||
# Auth model:
|
||||
# --private-repos : URL path must start with /<user>/ and the HTTP
|
||||
# basic-auth user must match. Per-host repos are
|
||||
# strictly isolated.
|
||||
# --append-only : on-disk data can be added but not removed or
|
||||
# rewritten; a compromised host can't wipe its own
|
||||
# history. Prune requires disabling this (see README).
|
||||
#
|
||||
# Credentials come from /data/.htpasswd — see README for populating it.
|
||||
#
|
||||
# All tunables live in .env — edit that, not this file.
|
||||
|
||||
services:
|
||||
rest-server:
|
||||
image: restic/rest-server:${REST_SERVER_VERSION}
|
||||
container_name: rest-server
|
||||
restart: unless-stopped
|
||||
# Run as the UID that owns the NFS-backed data dir, so file I/O
|
||||
# is not subject to NFS root_squash. On ana-docker this is lkraven (1000).
|
||||
user: "${REST_UID:-1000}:${REST_GID:-1000}"
|
||||
ports:
|
||||
- "${REST_PORT}:8000"
|
||||
volumes:
|
||||
- ${DATA_DIR}:/data
|
||||
environment:
|
||||
- OPTIONS=--private-repos --append-only --prometheus ${EXTRA_OPTIONS:-}
|
||||
- TZ=${TZ:-America/Los_Angeles}
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "wget -qO- http://localhost:8000/metrics >/dev/null 2>&1 || [ $? -eq 6 ] && exit 0 || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 15s
|
||||
networks:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=PFI-ANA
|
||||
- homepage.name=Restic (rest-server)
|
||||
- homepage.icon=mdi-cloud-upload
|
||||
- homepage.description=Anaheim restic endpoint (data on TrueNAS NFS)
|
||||
- homepage.href=http://10.250.50.70:${REST_PORT}
|
||||
|
||||
networks:
|
||||
tnet:
|
||||
name: traefik-net
|
||||
external: true
|
||||
Reference in New Issue
Block a user