vh
39da1d4a97
The board mixed tools with endpoints. A vLLM seat whose href is a /docs page
sat in the same band as ComfyUI; the MQTT broker and the RustDesk relay, which
have no page at all, sat in Apps; and `Service Networking` was thirteen members
spanning three AdGuards, five Dockges, two Traefiks and four headless agents.
Every group is now one of two kinds and they never mix. TOOLS are expanded and
sit at the top of their tab. ENDPOINTS — an API, a broker, a background agent,
an href that is /docs or /ping or nothing — carry `initiallyCollapsed: true`
and sit at the bottom. Collapsed is not hidden: the eyebrow and its rule still
render, so the tab still says the thing exists and one click expands it.
A second rule fell out of the same pass and now shapes the group boundaries: a
group's members should all carry a widget or none should. A stat strip makes a
card ~50px taller, so one widget card in a row of plain ones opens a void under
the plain ones. That is why AdGuard and Traefik get their own groups rather
than sharing one with Dockge, and it is most of why the old Service Networking
band looked broken. AdGuard (ANA) was the last short card in its row and now
carries the same query/blocked/latency strip as its two siblings — one
infra-ops AdGuard login authenticates against all three instances, verified
against each; it lives in that stack's .env on the host and is vaulted.
The sixteen GPU-backed model seats were deliberately NOT relabelled.
`homepage.group` is read at container creation, so clearer names for
`AI - Inference` and friends would have cost a recreate on six vLLM seats, four
eval seats and four TTS engines — multi-minute model reloads on endpoints peers
reach through the gateway. Order plus `initiallyCollapsed` buys the same
separation for nothing, so those names stay as they are on purpose.
28 containers that ARE cheap to bounce were relabelled, across five hosts, via
rerunnable elway playbooks. Their label steps are gated on the old value still
being present, so a second run reports skipped rather than churning. Two verify
steps were wrong on first contact and are fixed with the reason recorded: the
traefik check raced its own recreate, and asserting a model seat is "running"
cannot answer "did I bounce it" when a seat may be legitimately stopped —
container age can, and now does.
The canonical stacks/ tree was synced to the deployed labels afterwards, so
intent and reality agree again on all fourteen tracked stacks.
Also documents the real nature of the post-recreate blank dashboard, which cost
~25 minutes here and an hour on 2026-08-19. `initialSettings":{}` in the served
HTML is the catch branch of the page's data loader, not a warm-up and not a
cache — and the error can vanish entirely, because the logger is assigned inside
the same try and the catch only logs if the logger exists. Ruled out by
measurement this time: all four API routes return 200 with correct content while
the page serves {}, and the previous known-good settings.yaml reproduces it
identically. The README now carries the one-command test and the next lead.
Before/after, all four tabs: http://10.100.10.50:8090/b/homepage-relayout/
rest-server-ana
Anaheim-site restic backup endpoint. Replaces the older restic stack on ana-docker with the same auth model as rest-server-nh3 on the Synology, so every client host uses identical URL shapes against either endpoint.
Server: ana-docker (10.250.50.70)
Port: http://10.250.50.70:8000
Data root: /mnt/backup/restic/repo/ (NFS mount on the host, served by the Debian file server at 10.250.50.50)
Data layout
The NFS export at /mnt/backup/restic/repo/ is partitioned by site into three sibling subdirs:
| Path | Role |
|---|---|
/mnt/backup/restic/repo/ana/ |
Live data served by this rest-server. Each Anaheim-side host writes its repo under here (e.g. ana/ana-docker/, ana/ana-ml2/). .htpasswd also lives here. |
/mnt/backup/restic/repo/esh/ |
Destination for mirroring ESH-site backups to Anaheim (cross-site redundancy; not yet wired up). |
/mnt/backup/restic/repo/nh3/ |
Destination for mirroring NH3-site backups (from the Synology rest-server-nh3) into Anaheim. Same — pending. |
The rest-server container is configured with /mnt/backup/restic/repo/ana as its data root (see compose's DATA_DIR env), so clients use URLs of the form rest:http://user:pw@10.250.50.70:8000/<host>/ — the <host> path is relative to ana/.
The two mirror dirs (esh/, nh3/) aren't served by restic at all; they're rsync destinations.
Paired with:
rest-server-nh3on the Synology (10.100.50.50:8000, data on Btrfs).- A cross-site rsync job (TBD, on ana-docker) that mirrors each site's data tree to the other so either NAS can fully restore either site's hosts.
What changed from the old restic stack
old restic on ana-docker |
this stack | |
|---|---|---|
--private-repos |
no | yes |
--append-only |
no | yes |
--prometheus |
no | yes |
| healthcheck | no | yes |
.env-driven |
no | yes |
| restart policy | none | unless-stopped |
| image version | floating latest |
${REST_SERVER_VERSION} |
| stack dir on server | /opt/docker/compose/restic/ |
/opt/docker/compose/rest-server-ana/ |
Data path is unchanged (/mnt/backup/restic/repo/ana/) so nothing new needs to be allocated on the NAS.
Pre-deploy: clean the data dir and create htpasswd
Since there's nothing in the existing path we want to keep, start fresh so the on-disk layout matches --private-repos:
ssh ana-docker '
# Stop the old stack so port 8000 and the data dir are free
cd /opt/docker/compose/restic
docker compose down
# Wipe the old non-private-repos layout
sudo rm -rf /mnt/backup/restic/repo/ana/*
sudo rm -rf /mnt/backup/restic/repo/ana/.htpasswd # if present
# Create the htpasswd file. Use the same passwords here as on the NH3
# Synology so each host has one credential that works at either endpoint.
sudo touch /mnt/backup/restic/repo/ana/.htpasswd
sudo chmod 600 /mnt/backup/restic/repo/ana/.htpasswd
'
# Generate htpasswd entries locally (one per host) and append. Using the
# `httpd:2.4-alpine` throwaway container so we do not depend on
# apache2-utils being installed on ana-docker.
for user in ana-docker ana-ml2 nh3-docker esh-docker-vm; do
read -rs -p "password for $user (must match the NH3 Synology): " pw; echo
docker run --rm httpd:2.4-alpine htpasswd -nbB "$user" "$pw" \
| ssh ana-docker 'sudo tee -a /mnt/backup/restic/repo/ana/.htpasswd >/dev/null'
done
If you run that locally and don't have Docker here, equivalent on the server:
ssh ana-docker "docker run --rm httpd:2.4-alpine htpasswd -nbB <user> '<pw>'" \
| ssh ana-docker 'sudo tee -a /mnt/backup/restic/repo/ana/.htpasswd >/dev/null'
Deploy
Stage the new stack and push it:
# Stage the stack into the mirror (if not already done via sync-stacks.sh)
mkdir -p stacks-mirror/ana-docker/rest-server-ana
cp stacks/rest-server-ana/compose.yaml stacks/rest-server-ana/.env.example \
stacks-mirror/ana-docker/rest-server-ana/
scripts/deploy-stack.sh ana-docker rest-server-ana
Confirm at the prompt. Then on the server:
ssh ana-docker '
cd /opt/docker/compose/rest-server-ana
cp -n .env.example .env
docker compose config
docker compose up -d
docker compose logs --tail=30
'
Retire the old stack
Once the new one is healthy and the first repo has initialized successfully from a client:
ssh ana-docker '
cd /opt/docker/compose/restic
docker compose down
# Optionally remove the old stack dir (keep it for a release or two
# in case you need to roll back):
# rm -rf /opt/docker/compose/restic
'
Verify
# 401 from the root — service up, auth enforced
curl -sS -o /dev/null -w 'unauth status=%{http_code}\n' \
http://10.250.50.70:8000/
# 200 / 404 from a real user+password — auth valid, --private-repos path OK
curl -sS -o /dev/null -w 'auth status=%{http_code}\n' \
-u ana-docker:<password> http://10.250.50.70:8000/ana-docker/
# Init a repo from a client host (one-time per host)
ssh ana-docker '
export RESTIC_REPOSITORY="rest:http://ana-docker:<rest-pw>@10.250.50.70:8000/ana-docker/"
export RESTIC_PASSWORD="<client-side-encryption-passphrase>"
restic init
'
Prune ceremony
Same as rest-server-nh3 — prune is blocked by --append-only. Two options, pick one per endpoint:
- Temporary flag flip: edit compose, remove
--append-onlyfromOPTIONS,docker compose up -d, runrestic forget --prunefrom origin hosts, put the flag back,docker compose up -d. Quarterly change. - Second endpoint on a different port: stand up a sibling container (e.g. port
8001) against the same data dir without--append-only, reachable only from a trusted host. Everyday backups still hit:8000.
If you go the second-endpoint route, copy this stack to stacks/rest-server-ana-prune/ with REST_PORT=8001 and --append-only removed from the compose.
Off-site replication
Scheduled on ana-docker (to be written). Mirror destinations live in the sibling subdirs described under "Data layout" above:
# Pull NH3's tree into ana-docker's /mnt/backup/restic/repo/nh3/
rsync -avz --delete admin@10.100.50.50:/volume1/Backup/restic/ /mnt/backup/restic/repo/nh3/
# Pull the ESH rest-server's tree into /mnt/backup/restic/repo/esh/
# (once an ESH rest-server is set up — currently ESH hosts write directly
# to this Anaheim rest-server at ana/esh-*)
# Push our tree (ana/) to NH3 for redundancy in the other direction
rsync -avz --delete /mnt/backup/restic/repo/ana/ admin@10.100.50.50:/volume1/Backup/restic-mirror-ana/
Each sync unidirectional, running in the direction its data flows. Prune runs only at the origin so the mirror shrinks correctly.