Device has been removed from the NH3 site. Drop the homepage card
and the corresponding example in discover-fortigate.sh.
Note left in services.yaml so whoever adds the replacement edge
device knows where the old entry lived.
nh3-dev is the author's active dev workstation at the NH3 site
(nh3-dev.phasefinal.com). 22 GB home with 12 GB of development code,
shell history, dotfiles, Claude Code memory, and scripts that weren't
being captured by any fleet backup.
Writes site-local to the Synology rest-server (10.100.50.50:8000)
matching the nh3-docker pattern.
Excludes trim ~8 GB of regenerable caches + build outputs:
- language toolchains (.cargo, .rustup, .npm, .m2, .gradle, go/pkg, etc.)
- editor caches (.vscode-server, .vscode)
- project build directories (node_modules, __pycache__, .venv, target,
dist, build, .pytest_cache, .tox, .next, .nuxt) via `**/` patterns
- trash / browser caches / steam / snap
Expected first snapshot ~14 GB, incrementals much smaller due to
content-defined dedup across dev code.
README walks through the full setup: resticprofile install, Synology-
side htpasswd addition (via DSM File Station or Container Manager since
the DSM SSH setup is tabled), repo init, profile deploy, timer enable,
first backup. Schedule is nightly 01:00 with systemd's Persistent=true
behavior so missed runs catch up at next boot (workstation is often
off/sleeping at backup time).
README.md fleet-coverage table updated to reflect 6/6 file-level
coverage (ana-docker, ana-ml2, nh3-docker, esh-docker-vm, vm-esh-nas,
nh3-dev).
Closes the last file-level backup gap. Primary ESH home-lab Docker host
— five services with state worth consistent dumps:
- paperless-ngx → external Postgres on 10.0.50.60 (host pg_dump)
- home-assistant → local SQLite ~50MB (host sqlite3 .backup)
- calibre-web-automated → local SQLite (in-container sqlite3)
- pgadmin → local SQLite (host sqlite3)
- uptime-kuma → local SQLite (host sqlite3; container name may vary
after force-recreate, detect by label)
Unique to this host: HA/pgadmin/uptime-kuma images don't bundle sqlite3.
Rather than maintaining custom images, pre-backup.sh runs sqlite3 from
the HOST against the volume bind-mount paths. Requires sqlite3 +
postgresql-client installed on esh-docker-vm.
Cross-site writes to rest-server-ana since ESH has no local rest-server.
NFS mounts (/mnt/{backup,books,compose,documents}) explicitly excluded
— hundreds of GB of NAS-side content backed up at the NAS layer. Also
excludes offen-sidecar buffer volumes (paperless + pgadmin currently
run offen/docker-volume-backup alongside — retire once restic has a
week of clean runs).
Found in audit (non-blocking but noted in README follow-ups):
- paperless-ngx Postgres password is literally "paperless-ng" —
trivially weak, rotate at next opportunity.
The Anaheim NAS exports /mnt/backup with root_squash, so sudo-as-root
on ana-docker becomes nobody on the NFS side and `sudo tee` gets EACCES.
The repo tree is owned by lkraven:lkraven precisely because of this —
all writes go through the lkraven UID.
Drop sudo from the example; plain `>>` append as lkraven works.
Second ESH Docker host — light (filezilla + dockge + agents, /opt/docker
is 56 KB). Cross-site writes to the Anaheim rest-server since ESH has
no local rest-server.
Critical detail: vm-esh-nas NFS-mounts /mnt/{share,music,books,media}
from 10.0.50.50 (~400 TB). Profile's exclude patterns explicitly
reject those paths as a safety net on top of the source list not
including them — a careless future edit to sources can't nuke the
backup repo by pulling in 400 TB.
Sources include /home/ (lkraven/nas/user dirs) in addition to the
usual /opt/docker /etc /root /var/lib/docker/volumes — this host has
multiple user accounts worth preserving dotfiles for.
No pre-backup hook: no relational DBs.
README walks through the full setup: install restic (not present on
this host), add htpasswd entry on rest-server-ana, install creds,
init, install resticprofile, schedule timers. Also flags the 3.8 GB
RAM constraint.
The compressed zip defeated restic's content-defined chunking: each
day's dump looked completely different to restic even when the repo
content barely changed, causing repo growth of ~full dump size (821 MB
at last measurement) every day until forget/prune aged snapshots out.
Uncompressed tar is dedup-friendly. After the first snapshot, daily
incrementals cost only the actual new-data delta — typically a few
MB for an active repo.
Tradeoff: stage file on the client host is ~2-3x the zip size while
the dump is in flight, but that's transient (purged at the start of
each run). Repo-side storage is much smaller over time.
Adds file-level restic for the NH3 Docker VM. Targets the Synology
rest-server (rest-server-nh3 at 10.100.50.50:8000) as user nh3-docker —
site-local writes matching the fleet pattern.
No pre-backup hook needed: none of the stacks on this host (adguard,
dockge, beszel-agent, dozzle-agent, portainer) run relational DBs. The
SQLite state in their named volumes is WAL-mode and restores cleanly
from raw restic capture.
Source paths mirror ana-ml2's template (/opt/docker, /etc, /root,
/var/lib/docker/volumes) with the same exclude list (docker internals,
logs, pids, root's shell/caches). ~1.6 MB of compose files + small
volumes — expected snapshot size a few hundred MB at most, dominated
by any AdGuard query log history.
README walks through reuse (existing repo + htpasswd) vs fresh init
paths, resticprofile install, timer generation, and verification via
the Backrest UI.
Audit surfaced three DB-backed services not being dumped consistently
by the existing pre-backup.sh:
- vaultwarden — migrated to external Postgres on PFI-Postgres
(10.250.50.80); old sqlite block was dumping stale pre-migration
files. Replace with pg_dump against the live database. Requires
postgresql-client on ana-docker and VW_PG* creds in
/etc/restic/dbcreds.env.
- gitea — also on PFI-Postgres; no hook existed at all. Use
`gitea dump` for a single zip that captures DB + repos + config +
LFS + attachments consistently. No explicit creds needed; the
container reads its own GITEA__database__* env.
- openwebui — two local SQLite databases (webui.db + the ChromaDB
vector store). .backup command if sqlite3 is in the image, volume-
level fallback otherwise.
Refactor: each block now logs a WARN on failure instead of aborting the
whole script — partial dumps > no dumps when one service has an issue.
dbcreds.env.example committed as a template; real file goes to
/etc/restic/dbcreds.env root:600 on the host and is never in the repo.
Mattermost retired (user confirmed 2026-04-21); removed from the
pre-backup.sh list and flagged in README's stacks section. Mattermost
container was not running regardless; the audit surfaced that it was
already effectively gone. Compose dir on ana-docker can be removed as
separate cleanup.
First backup run pulled in 9 GB due to /var/lib/docker/volumes/
parakeet_parakeet_cache — Parakeet is the only AI stack on ana-ml2
using a docker named volume for its HF model cache (kokoro, vibevoice,
comfyui, llama-swap, vllm-qwen3 all bind-mount from /tank which is
already outside source paths).
Excluding brings expected snapshot size back to ~100-300 MB.
rest-server-ana README now describes the /mnt/backup/restic/repo/
top-level NFS mount and its three per-site subdirs:
ana/ — live data served by this rest-server (per-host repos +
.htpasswd) — what DATA_DIR points at
esh/ — mirror destination for ESH-site backups (pending)
nh3/ — mirror destination for NH3 Synology's tree (pending)
ana-ml2 README gains a proper "Recreating the repo" section with the
correct /mnt/backup/restic/repo/ana/ana-ml2/ path for wiping the old
repo after a lost passphrase, and two paths for regenerating keys:
- interactive: type a user-generated passphrase at restic's init
prompt, then install it into /etc/restic/password via `cat > file`
+ Ctrl-D (no shell history or transcript exposure)
- scripted: openssl rand -base64 48, passphrase prints once and must
be captured into the password manager immediately
Cross-site replication snippet in rest-server-ana README updated to
use the unified /mnt/backup/restic/repo/{esh,nh3}/ destinations
instead of the earlier restic-mirror-*/ staging paths.
Original used `sudo env \$(cat /etc/restic/restic.env) …` but the `cat`
in command substitution runs as the login user, not root. Since the
env-file is root:600, the substitution silently yielded an empty
RESTIC_REPOSITORY and restic errored with "Please specify repository
location".
Wrap the whole dance in `sudo bash -c "…"` so the env-file read and
the restic call both happen as root.
An ana-ml2 user and repo were created during the original backup
pipeline pass. Reuse keeps snapshot history consolidated and avoids
duplicate infrastructure. Reworks the README to:
- Skip `restic init` (repo exists) and the htpasswd step (user exists)
- Install the two existing secrets (REST URL w/ htpasswd password,
repo passphrase) into /etc/restic/{restic.env,password}
- Verify credentials against the existing repo via `restic snapshots`
Fresh-setup flow retained below as a fallback for zero-state rebuilds.
ana-ml2 is not on any Proxmox hypervisor, so vzdump doesn't touch it.
This closes the biggest single backup gap per the 2026-04-20 pipeline
audit.
Sources: /opt/docker (~110 MB), /etc, /root, /var/lib/docker/volumes.
Excludes /tank/* (model weights — regenerable from Hugging Face and
would blow repo size budget). No pre-backup DB hook — none of the
llama-swap / vllm / comfyui / kokoro / parakeet / vibevoice stacks
use relational databases.
README walks through the one-time setup: rest-server .htpasswd entry,
restic init with fresh passphrase, resticprofile install, systemd timer
generation, verification against the Backrest UI.
services.yaml:
- ANA-Firewall: href + siteMonitor both point at the IP now (was href
to FQDN but siteMonitor to IP — inconsistent)
- PFI-VM-Docker, NH3-SW1, NH3-VM-Docker, ESH-VM-Docker: upgrade from
`ping:` to `siteMonitor:` against the href URL so the up/down dot
reflects whether the web UI actually responds, not just ICMP
dockge canonical:
- icon sh-dockge.png (was si-portainer — wrong project). Applies to all
five fleet Dockge instances once their compose files are redeployed.
Per-group colors investigated but not supported by homepage (only site-
wide `color:` exists); skipped.
Flat-list layout didn't scale well once auto-discovery filled Apps and
Service Networking with a dozen+ cards each. Splits the dashboard:
Main — Monitoring, AI Systems, Apps, Media, Games, UltraSeedbox
Infra — three per-site hardware groups
Plumbing — Service Networking (dockge x5, traefik, adguard, etc.)
Row counts set to 4 columns on dense groups so they render as grids
rather than vertical walls.
Reorganize the gethomepage dashboard from site-based (PFI-ANA, ESH, NH3)
to function-first grouping (Monitoring, AI Systems, Apps, Media, Games,
Infra-<site>, Service Networking). Canonical config now tracked in
configs/homepage/ with Plex/Jellyfin widget keys moved to env
substitution.
Label sweep across fleet compose files:
- beszel, dozzle, backrest -> Monitoring
- rest-server-ana -> Service Networking
Healthcheck fixes (previous wget/curl paths broke on distroless +
--private-repos 401):
- beszel hub: /beszel health --url ...
- beszel agent: /agent health (newly added)
- rest-server: nc -z localhost 8000 (TCP probe)
Group name originally "Wiring / Plumbing" collapsed to single-word group
on homepage's parser; renamed to "Service Networking" everywhere.
Network probes (no web admin, Debian SSH banner, only file-server ports
open) show 10.250.50.50 is vanilla Debian 12 with hand-configured NFS/SMB,
reprovisioned from the original TrueNAS SCALE install. Update stack
comments, README storage notes, Backrest description, proxmox-vms.md
entry for VM 100, and the restic configs intro to match.
Cross-site sync plan simplifies to plain rsync over SSH on both sides —
no appliance-specific tooling needed.
Captures the full workspace state built up to this point:
- CLAUDE.md + README.md describing conventions and the four-host fleet
(ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
- Per-host notes under servers/<host>/ with ssh-target fallback files
and latest system-details snapshots (two in-compose credential leaks
scrubbed; the upstream compose files still need to move those to .env).
- scripts/: server_inspect.sh (read-only remote diagnostic),
refresh-server-info.sh (dir-driven discovery + snapshot capture with
validation warnings), add-host.sh, sync-stacks.sh (pull
compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
- stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
infinity reference. All use the .env-driven + traefik-net + homepage
label pattern.
- configs/restic/ana-docker/: first resticprofile config + pre-backup
hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
for the other three hosts to come.
- docs/pfi/: general infrastructure reference carried over.
- .gitignore excludes .env, stacks-mirror/, and assorted secret/state
filenames to prevent re-leaks on later commits.
vh