stacks/task-board/compose.yaml + .env.example describe the runtime —
image tag is task-board:local (built on the host), 7878 on host maps
to 7878 in container, SQLite lives at /opt/docker/conf/task-board/data/
(bind mount, uid 1000 friendly), homepage auto-card labels under
Toolchain group, on traefik-net like the rest of the fleet.
playbooks/deploy-task-board.yaml is the first real elway playbook —
exercises everything we built tier-1 + tier-2 idempotency for:
- `creates:` on mkdir + first-time clone + compose dir + data dir
- `when:` to chown /opt/docker/build only if it came up root-owned
- `when:` to seed .env only if one doesn't already exist (never
clobbers user edits on rerun)
- `changed_when:` on the `git reset --hard` step so repeat runs
against the same ref report `ok` instead of `changed`
- `changed_when: "false"` on every verify step (they attest, not
change)
- `upload:` with mode for compose.yaml + .env
Post-up the playbook polls /api/health for 30s before handing off to
the verify phase, so verification doesn't race the healthcheck's
start_period. Verify covers: /api/health 200, /api/tasks shape, /mcp
reachable, container on traefik-net.
Prereqs documented in the playbook header: Docker + compose plugin,
traefik-net network, git SSH access to gitea from the target host.
Tier 1 — pre-step skip conditions:
when: <remote shell expr> skip unless expr exits 0
creates: <remote path> skip if path already exists
removes: <remote path> skip if path is already absent
Any of the three saying "skip" marks the step `skipped` and moves on.
Evaluated under bash -c on the remote so `!`, `[[`, pipes etc. behave
consistently regardless of the default remote shell.
Tier 2 — post-step change detection:
changed_when: <remote shell expr>
Evaluated after a successful step. Exit 0 → step counts as `changed`
(default). Exit != 0 → `ok` (ran, nothing actually different).
Without this field, successful steps default to `changed`, matching
Ansible's shell/command defaults. Useful on verify steps:
`changed_when: "false"` reports them as `ok` since they only attest.
Status model moved from pass/fail to four states:
ok / changed / failed / skipped
Summary reports each count; overall outcome is CHANGED if any step
changed, OK if none did, FAILED on any non-skipped failure.
Rerunnable smoke: playbooks/elway-smoke.yaml now proves it. On a
clean target the cold run reports 4 changed, 3 ok. Rerunning with
the same vars reports 2 skipped / 2 changed (upload + log-record
have no idempotency hooks and are always `changed`). Overriding
--var greeting=... re-runs the gated step exactly as intended.
Doc block at the top of the script updated with the new schema
fields and state machine.
`scripts/elway` is a ~600-line Python tool (stdlib + python3-yaml) for
driving one-off ssh commands, ad-hoc file uploads, and YAML playbooks
against a single host. Fills the gap between "single ssh one-liner"
and "reach for Ansible."
Highlights:
- Three invocation modes: --shell, --upload (LOCAL:REMOTE[:MODE]),
and --playbook <path>
- Playbook schema: inline vars, list of steps, optional verify block.
Template via {{ var }}; CLI --var overrides inline defaults
- stop_on_fail global (default on), per-step override. Verify phase
always runs, even after a halt — you see end-state regardless
- Sudo handled once: probes NOPASSWD; if not, prompts locally via
getpass, validates up-front, then feeds via `sudo -S` per step.
Password never written to disk/logs. Upload-with-sudo stages to
/tmp then sudo-mv + sudo-chmod
- SSH connection reuse via ControlMaster (60s persist) keeps
multi-step playbooks responsive (~30ms/step reuse vs ~550ms cold)
- Live interleaved stdout/stderr with per-step prefix and colored
pass/fail summary. --dry-run prints the plan without executing
- Shebang pinned to /usr/bin/python3 to bypass venv-shadowing
when python3-yaml lives in the system site-packages
Smoke test (playbooks/elway-smoke.yaml) covers vars + upload + verify;
drove out a YAML-scalar-coercion bug before first commit (`shell: false`
parsed to Python bool, crashed the templater — now coerced to string
at load time with a clear error on nulls).
The original smoke-test curl used voice="default" which doesn't
exist — the neosun wrapper ships zero preset voices, and the
built-in SFT speakers (中文男/女 etc.) are not surfaced. Calling
/v1/audio/speech with any unregistered voice returns a 400 whose
JSON body curl happily writes into the .wav (124-byte phantom).
Replaced the smoke test with the full clone → synthesize flow and
added a gotchas section covering:
- No default voice; /v1/voices/create is mandatory
- Reference audio ≤30s (frontend asserts; longer clips 500 at
synthesis time, not at upload)
- Providing an explicit transcript beats the auto-ASR fallback
- voice_id is the handle, not name
- Both cosyvoice-v3 and cosyvoice-v2 ship in the image
Also documented streaming: available via /api/tts with stream=true
(~150ms TTFB), NOT on /v1/audio/speech. Clarified field-name
differences between the OpenAI-compat and native endpoints in a
table. No WebSocket / SSE in this wrapper despite upstream support.
The Shadowfita FastAPI wrapper hit two unfixed upstream bugs on the
first real /transcribe call — chunker return-shape mismatch (open
issue #16) and a `torchaudio.tensor` that doesn't exist (open #10).
Rather than babysit someone else's half-tested code, switched to
sherpa-onnx with the prebuilt int8 Parakeet-TDT tarball from k2-fsa,
and wrote our own ~60-line FastAPI wrapper.
Moving parts now owned in-tree:
Dockerfile CUDA 12.8 + cuDNN 9 runtime base, installs
sherpa-onnx==1.12.39+cuda12.cudnn9 + fastapi +
soundfile + libasound2 (sherpa-onnx links to ALSA
at load time even when we never touch a mic).
app.py OfflineRecognizer.from_transducer() once at startup;
/transcribe and /v1/audio/transcriptions both accept
multipart uploads and return {"text": ...}.
entrypoint.sh Idempotent model download to /models on first run
(~400 MB int8 tarball), then exec uvicorn.
Smoke test: 0.wav (bundled in the tarball, The House of the Seven
Gables excerpt) transcribes cleanly in ~1.2s on GPU.
PARAKEET_MODEL_URL in .env lets you swap to the v3 (25-language)
tarball without touching any other files. Wipe *.onnx + tokens.txt
from the models dir and the entrypoint re-downloads.
Selectively include /worktank/<stack>/ subtrees now that comfyui,
parakeet, and cosyvoice place real user state there. Bulk weights,
scratch outputs, and the ~8 GB disposable comfyui run/ venv stay
out — both by the include list being precise and by belt-and-
suspenders exclude patterns.
Added sources:
/worktank/comfyui/basedir/user workflows + per-user settings
/worktank/comfyui/basedir/custom_nodes hand-installed extensions
/worktank/comfyui/basedir/input user-uploaded source images
/worktank/cosyvoice/voices cloned speaker profiles
Belt-and-suspenders excludes (inert under current sources; guards
against a future wholesale-add of /worktank):
/worktank/comfyui/basedir/{models,output,temp}
/worktank/comfyui/run
/worktank/parakeet/models
/worktank/cosyvoice/{input,output}
Verified by `resticprofile backup --dry-run` on irv-ml1 — 1.5 GiB
scanned across all 9 sources, 13 MiB new to the repo. If any bulk
dir had leaked in, the total would be multi-GB.
Also fixed a stale /home row in the README (profile only sources
/home/lkraven; llmuser + sduser are explicitly excluded).
Two new speech stacks on irv-ml1, both on the /worktank/<stack>/
pattern, no tnet (irv-ml1 is local-endpoints-only for now).
parakeet — ASR via Shadowfita/parakeet-tdt-0.6b-v2-fastapi:
- docker buildx git context pinned to SHA 31c5652; no source
vendored. Rebuild on SHA bump.
- GPU-capable FastAPI + Silero VAD + WS streaming.
- API: POST /transcribe, WS /ws/transcribe, GET /healthz. Not the
literal OpenAI `/v1/audio/transcriptions` path — note in README.
- HF cache at /worktank/parakeet/models/ (excluded from restic).
- Build ~158s first time; steady-state start ~40s.
cosyvoice — TTS via neosun/cosyvoice:v1.3.2 shipping
Fun-CosyVoice3-0.5B-2512 (CosyVoice 3, chosen over v2 for the
expanded 5,000-hour instruction-following data covering emotions,
speed, tones, dialects, accents, role-playing; ~150ms streaming
TTFB matches v2). API: /v1/audio/speech (OpenAI drop-in),
/v1/voices/create (cloning), /health.
- Host port 8190 (container 8188; host 8188 already taken by comfyui).
- /worktank/cosyvoice/{voices,input,output}/; voices include in
restic (precious — reproducing a clone needs the original ref
audio), input+output excluded (scratch).
- Model weights (~2-3 GB) live inside image layer; re-download on
tag bump, persist across `compose up -d`.
Both healthy on first deploy.
New stack mirroring PFI convention (stacks/comfyui/) using
mmartial/comfyui-nvidia-docker:ubuntu24_cuda12.8-20260312. Both GPUs
exposed, pinned to CUDA 12.8 to match the host's 570.x driver and the
native cuda-toolkit already in place.
Layout — single tree under /worktank/comfyui/ (462G dedicated, 1%
used pre-deploy):
- basedir/ → /basedir user state (models, workflows, custom_nodes,
input, output); owned 1000:1000 so external
tools can edit workflow JSON directly.
- run/ → /comfy/mnt ComfyUI source + venv + pip cache (~7.8G
after bootstrap). Bind mount instead of
named volume — the image refuses to chown
mounted paths at startup, so keeping this
lkraven-owned avoids the sudo dance.
servers/irv-ml1/README.md refreshed: Docker upgraded to 29.4.1 with
traefik-net in place; dockge + beszel-agent + dozzle-agent already
present; /storetank dropped 92% → 64%; restic coverage to
rest-server-nh3 is operational (not "currently none" as prior text).
VMIDs aren't globally unique across PVE hosts (esh-pve and sfsrv-ana
both have VM 100, etc.). Without namespaces, per-hypervisor backups
collide under the same /vm/<vmid>/ path in the shared datastore.
Changes:
- New Phase 1.4b: create one namespace per hypervisor up front
(pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana).
- Phase 2.1 storage-entry template now lists Namespace as a required
field, set to the hypervisor's own name.
- Critical-note explaining the collision risk so future deployers
don't skip this step.
ACL grants remain at the datastore level; they apply across all child
namespaces so the existing fleet-vzdump token continues to work. Sync
job (Phase 6) preserves namespace tree to PBS-NH3 automatically.
Discovered during Phase 1.3 (mount NAS datastore). The Debian NAS's
/mnt/backup is on ZFS with casesensitivity=insensitive; NFSv4 writes
fail with EACCES even for root with no_root_squash. Known-bad
combination at the ZFS-on-Linux + NFSv4 layer.
Workaround: mount the NFS share with vers=3. PBS's chunk-based
access pattern works fine over NFSv3.
Added a section 0.5 to Phase 0 documenting the issue + both fixes
(quick: use NFSv3; cleaner: create case-sensitive child dataset).
Future deployments against this NAS won't rediscover the same wall.
Adds a new `pinned` group with swap: false (models coexist in VRAM),
exclusive: false (group shares with other groups), persistent: true
(never unload). Each member also gets ttl: 0 so the per-model
idle-timeout can't drop them either — belt + suspenders.
Pair is currently qwen3.5-9b (~6 GB Q4) + qwen3.6-35-a3b (~29 GB Q6).
Plus the 128K KV caches, roughly 50-60 GB VRAM resident. Appropriate
for an A6000/H100-class card; verify fit after deploy.
Committed as a canonical change; push + restart still needed on
ana-ml2.
Current NFS exports on the Debian NAS (10.250.50.50) use root_squash,
which blocks PBS from writing its datastore metadata (chunks, locks,
GC state — all root-owned operations). Rest-server-ana worked around
this by running its container as UID 1000, but PBS's service model
doesn't accommodate that pattern cleanly.
Solution baked into Phase 0: create a dedicated NFS export for the
PBS-ANA datastore subtree, scoped to only the PBS-ANA VM's IP, with
no_root_squash. Bounded exposure (single client), kept in a separate
.exports file so Cockpit's File Sharing module doesn't clobber it.
Flag world-scoped export on /mnt/pve-VMStorage as a non-blocking
hygiene item for a later Cockpit pass.
End-to-end runbook for standing up Proxmox Backup Server across the
fleet. Path A architecture: single primary at ANA, one-way sync to NH3
for disaster recovery. All 5 hypervisors (pfi-pve, nh3-pve, esh-pve,
esh-pve-nas, sfsrv-ana) migrate from local-dump vzdump to PBS-ANA.
Key decisions captured in the runbook:
- PBS in a Debian VM (not LXC) for clean capability model.
- PBS-ANA on pfi-pve, datastore via NFS from 10.250.50.50 —
separates backup data from hypervisor boot disk.
- PBS-NH3 on nh3-pve with local storage (independent failure
domain from ANA).
- Dedicated fleet-vzdump API token; read-only sync token for
PBS-NH3's pull job.
- sfsrv-ana specifically goes from zero backup coverage to full
vzdump coverage in Phase 3.
9 phases, each self-contained with a done-state and rollback
posture. User can stop between phases without leaving the fleet in a
bad state.
STATUS.md: added item 6b tracking this deployment. Original item 6
(cross-site rsync) now scoped to restic-only since PBS handles the
VM-image cross-site redundancy directly.
Forget against an --append-only rest-server fails every night (delete
ops blocked). The resulting daily failure cluttered service status and
logs without ever actually retiring old snapshots. Schedule is now
removed from the forget block in all six profiles; the keep-daily /
keep-weekly / keep-monthly / keep-yearly policy remains so manual
invocations (during prune ceremonies, when --append-only is
temporarily off) honor the intended retention.
Files:
configs/restic/ana-docker/profiles.yaml
configs/restic/ana-ml2/profiles.yaml
configs/restic/nh3-docker/profiles.yaml
configs/restic/esh-docker-vm/profiles.yaml
configs/restic/vm-esh-nas/profiles.yaml
configs/restic/nh3-dev/profiles.yaml
Each file has an inline comment marking why the schedule was dropped
so a future reader doesn't re-add it thinking it was an oversight.
STATUS.md: removed the "install Backrest nightly-restart timer" line
item. User confirmed the UI timeout hits even at startup, so periodic
restart wouldn't actually help. Root cause remains deferred.
BatchMode=yes (which the scripts set) implies strict host key checking
and refuses to prompt — so first-time SSH to a host that isn't in
known_hosts fails with "Host key verification failed". Every new host
we register needs a manual `ssh <host>` round-trip first to store the
key before the refresh scripts can reach it.
accept-new fixes that: unknown hosts are auto-accepted into
known_hosts on first connect; subsequent key CHANGES still fail loudly
(as they should — that'd be a MITM signal).
Matches the pattern already used by deploy-stack.sh.
Affects only refresh-server-info.sh and refresh-proxmox-info.sh;
deploy-stack.sh + sync-stacks.sh use their own targets that may or may
not want the same treatment (leaving alone for now).
Initial framing was wrong. PFI runs these under a managed-hosting
agreement: SSH, OS ops, backups are all PFI's responsibility. Hardware
and data belong to the client.
Changes:
- ssh-target files added for sfsrv-ana (root@10.250.250.115 — same
pattern as other PVE nodes) and sf-ana-container
(lkraven@10.250.150.100 guess, adjust if different user).
- sf-r630 still lacks an ssh-target — the OS-side LAN IP isn't in
FortiGate DHCP (static config somewhere). Will fill in once
identified; README flags that gap.
- READMEs rewritten: dropped "tenant-scoped" / "not SSH-managed"
language, added "client context" section that explains the
managed-hosting relationship. Backup coverage now listed as
planned rather than blocked on tenant coordination.
- CLAUDE.md fleet table: SF rows re-labeled "SureFire client
(PFI-managed)". Placement-rules section updated to note that
SF hosts are first-class PFI-ops targets, just client-owned.
- Memory (project_surefire_tenant.md) rewritten to reflect
managed-services reality + hosts-file entries needed for name
resolution since these aren't in PFI DNS.
Snapshot of what's in place and what's outstanding as of end of
2026-04-20/21 session. Grouped by urgency (red/orange/yellow/green/blue)
so a glance tells you what's next regardless of who's picking it up.
Notable open items:
- Backrest esh-docker-vm URI mismatch (still pointed at NH3 Synology
instead of rest-server-ana)
- ssh-target verification on the 9 newly-added host entries
- Forget schedules need patching (fail nightly against --append-only)
- ~6 secrets captured in this session's transcripts need rotation
- SureFire tenant backup plan pending decision
Lists session milestones (homepage reorg, 6/6 restic coverage, discovery
scripts, CWA migration, 9 host registrations, etc.) and memory
pointers so future sessions have context without re-reading the full
chat log.
Six PFI VMs/LXCs previously known only via proxmox_inspect.sh —
covered by vzdump but not in servers/, so operational context
(roles, backup posture, ssh target) was missing:
pfi-ana-webhost (VMID 110) — web workload
ana-filebot (LXC 112) — file-task automation
pfi-pteradactyl (VMID 107) — Pterodactyl game panel
pfi-tacticalrmm (VMID 111) — TacticalRMM remote-management
pfi-postgres (VMID 105) — shared Postgres (vaultwarden/gitea/
paperless backends)
ana-wg (LXC 113) — WireGuard VPN gateway
Plus three SureFire tenant hosts at the Anaheim colo:
sfsrv-ana — tenant Proxmox hypervisor (10.250.250.115:8006)
sf-ana-container — container workload on that Proxmox
sf-r630 — physical R630 (iDRAC 10.250.250.110 for PFI-side
hardware mgmt; OS is tenant-scoped)
Each server dir has README + ssh-target where applicable. SureFire
entries explicitly document tenancy scope: PFI provides hosting,
SureFire owns the OS; management actions need tenant coordination.
SureFire hosts have no ssh-target by default.
Homepage Infra - ANA gains two new cards:
- SFsrv-ANA (https://10.250.250.115:8006, si-proxmox icon)
- SF-R630-iDRAC (https://10.250.250.110, si-dell icon)
PFI-ANA-ML2 BMC gained an href since it has a usable web UI.
CLAUDE.md fleet table extended with all 9 new rows. Placement-rules
section notes the SureFire tenant boundary.
Memory: new project_surefire_tenant.md so future sessions know sf-*
hosts are tenant-scoped by default.
First real run surfaced 31 gap rows, ~20 of which were noise. These
changes reduce the output to actionable signal.
1. discover-unifi: /ea/devices now filters out
- IPs outside the fleet LAN range (UDM's WAN IP appearing as a
"device", ISP uplink records with public IPs)
- UDM self-records (isConsole=true, or IP matches wans[].ipv4)
- UCI records (UniFi Cable Internet = ISP modem tracking)
LAN filter regex defaults to ^10\. (matches 10.0.0.0/8); override
via UNIFI_LAN_FILTER env var if you run other private ranges.
2. discover-gaps: new --ignore-unifi flag drops rows where the final
SOURCE column starts with "unifi:". Useful for "show me servery
things to manage, not the fleet's network hardware."
3. discover-gaps: known-IP set now pulls IPs from
servers/*/proxmox-details.txt AND servers/*/system-details.txt in
addition to README.md and ssh-target. Consequence: VMs tracked by
proxmox_inspect.sh are automatically counted as known without
needing a separate servers/<vmname>/ dir. Also strips meaningless
addresses (127.*, 0.0.0.0, 169.254.*) so they can't false-positive
a "known" match.
4. MAC normalization: both discover-fortigate and discover-unifi now
emit xx:xx:xx:xx:xx:xx lowercase. Previously FortiGate used colon
format, UniFi used no-separator uppercase — same MAC looked
different per source. Fortigate does tolower() in awk; UniFi uses
a shared jq `norm_mac` function.
Raw dumps of /ea/hosts and /ea/devices surfaced the actual JSON:
- /ea/hosts: LAN IP isn't at top-level ipAddress (that's WAN public);
it's buried in reportedState.ipAddrs[] mixed with WAN + link-local.
Have to pick the first RFC1918 entry that ISN'T also a WAN interface
IP (reportedState.wans[].ipv4). Name/mac/model all live under
reportedState.{hostname,mac,hardware.shortname}.
- /ea/devices: outer records are per-host wrappers; real AP/switch
records are in the nested `devices` array with top-level `ip`, `mac`,
`name`, `model` fields. Previous parser was reading the wrapper and
getting all `-`.
Reorder all TSV outputs so IP is column 1 — makes discover-gaps.sh
work uniformly against both FortiGate and UniFi sources. Sites TSV
dropped its IP slot since sites have no meaningful IP (metadata only).
Verified against the real payloads the user captured: ESH-UDMPM now
surfaces as 10.0.0.1 (LAN) instead of 192.168.200.111 (WAN2, RFC1918
but excluded via the wans cross-check). A sample device record
(E7-ESH-Media at 10.0.250.176) flattens correctly into a single TSV row.
Rewrite to use Ubiquiti's public cloud API at api.ui.com instead of
logging into individual controllers via session cookies. Benefits:
- One API key covers every UniFi OS device on the account (no
per-controller login logic, no cookie jar lifecycle).
- Read-only by design (auth keys are scoped).
- Works across sites transparently.
Three endpoints wired up: hosts (controllers / Cloud Keys), sites,
and devices (APs / switches). Each emits a distinct TSV shape so the
output can be concatenated and still parsed.
`all` mode runs all three and prints section markers on stderr so
the stdout stream stays clean TSV suitable for discover-gaps.sh.
Pagination handled via nextToken. Rate limit not enforced locally;
Ubiquiti documents generous defaults for read endpoints.
Note: Site Manager API (early access) doesn't appear to expose a
connected-client list directly. For endpoint discovery (IP + MAC of
connected clients like laptops, IoT, etc.) we'd still need to hit
each local controller's REST API — follow-up if the infrastructure-
level data isn't enough.
Requires: curl (present), jq (apt install jq).
Bug: FortiOS 7.x ana-gw replied to 'execute dhcp lease-list all' with
"Interface name 'all' does not exist." — my error-pattern grep didn't
include that phrase, so the script thought it got valid data, bailed
out of the retry loop, and handed empty/garbage to the parser, which
produced zero output with no error.
Fix: try the plain `execute dhcp lease-list` form first (works across
versions we've seen), fall back to the `all` variant only if the plain
form returns nothing. Validate acceptance by grepping for an actual
IP-shaped token — the parser needs IPs anyway, so "got real data"
and "has at least one IP" are equivalent conditions.
Device has been removed from the NH3 site. Drop the homepage card
and the corresponding example in discover-fortigate.sh.
Note left in services.yaml so whoever adds the replacement edge
device knows where the old entry lived.
First live run against ana-fw.phasefinal.com surfaced two bugs:
1. Script double-prefixed user@ when the arg already contained it
(e.g. `admin@10.250.250.1` became `admin@admin@10.250.250.1` →
auth prompt loop). Accept either "host" or "user@host" and only
prepend the default user if missing.
2. Parser assumed the wrong output format. Real FortiOS (tested on
7.x) emits:
<prompt> # <iface>
IP MAC-Address Hostname VCI SSID AP SERVER-ID Expiry
10.x.x.x ...
<next-iface>
IP MAC-Address ...
- Interface names are flush-left (no "Interface:" prefix)
- First line has the shell prompt embedded before the iface
- Hostnames don't contain spaces in practice
- 8 columns, not 4; VCI can contain "udhcp 1.32.1" etc.
Rewrote awk to detect interfaces via indentation (flush-left = iface,
indented = header or lease) and extract IP/MAC/Hostname from the
first three tokens of each lease row.
Verified against a captured sample; emits clean TSV.
First real run returned empty and we had no idea why — the script was
silently swallowing stderr via `2>/dev/null`. Remove the suppression
and try both `execute dhcp lease-list all` and the no-arg form, keeping
whichever returns non-error output.
Also emit a clearer diagnostic when both fail, pointing the user at
an interactive SSH to poke at command syntax.
Three scripts that surface hosts on the fleet's networks that aren't
already tracked under servers/*/. Goal: spot servers that need management
coverage (inventory, backup, monitoring) without wandering the LAN by
hand.
discover-fortigate.sh SSH to a FortiGate admin, run
`execute dhcp lease-list all`, emit TSV
(IP, MAC, hostname, vdom, source).
SSH was picked over the REST API for now
because it needs no API-token plumbing. The
parser is defensive about FortiOS output
format drift (multiple VDOM sections,
optional hostname). API variant can replace
it when the extra robustness is worth the
token setup.
discover-unifi.sh Cookie-auth REST call against a UniFi
Controller. Tries /api/auth/login (UniFi OS
/ UDM / Cloud Key Gen2+) first; falls back
to legacy /api/login for self-hosted
controllers. Output is the same TSV shape
as the FortiGate script so the two mix.
Needs UNIFI_USER / UNIFI_PASS env and jq.
discover-gaps.sh Consumes one or more TSVs from the sources
above. Builds the set of managed IPs from
servers/*/ssh-target (plus a grep of README
files for documented IPs) and prints any
discovered IPs not in that set.
Exit code is 1 if gaps found — suitable for
cron alerting.
Common pipeline:
scripts/discover-fortigate.sh ana-fw.phasefinal.com > /tmp/ana.tsv
scripts/discover-fortigate.sh nh3-gw.phasefinal.com > /tmp/nh3.tsv
UNIFI_USER=admin UNIFI_PASS=… scripts/discover-unifi.sh esh-uc.esteban.net > /tmp/esh.tsv
scripts/discover-gaps.sh /tmp/ana.tsv /tmp/nh3.tsv /tmp/esh.tsv
First-time use probably needs:
- SSH access configured to each FortiGate (admin login, key preferred)
- UniFi user with read access (the built-in API read-only role works)
- `jq` installed on the runner (for UniFi script)
nh3-dev is the author's active dev workstation at the NH3 site
(nh3-dev.phasefinal.com). 22 GB home with 12 GB of development code,
shell history, dotfiles, Claude Code memory, and scripts that weren't
being captured by any fleet backup.
Writes site-local to the Synology rest-server (10.100.50.50:8000)
matching the nh3-docker pattern.
Excludes trim ~8 GB of regenerable caches + build outputs:
- language toolchains (.cargo, .rustup, .npm, .m2, .gradle, go/pkg, etc.)
- editor caches (.vscode-server, .vscode)
- project build directories (node_modules, __pycache__, .venv, target,
dist, build, .pytest_cache, .tox, .next, .nuxt) via `**/` patterns
- trash / browser caches / steam / snap
Expected first snapshot ~14 GB, incrementals much smaller due to
content-defined dedup across dev code.
README walks through the full setup: resticprofile install, Synology-
side htpasswd addition (via DSM File Station or Container Manager since
the DSM SSH setup is tabled), repo init, profile deploy, timer enable,
first backup. Schedule is nightly 01:00 with systemd's Persistent=true
behavior so missed runs catch up at next boot (workstation is often
off/sleeping at backup time).
README.md fleet-coverage table updated to reflect 6/6 file-level
coverage (ana-docker, ana-ml2, nh3-docker, esh-docker-vm, vm-esh-nas,
nh3-dev).
Closes the last file-level backup gap. Primary ESH home-lab Docker host
— five services with state worth consistent dumps:
- paperless-ngx → external Postgres on 10.0.50.60 (host pg_dump)
- home-assistant → local SQLite ~50MB (host sqlite3 .backup)
- calibre-web-automated → local SQLite (in-container sqlite3)
- pgadmin → local SQLite (host sqlite3)
- uptime-kuma → local SQLite (host sqlite3; container name may vary
after force-recreate, detect by label)
Unique to this host: HA/pgadmin/uptime-kuma images don't bundle sqlite3.
Rather than maintaining custom images, pre-backup.sh runs sqlite3 from
the HOST against the volume bind-mount paths. Requires sqlite3 +
postgresql-client installed on esh-docker-vm.
Cross-site writes to rest-server-ana since ESH has no local rest-server.
NFS mounts (/mnt/{backup,books,compose,documents}) explicitly excluded
— hundreds of GB of NAS-side content backed up at the NAS layer. Also
excludes offen-sidecar buffer volumes (paperless + pgadmin currently
run offen/docker-volume-backup alongside — retire once restic has a
week of clean runs).
Found in audit (non-blocking but noted in README follow-ups):
- paperless-ngx Postgres password is literally "paperless-ng" —
trivially weak, rotate at next opportunity.
The Anaheim NAS exports /mnt/backup with root_squash, so sudo-as-root
on ana-docker becomes nobody on the NFS side and `sudo tee` gets EACCES.
The repo tree is owned by lkraven:lkraven precisely because of this —
all writes go through the lkraven UID.
Drop sudo from the example; plain `>>` append as lkraven works.
Second ESH Docker host — light (filezilla + dockge + agents, /opt/docker
is 56 KB). Cross-site writes to the Anaheim rest-server since ESH has
no local rest-server.
Critical detail: vm-esh-nas NFS-mounts /mnt/{share,music,books,media}
from 10.0.50.50 (~400 TB). Profile's exclude patterns explicitly
reject those paths as a safety net on top of the source list not
including them — a careless future edit to sources can't nuke the
backup repo by pulling in 400 TB.
Sources include /home/ (lkraven/nas/user dirs) in addition to the
usual /opt/docker /etc /root /var/lib/docker/volumes — this host has
multiple user accounts worth preserving dotfiles for.
No pre-backup hook: no relational DBs.
README walks through the full setup: install restic (not present on
this host), add htpasswd entry on rest-server-ana, install creds,
init, install resticprofile, schedule timers. Also flags the 3.8 GB
RAM constraint.
The compressed zip defeated restic's content-defined chunking: each
day's dump looked completely different to restic even when the repo
content barely changed, causing repo growth of ~full dump size (821 MB
at last measurement) every day until forget/prune aged snapshots out.
Uncompressed tar is dedup-friendly. After the first snapshot, daily
incrementals cost only the actual new-data delta — typically a few
MB for an active repo.
Tradeoff: stage file on the client host is ~2-3x the zip size while
the dump is in flight, but that's transient (purged at the start of
each run). Repo-side storage is much smaller over time.
Adds file-level restic for the NH3 Docker VM. Targets the Synology
rest-server (rest-server-nh3 at 10.100.50.50:8000) as user nh3-docker —
site-local writes matching the fleet pattern.
No pre-backup hook needed: none of the stacks on this host (adguard,
dockge, beszel-agent, dozzle-agent, portainer) run relational DBs. The
SQLite state in their named volumes is WAL-mode and restores cleanly
from raw restic capture.
Source paths mirror ana-ml2's template (/opt/docker, /etc, /root,
/var/lib/docker/volumes) with the same exclude list (docker internals,
logs, pids, root's shell/caches). ~1.6 MB of compose files + small
volumes — expected snapshot size a few hundred MB at most, dominated
by any AdGuard query log history.
README walks through reuse (existing repo + htpasswd) vs fresh init
paths, resticprofile install, timer generation, and verification via
the Backrest UI.
Audit surfaced three DB-backed services not being dumped consistently
by the existing pre-backup.sh:
- vaultwarden — migrated to external Postgres on PFI-Postgres
(10.250.50.80); old sqlite block was dumping stale pre-migration
files. Replace with pg_dump against the live database. Requires
postgresql-client on ana-docker and VW_PG* creds in
/etc/restic/dbcreds.env.
- gitea — also on PFI-Postgres; no hook existed at all. Use
`gitea dump` for a single zip that captures DB + repos + config +
LFS + attachments consistently. No explicit creds needed; the
container reads its own GITEA__database__* env.
- openwebui — two local SQLite databases (webui.db + the ChromaDB
vector store). .backup command if sqlite3 is in the image, volume-
level fallback otherwise.
Refactor: each block now logs a WARN on failure instead of aborting the
whole script — partial dumps > no dumps when one service has an issue.
dbcreds.env.example committed as a template; real file goes to
/etc/restic/dbcreds.env root:600 on the host and is never in the repo.
Mattermost retired (user confirmed 2026-04-21); removed from the
pre-backup.sh list and flagged in README's stacks section. Mattermost
container was not running regardless; the audit surfaced that it was
already effectively gone. Compose dir on ana-docker can be removed as
separate cleanup.
First backup run pulled in 9 GB due to /var/lib/docker/volumes/
parakeet_parakeet_cache — Parakeet is the only AI stack on ana-ml2
using a docker named volume for its HF model cache (kokoro, vibevoice,
comfyui, llama-swap, vllm-qwen3 all bind-mount from /tank which is
already outside source paths).
Excluding brings expected snapshot size back to ~100-300 MB.
rest-server-ana README now describes the /mnt/backup/restic/repo/
top-level NFS mount and its three per-site subdirs:
ana/ — live data served by this rest-server (per-host repos +
.htpasswd) — what DATA_DIR points at
esh/ — mirror destination for ESH-site backups (pending)
nh3/ — mirror destination for NH3 Synology's tree (pending)
ana-ml2 README gains a proper "Recreating the repo" section with the
correct /mnt/backup/restic/repo/ana/ana-ml2/ path for wiping the old
repo after a lost passphrase, and two paths for regenerating keys:
- interactive: type a user-generated passphrase at restic's init
prompt, then install it into /etc/restic/password via `cat > file`
+ Ctrl-D (no shell history or transcript exposure)
- scripted: openssl rand -base64 48, passphrase prints once and must
be captured into the password manager immediately
Cross-site replication snippet in rest-server-ana README updated to
use the unified /mnt/backup/restic/repo/{esh,nh3}/ destinations
instead of the earlier restic-mirror-*/ staging paths.
Original used `sudo env \$(cat /etc/restic/restic.env) …` but the `cat`
in command substitution runs as the login user, not root. Since the
env-file is root:600, the substitution silently yielded an empty
RESTIC_REPOSITORY and restic errored with "Please specify repository
location".
Wrap the whole dance in `sudo bash -c "…"` so the env-file read and
the restic call both happen as root.
An ana-ml2 user and repo were created during the original backup
pipeline pass. Reuse keeps snapshot history consolidated and avoids
duplicate infrastructure. Reworks the README to:
- Skip `restic init` (repo exists) and the htpasswd step (user exists)
- Install the two existing secrets (REST URL w/ htpasswd password,
repo passphrase) into /etc/restic/{restic.env,password}
- Verify credentials against the existing repo via `restic snapshots`
Fresh-setup flow retained below as a fallback for zero-state rebuilds.
ana-ml2 is not on any Proxmox hypervisor, so vzdump doesn't touch it.
This closes the biggest single backup gap per the 2026-04-20 pipeline
audit.
Sources: /opt/docker (~110 MB), /etc, /root, /var/lib/docker/volumes.
Excludes /tank/* (model weights — regenerable from Hugging Face and
would blow repo size budget). No pre-backup DB hook — none of the
llama-swap / vllm / comfyui / kokoro / parakeet / vibevoice stacks
use relational databases.
README walks through the one-time setup: rest-server .htpasswd entry,
restic init with fresh passphrase, resticprofile install, systemd timer
generation, verification against the Backrest UI.
Parallel to refresh-server-info.sh but pipes proxmox_inspect.sh and writes
to servers/<host>/proxmox-details.txt. Same discovery / ssh-target /
validate-only / dry-run behavior.
Fleet-wide `all` matches dir names containing `-pve` (covers *-pve and
*-pve-* so esh-pve-nas is included alongside pfi-pve / nh3-pve / esh-pve).
Explicit names are never filtered — useful for one-off PVE hosts with
non-matching names.
Validation checks the captured snapshot for: truncation, missing PVE
version line, non-Proxmox target, and surfaces backup-coverage "NO"
verdict counts so gaps show up in the validate-only output.
Initial fleet snapshot refreshed.
services.yaml:
- ANA-Firewall: href + siteMonitor both point at the IP now (was href
to FQDN but siteMonitor to IP — inconsistent)
- PFI-VM-Docker, NH3-SW1, NH3-VM-Docker, ESH-VM-Docker: upgrade from
`ping:` to `siteMonitor:` against the href URL so the up/down dot
reflects whether the web UI actually responds, not just ICMP
dockge canonical:
- icon sh-dockge.png (was si-portainer — wrong project). Applies to all
five fleet Dockge instances once their compose files are redeployed.
Per-group colors investigated but not supported by homepage (only site-
wide `color:` exists); skipped.
Flat-list layout didn't scale well once auto-discovery filled Apps and
Service Networking with a dozen+ cards each. Splits the dashboard:
Main — Monitoring, AI Systems, Apps, Media, Games, UltraSeedbox
Infra — three per-site hardware groups
Plumbing — Service Networking (dockge x5, traefik, adguard, etc.)
Row counts set to 4 columns on dense groups so they render as grids
rather than vertical walls.
Closes the last open backup-coverage gap identified in the 2026-04-20
audit. All guests on all four hypervisors are now covered by vzdump
jobs (pfi-pve 11/11, nh3-pve 5/5, esh-pve 3/3, esh-pve-nas 5/5).
VM images are backed up via Proxmox vzdump, but ana-ml2 (bare metal) has
no backup at all, and file-level restic + DB dumps are still missing on
three of four Docker-host VMs. Adds an explicit coverage table to the
README backup section so the gaps don't get overlooked while planning.
Goal (per user): every Docker host + configs + every database, not just
the VM image layer.
vh