From 2cda9fd8a6615ab02594a585088745c8fb1af6bc Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 21 Apr 2026 15:03:56 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20STATUS.md=20=E2=80=94=20current=20fleet?= =?UTF-8?q?=20state=20+=20open=20issues?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Snapshot of what's in place and what's outstanding as of end of 2026-04-20/21 session. Grouped by urgency (red/orange/yellow/green/blue) so a glance tells you what's next regardless of who's picking it up. Notable open items: - Backrest esh-docker-vm URI mismatch (still pointed at NH3 Synology instead of rest-server-ana) - ssh-target verification on the 9 newly-added host entries - Forget schedules need patching (fail nightly against --append-only) - ~6 secrets captured in this session's transcripts need rotation - SureFire tenant backup plan pending decision Lists session milestones (homepage reorg, 6/6 restic coverage, discovery scripts, CWA migration, 9 host registrations, etc.) and memory pointers so future sessions have context without re-reading the full chat log. --- STATUS.md | 173 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 173 insertions(+) create mode 100644 STATUS.md diff --git a/STATUS.md b/STATUS.md new file mode 100644 index 0000000..6b48750 --- /dev/null +++ b/STATUS.md @@ -0,0 +1,173 @@ +# Status + Open Issues + +Last updated: 2026-04-21 + +Snapshot of fleet state and open work. Refresh this file when a pass of +significant work lands — don't let it drift quietly. + +## What's in place + +### Backup coverage + +- **VM-level vzdump:** 24/24 guests across pfi-pve, nh3-pve, esh-pve, esh-pve-nas. +- **File-level restic:** 6/6 hosts configured with systemd timers: + | Host | Target rest-server | DB hooks | + |---|---|---| + | ana-docker | rest-server-ana (local) | synapse, seafile, vaultwarden-pg, gitea (native dump), openwebui | + | ana-ml2 | rest-server-ana (cross-site) | — (no DBs) | + | nh3-docker | rest-server-nh3 (local, Synology) | — | + | esh-docker-vm | rest-server-ana (cross-site) | paperless-pg, HA SQLite, CWA SQLite, pgadmin SQLite, uptime-kuma SQLite | + | vm-esh-nas | rest-server-ana (cross-site) | — | + | nh3-dev (workstation) | rest-server-nh3 (local) | — | + +### Inventory + +- **18 tracked hosts** under `servers/` (5 Docker + 4 Proxmox + 1 workstation + + 6 PFI VMs + 3 SureFire tenant + 1 NH3-dev workstation... audit the actual count + vs this when updating). +- **Homepage** at shows function-first layout with + Main / Infrastructure / Toolchain tabs. Per-group icons, row-count + layout, useEqualHeights. +- **FortiGate + UniFi discovery** scripts produce TSVs; gap analysis + against `servers/*` for unmanaged IPs. + +### Tooling + +- `scripts/server_inspect.sh` + `refresh-server-info.sh` for Docker hosts. +- `scripts/proxmox_inspect.sh` + `refresh-proxmox-info.sh` for PVE nodes + (reports VMs/LXCs/storage/backup-coverage). +- `scripts/discover-fortigate.sh`, `scripts/discover-unifi.sh`, + `scripts/discover-gaps.sh` for network-level inventory discovery. +- `scripts/deploy-stack.sh` + `scripts/sync-stacks.sh` for compose push/pull. +- `scripts/add-host.sh` for new-host registration. + +## Open issues + +### 🟥 Quick wins (do next) + +1. **Fix Backrest's `esh-docker-vm` URI** — Backrest is pointed at + `10.100.50.50` (NH3 Synology) for that repo, while the actual backup + writes go to `10.250.50.70` (rest-server-ana). Edit `config.json` in + the backrest container, swap the host portion. ~5 min. + +2. **Verify SSH on the 9 newly-registered hosts** and pull first snapshots: + ``` + scripts/refresh-server-info.sh --validate-only all + scripts/refresh-server-info.sh pfi-ana-webhost ana-filebot pfi-pteradactyl \ + pfi-tacticalrmm pfi-postgres ana-wg + ``` + ssh-target files were guessed (`lkraven@` for VMs, `root@` for LXCs) — + first run surfaces mismatches. + +3. **Patch out `forget` schedules from all 6 restic profiles.** They fail + nightly against `--append-only` rest-servers. One-liner per host: + comment out `schedule:` under the `forget` block in profiles.yaml; + redeploy; run `resticprofile unschedule forget` on each host. ~15 min. + +4. **Install the Backrest nightly-restart systemd timer** on ana-docker as + a safety net for the intermittent UI timeout. Cheap insurance + regardless of root cause. ~2 min. Commands in session transcript + (search for `backrest-nightly-restart.timer`). + +### 🟧 Real work (dedicated session each) + +5. **Rotate exposed secrets** (captured in session transcripts 2026-04-20/21): + - `vaultwarden` Postgres password (on pfi-postgres) + - `gitea` Postgres password — currently literally `gitea` (trivially weak) + - `paperless-ng` Postgres password — currently literally `paperless-ng` (trivially weak) + - `ana-docker` rest-server htpasswd + repo passphrase + - `ana-ml2` rest-server htpasswd (repo passphrase rotated during wipe/reinit) + - `esh-docker-vm` rest-server htpasswd + repo passphrase + + For each: rotate at the source (DB `ALTER USER …` / `htpasswd` / + `restic key add` + `restic key remove`) → update consumer config + files (`/etc/restic/restic.env`, `/etc/restic/dbcreds.env`, compose + env files, Backrest config.json) → restart consumer services. ~45 min + batched. + +6. **Cross-site rsync** between `rest-server-ana` data dir + (`/mnt/backup/restic/repo/ana/`) and NH3 Synology data dir. Planned + since initial rest-server setup; not built. Needs Synology SSH + access first (item 8). ~20 min once access is there. + +7. **SureFire tenant backup plan decision.** Three options documented in + `servers/sfsrv-ana/README.md`: + - Tenant handles own backups + - PFI provides dedicated scoped repo on rest-server-ana + - Shared vzdump target + + Blocks any actual SF backup work until the hosting-agreement side of + this is clear. + +### 🟨 Prereqs / polish + +8. **Synology SSH setup** — tabled earlier. Unlocks: cross-site rsync + (item 6), rest-server-nh3 healthcheck deploy, easier `.htpasswd` + edits on the NH3 side. + +9. **`scripts/restic-prune.sh`** — temporarily flip `--append-only` off, + run forget + prune across all hosts, flip back on. Needed quarterly + for disk hygiene. Not urgent; blocks only the "I need to reclaim + disk space now" scenario. + +10. **Retire `offen/docker-volume-backup` sidecars on esh-docker-vm** once + restic has ~1 week of clean runs. Paperless + pgadmin currently run + offen sidecars that write tarballs to `/mnt/backup/...` redundantly. + +11. **Clean up retired mattermost dir** on ana-docker — compose dir at + `/opt/docker/compose/mattermost/` may still linger. Single `rm -rf` + when convenient. + +### 🟩 Research / deferred / intermittent + +12. **Backrest UI intermittent timeout.** Backend is confirmed healthy + (direct `curl GetConfig` returns in 200ms). UI chokes on the same + endpoint. Likely cause: SSE stream wedge in browser, or service + worker stale. Workaround: Ctrl+Shift+R. Root cause investigation + deferred. + +13. **UniFi controller homepage cards** (ESH-UDMPM at 10.0.0.1, PFI-UDMSE + at 10.100.0.1). Auto-discovered via the Site Manager API; not yet + linked as homepage cards. + +14. **Prune + credential-rotation scripts** as repeatable tooling (vs + per-incident manual work). + +### 🟦 Memory / documentation housekeeping + +15. **`docs/` organization** could use a pass — multiple READMEs and + reference files in different spots. Not urgent. + +16. **`STATUS.md` (this file) drift.** Update whenever significant work + lands. Consider a `scripts/status-regen.sh` if manual updates + slip. + +## Session milestones — 2026-04-20 / 2026-04-21 + +- Homepage reorganized to function-first with tabs (Main / Infrastructure / Toolchain). +- Per-group icons + equal-height layout + 4-column grids. +- Fleet-wide label sweep (function groups, Service Networking renamed from + Wiring/Plumbing due to homepage-parser slash bug). +- 6 restic clients deployed end-to-end (client creds, repo init, profile + install, systemd timers, first backups verified). +- 9 hosts registered under `servers/` from gap-analysis (PFI VMs + SureFire + tenant, with tenancy awareness). +- Discovery scripts (FortiGate SSH + UniFi cloud API + TSV diff vs inventory). +- Proxmox inspect script + fleet-wide refresh wrapper. +- Calibre-Web-Automated migration replacing calibre + calibre-web pair. +- llama-swap Qwen3.6 stock + mradermacher abliterated, 128K ctx. +- Mattermost retired (not running, compose dir cleanup pending). +- FortiGate 101F at NH3 retired; homepage card removed. + +## Memory pointers (for future Claude sessions) + +Relevant `~/.claude/.../memory/` entries: + +- `server_split.md` — host placement rules +- `feedback_ssh_sudo.md` — use `ssh -t` for remote sudo +- `feedback_git_autonomous.md` — handle git commits without asking +- `feedback_git_commits.md` — no Claude attribution in commit messages +- `project_backup_pipeline_gaps.md` — user's explicit goal of "all hosts + + configs + DBs backed up" +- `project_surefire_tenant.md` — SureFire tenancy boundary awareness +- `storage_ana_nas.md` — ana NAS is Debian, not TrueNAS