docs: STATUS.md — current fleet state + open issues
Snapshot of what's in place and what's outstanding as of end of
2026-04-20/21 session. Grouped by urgency (red/orange/yellow/green/blue)
so a glance tells you what's next regardless of who's picking it up.
Notable open items:
- Backrest esh-docker-vm URI mismatch (still pointed at NH3 Synology
instead of rest-server-ana)
- ssh-target verification on the 9 newly-added host entries
- Forget schedules need patching (fail nightly against --append-only)
- ~6 secrets captured in this session's transcripts need rotation
- SureFire tenant backup plan pending decision
Lists session milestones (homepage reorg, 6/6 restic coverage, discovery
scripts, CWA migration, 9 host registrations, etc.) and memory
pointers so future sessions have context without re-reading the full
chat log.
This commit is contained in:
@@ -0,0 +1,173 @@
|
||||
# Status + Open Issues
|
||||
|
||||
Last updated: 2026-04-21
|
||||
|
||||
Snapshot of fleet state and open work. Refresh this file when a pass of
|
||||
significant work lands — don't let it drift quietly.
|
||||
|
||||
## What's in place
|
||||
|
||||
### Backup coverage
|
||||
|
||||
- **VM-level vzdump:** 24/24 guests across pfi-pve, nh3-pve, esh-pve, esh-pve-nas.
|
||||
- **File-level restic:** 6/6 hosts configured with systemd timers:
|
||||
| Host | Target rest-server | DB hooks |
|
||||
|---|---|---|
|
||||
| ana-docker | rest-server-ana (local) | synapse, seafile, vaultwarden-pg, gitea (native dump), openwebui |
|
||||
| ana-ml2 | rest-server-ana (cross-site) | — (no DBs) |
|
||||
| nh3-docker | rest-server-nh3 (local, Synology) | — |
|
||||
| esh-docker-vm | rest-server-ana (cross-site) | paperless-pg, HA SQLite, CWA SQLite, pgadmin SQLite, uptime-kuma SQLite |
|
||||
| vm-esh-nas | rest-server-ana (cross-site) | — |
|
||||
| nh3-dev (workstation) | rest-server-nh3 (local) | — |
|
||||
|
||||
### Inventory
|
||||
|
||||
- **18 tracked hosts** under `servers/` (5 Docker + 4 Proxmox + 1 workstation +
|
||||
6 PFI VMs + 3 SureFire tenant + 1 NH3-dev workstation... audit the actual count
|
||||
vs this when updating).
|
||||
- **Homepage** at <http://10.0.50.45:5100> shows function-first layout with
|
||||
Main / Infrastructure / Toolchain tabs. Per-group icons, row-count
|
||||
layout, useEqualHeights.
|
||||
- **FortiGate + UniFi discovery** scripts produce TSVs; gap analysis
|
||||
against `servers/*` for unmanaged IPs.
|
||||
|
||||
### Tooling
|
||||
|
||||
- `scripts/server_inspect.sh` + `refresh-server-info.sh` for Docker hosts.
|
||||
- `scripts/proxmox_inspect.sh` + `refresh-proxmox-info.sh` for PVE nodes
|
||||
(reports VMs/LXCs/storage/backup-coverage).
|
||||
- `scripts/discover-fortigate.sh`, `scripts/discover-unifi.sh`,
|
||||
`scripts/discover-gaps.sh` for network-level inventory discovery.
|
||||
- `scripts/deploy-stack.sh` + `scripts/sync-stacks.sh` for compose push/pull.
|
||||
- `scripts/add-host.sh` for new-host registration.
|
||||
|
||||
## Open issues
|
||||
|
||||
### 🟥 Quick wins (do next)
|
||||
|
||||
1. **Fix Backrest's `esh-docker-vm` URI** — Backrest is pointed at
|
||||
`10.100.50.50` (NH3 Synology) for that repo, while the actual backup
|
||||
writes go to `10.250.50.70` (rest-server-ana). Edit `config.json` in
|
||||
the backrest container, swap the host portion. ~5 min.
|
||||
|
||||
2. **Verify SSH on the 9 newly-registered hosts** and pull first snapshots:
|
||||
```
|
||||
scripts/refresh-server-info.sh --validate-only all
|
||||
scripts/refresh-server-info.sh pfi-ana-webhost ana-filebot pfi-pteradactyl \
|
||||
pfi-tacticalrmm pfi-postgres ana-wg
|
||||
```
|
||||
ssh-target files were guessed (`lkraven@` for VMs, `root@` for LXCs) —
|
||||
first run surfaces mismatches.
|
||||
|
||||
3. **Patch out `forget` schedules from all 6 restic profiles.** They fail
|
||||
nightly against `--append-only` rest-servers. One-liner per host:
|
||||
comment out `schedule:` under the `forget` block in profiles.yaml;
|
||||
redeploy; run `resticprofile unschedule forget` on each host. ~15 min.
|
||||
|
||||
4. **Install the Backrest nightly-restart systemd timer** on ana-docker as
|
||||
a safety net for the intermittent UI timeout. Cheap insurance
|
||||
regardless of root cause. ~2 min. Commands in session transcript
|
||||
(search for `backrest-nightly-restart.timer`).
|
||||
|
||||
### 🟧 Real work (dedicated session each)
|
||||
|
||||
5. **Rotate exposed secrets** (captured in session transcripts 2026-04-20/21):
|
||||
- `vaultwarden` Postgres password (on pfi-postgres)
|
||||
- `gitea` Postgres password — currently literally `gitea` (trivially weak)
|
||||
- `paperless-ng` Postgres password — currently literally `paperless-ng` (trivially weak)
|
||||
- `ana-docker` rest-server htpasswd + repo passphrase
|
||||
- `ana-ml2` rest-server htpasswd (repo passphrase rotated during wipe/reinit)
|
||||
- `esh-docker-vm` rest-server htpasswd + repo passphrase
|
||||
|
||||
For each: rotate at the source (DB `ALTER USER …` / `htpasswd` /
|
||||
`restic key add` + `restic key remove`) → update consumer config
|
||||
files (`/etc/restic/restic.env`, `/etc/restic/dbcreds.env`, compose
|
||||
env files, Backrest config.json) → restart consumer services. ~45 min
|
||||
batched.
|
||||
|
||||
6. **Cross-site rsync** between `rest-server-ana` data dir
|
||||
(`/mnt/backup/restic/repo/ana/`) and NH3 Synology data dir. Planned
|
||||
since initial rest-server setup; not built. Needs Synology SSH
|
||||
access first (item 8). ~20 min once access is there.
|
||||
|
||||
7. **SureFire tenant backup plan decision.** Three options documented in
|
||||
`servers/sfsrv-ana/README.md`:
|
||||
- Tenant handles own backups
|
||||
- PFI provides dedicated scoped repo on rest-server-ana
|
||||
- Shared vzdump target
|
||||
|
||||
Blocks any actual SF backup work until the hosting-agreement side of
|
||||
this is clear.
|
||||
|
||||
### 🟨 Prereqs / polish
|
||||
|
||||
8. **Synology SSH setup** — tabled earlier. Unlocks: cross-site rsync
|
||||
(item 6), rest-server-nh3 healthcheck deploy, easier `.htpasswd`
|
||||
edits on the NH3 side.
|
||||
|
||||
9. **`scripts/restic-prune.sh`** — temporarily flip `--append-only` off,
|
||||
run forget + prune across all hosts, flip back on. Needed quarterly
|
||||
for disk hygiene. Not urgent; blocks only the "I need to reclaim
|
||||
disk space now" scenario.
|
||||
|
||||
10. **Retire `offen/docker-volume-backup` sidecars on esh-docker-vm** once
|
||||
restic has ~1 week of clean runs. Paperless + pgadmin currently run
|
||||
offen sidecars that write tarballs to `/mnt/backup/...` redundantly.
|
||||
|
||||
11. **Clean up retired mattermost dir** on ana-docker — compose dir at
|
||||
`/opt/docker/compose/mattermost/` may still linger. Single `rm -rf`
|
||||
when convenient.
|
||||
|
||||
### 🟩 Research / deferred / intermittent
|
||||
|
||||
12. **Backrest UI intermittent timeout.** Backend is confirmed healthy
|
||||
(direct `curl GetConfig` returns in 200ms). UI chokes on the same
|
||||
endpoint. Likely cause: SSE stream wedge in browser, or service
|
||||
worker stale. Workaround: Ctrl+Shift+R. Root cause investigation
|
||||
deferred.
|
||||
|
||||
13. **UniFi controller homepage cards** (ESH-UDMPM at 10.0.0.1, PFI-UDMSE
|
||||
at 10.100.0.1). Auto-discovered via the Site Manager API; not yet
|
||||
linked as homepage cards.
|
||||
|
||||
14. **Prune + credential-rotation scripts** as repeatable tooling (vs
|
||||
per-incident manual work).
|
||||
|
||||
### 🟦 Memory / documentation housekeeping
|
||||
|
||||
15. **`docs/` organization** could use a pass — multiple READMEs and
|
||||
reference files in different spots. Not urgent.
|
||||
|
||||
16. **`STATUS.md` (this file) drift.** Update whenever significant work
|
||||
lands. Consider a `scripts/status-regen.sh` if manual updates
|
||||
slip.
|
||||
|
||||
## Session milestones — 2026-04-20 / 2026-04-21
|
||||
|
||||
- Homepage reorganized to function-first with tabs (Main / Infrastructure / Toolchain).
|
||||
- Per-group icons + equal-height layout + 4-column grids.
|
||||
- Fleet-wide label sweep (function groups, Service Networking renamed from
|
||||
Wiring/Plumbing due to homepage-parser slash bug).
|
||||
- 6 restic clients deployed end-to-end (client creds, repo init, profile
|
||||
install, systemd timers, first backups verified).
|
||||
- 9 hosts registered under `servers/` from gap-analysis (PFI VMs + SureFire
|
||||
tenant, with tenancy awareness).
|
||||
- Discovery scripts (FortiGate SSH + UniFi cloud API + TSV diff vs inventory).
|
||||
- Proxmox inspect script + fleet-wide refresh wrapper.
|
||||
- Calibre-Web-Automated migration replacing calibre + calibre-web pair.
|
||||
- llama-swap Qwen3.6 stock + mradermacher abliterated, 128K ctx.
|
||||
- Mattermost retired (not running, compose dir cleanup pending).
|
||||
- FortiGate 101F at NH3 retired; homepage card removed.
|
||||
|
||||
## Memory pointers (for future Claude sessions)
|
||||
|
||||
Relevant `~/.claude/.../memory/` entries:
|
||||
|
||||
- `server_split.md` — host placement rules
|
||||
- `feedback_ssh_sudo.md` — use `ssh -t` for remote sudo
|
||||
- `feedback_git_autonomous.md` — handle git commits without asking
|
||||
- `feedback_git_commits.md` — no Claude attribution in commit messages
|
||||
- `project_backup_pipeline_gaps.md` — user's explicit goal of "all hosts +
|
||||
configs + DBs backed up"
|
||||
- `project_surefire_tenant.md` — SureFire tenancy boundary awareness
|
||||
- `storage_ana_nas.md` — ana NAS is Debian, not TrueNAS
|
||||
Reference in New Issue
Block a user