esh-pfi-infrastructure/STATUS.md

Status + Open Issues

Last updated: 2026-04-24

Snapshot of fleet state and open work. Refresh this file when a pass of significant work lands — don't let it drift quietly.

New session? Read docs/orientation.md first.

What's in place

Backup coverage (2-layer, fully operational)

• PBS fleet-wide. All 5 hypervisors (pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) → pbs-ana (primary, VM on pfi-pve with NFSv3 datastore on ana-nas). Per-hypervisor namespaces to avoid VMID collisions. DR mirror at pbs-nh3 (VM on nh3-pve with NFSv3 datastore on nh3-nas Synology) pulls daily at 06:00. Verify jobs armed on both sides.
• restic 8/8 hosts with resticprofile + systemd timers at 01:00:

  Host	Target rest-server	DB hooks
  ana-docker	rest-server-ana (local)	synapse, seafile, vaultwarden-pg, gitea (native dump), openwebui
  ana-ml2	rest-server-ana (cross-site)	— (no DBs)
  nh3-docker	rest-server-nh3 (local, Synology)	—
  esh-docker-vm	rest-server-ana (cross-site)	paperless-pg, HA SQLite, CWA SQLite, pgadmin SQLite, uptime-kuma SQLite
  vm-esh-nas	rest-server-ana (cross-site)	—
  nh3-dev (workstation)	rest-server-nh3 (local)	—
  irv-ml1	rest-server-nh3 (via WG)	—
  esh-vm-db	rest-server-ana (cross-site)	pg_dumpall, mongodump

• Cross-site restic rsync. ana-nas → nh3-nas at 04:00 daily (mirrors rest-server-ana data). nh3-nas → ana-nas at 05:00 daily (mirrors rest-server-nh3 data, runs as root since DSM writes files as admin mode 400). Tracked at configs/rsync/{ana-nas-to-nh3,nh3-nas-to-ana}/.

Inventory

• ~22 tracked hosts under servers/ across ANA / NH3 / ESH / IRV sites. SSH config aliases in ~/.ssh/config for every host — ssh <name> just works.
• Homepage at http://10.0.50.45:5100 — function-first layout (Main / Infrastructure / Toolchain tabs), per-group icons, four Infra sub-groups (ANA / NH3 / IRV / ESH). Docker auto-discovery active for 5 hosts (ana-docker, ana-ml2, nh3-docker, esh-docker-vm, irv-ml1).
• FortiGate + UniFi discovery scripts produce TSVs; gap analysis against servers/*/ for unmanaged IPs.

Architecture decisions (durable)

• DB data on local disk, not NFS. pfi-postgres migrated 2026-04-23; fstab + NFS mount removed 2026-04-24 (pfi-postgres now fully decoupled from ana-nas). Mongo on esh-vm-db confirmed already local. Removes the biggest ana-nas blast-radius risk. Memory: project_db_migrate_off_nfs.md.
• Backups must not risk production. Rule adopted after 2026-04-23 ana-nas self-backup crash. Memory: feedback_backups_must_not_risk_production.md.
• offen sidecars retired fleet-wide 2026-04-23. restic covers equivalent scope; reclaimed 16 GB of redundant tarballs on /mnt/backup/docker/esh-vm-docker/.
• Gitea remote for this repo (2026-04-23). origin is vh/esh-pfi-infrastructure on gitea.phasefinal.com. Also: vh/task-board (new 2026-04-24) hosts a Claude Code plugin + marketplace serving the assistant task-state dashboard.
• Prefer elway for multi-step SSH work (2026-04-24). The new scripts/elway mini-playbook runner replaces chained ssh -t sudo … commands — handles sudo once up front, structured reporting, idempotency (creates/when/changed_when). Memory: feedback_use_elway.md; template playbook: playbooks/elway-smoke.yaml.

Tooling

• scripts/server_inspect.sh + refresh-server-info.sh for Docker hosts.
• scripts/proxmox_inspect.sh + refresh-proxmox-info.sh for PVE nodes (reports VMs/LXCs/storage/backup-coverage).
• scripts/discover-fortigate.sh, scripts/discover-unifi.sh, scripts/discover-gaps.sh for network-level inventory discovery.
• scripts/deploy-stack.sh + scripts/sync-stacks.sh for compose push/pull.
• scripts/add-host.sh for new-host registration.
• scripts/elway (2026-04-24) — mini-ansible playbook runner over SSH. Playbooks live under playbooks/. Tier 1 (creates/when) + tier 2 (changed_when) idempotency; handlers + register + multi-host fan-out deferred as gitea issues #3–#5.
• tea CLI at /usr/local/bin/tea — gitea CLI, already logged in as vh. Use for issue / PR work instead of inventing URLs.

Open issues

🟥 Blocked — irv-ml1 stalled after CUDA driver upgrade (2026-04-25)

Driver bump on irv-ml1 (570.124.06 → 595.58.03 via NVIDIA's CUDA APT repo) was attempted to unblock Kokoro's GPU image (which requires CUDA ≥ 12.9; old driver capped at 12.8). The host did not come up cleanly after the post-install reboot. Stop touching irv-ml1 until this is resolved.

What was attempted: scripts/upgrade-irv-ml1-cuda.sh phase1 on 2026-04-25. Phase 1 is supposed to: snapshot current nvidia packages → stop GPU containers → apt install cuda-drivers → DKMS rebuild → reboot. Whether all those steps actually completed before the unclean boot is unknown — the snapshot file at /tmp/nvidia-pre-upgrade.txt on irv-ml1 (if it survived) will say.

Recovery options (in increasing invasiveness):

1. Console diagnose first. SSH may or may not work depending on the failure mode. Check via console (irv-ml1 is bare-metal, has physical access via Lenovo IPMI). Look at journalctl -b -1 from the previous boot to see what failed: DKMS build failure, kernel module load error, nvidia-container-toolkit init regression, etc.

2. Rollback the driver if the host is reachable but GPU is broken:

   scripts/upgrade-irv-ml1-cuda.sh rollback
   

   This reinstalls the snapshot from /tmp/nvidia-pre-upgrade.txt with --allow-downgrades and reboots. Restores 570.124.06.

3. Manual driver pin if the script can't recover:

   ssh irv-ml1 sudo apt install --allow-downgrades \
     nvidia-driver=570.124.06-1 nvidia-driver-cuda=570.124.06-1 \
     nvidia-driver-libs=570.124.06-1 nvidia-kernel-open-dkms=570.124.06-1
   ssh irv-ml1 sudo apt-mark hold nvidia-driver nvidia-driver-cuda \
     nvidia-driver-libs nvidia-kernel-open-dkms
   ssh irv-ml1 sudo systemctl reboot
   

4. Try a different driver version if 595 is genuinely incompatible with kernel 6.1.0-37: 580 series may build cleanly. Check via apt-cache madison cuda-drivers after recovery.

Once irv-ml1 is healthy again, deferred work to revisit:

• Kokoro GPU variant: deployed CPU works as a fallback. Switch .env to KOKORO_VARIANT=gpu / KOKORO_USE_GPU=true / KOKORO_GPU_DEVICES=0 and redeploy via scripts/elway irv-ml1 --playbook playbooks/deploy-kokoro.yaml.
• VibeVoice 1.5B deploy (long-form / multi-speaker dialogue): scripts/elway irv-ml1 --playbook playbooks/deploy-vibevoice.yaml. Stack files at stacks/vibevoice/. ~12 min build + ~7 GB model.
• Chatterbox Turbo deploy (low-latency English voice cloning): scripts/elway irv-ml1 --playbook playbooks/deploy-chatterbox.yaml. Stack files at stacks/chatterbox/. ~10 min build + ~6 GB model.
• Verify the existing 5 GPU stacks still work post-upgrade: comfyui, cosyvoice, qwen3-tts, index-tts, parakeet. They were all Up + healthy before the driver attempt. Smoke-test each.
• Re-evaluate Kokoro variant default after recovery. If a 595+ series driver doesn't end up working, the .env.example default of KOKORO_VARIANT=cpu is correct as-is. If a working GPU driver lands, flip the default to gpu for future deploys.

Don't run on irv-ml1 until this is closed:

• Any of the three deploy playbooks above
• Any host-state-mutating playbook against irv-ml1
• The driver upgrade script itself unless intentionally retrying

Once recovered, update this section with what fixed it and remove the 🟥 marker.

🟥 Quick wins (do next)

1. Fix Backrest's esh-docker-vm URI — done.

2. Push SSH keys + pull snapshots for 6 unrefreshed hosts — done 2026-04-23. 4 VMs (pfi-ana-webhost, pfi-pteradactyl, pfi-tacticalrmm, pfi-postgres) took the workstation key via ssh-copy-id with lkraven@. 2 LXCs (ana-filebot, ana-wg) needed key installed via pct push from pfi-pve because PermitRootLogin prohibit-password blocked ssh-copy-id. Minor: pfi-pteradactyl's server_inspect Docker section is blank because lkraven is not in the docker group there — sudo usermod -aG docker lkraven + re-login to fix.

3. Patch out forget schedules from all 6 restic profiles — done.

3b. Discover sf-r630 OS-side IP — resolved 2026-04-23: the R630 with iDRAC 10.250.250.110 is the same physical box that runs sfsrv-ana (Proxmox VE at 10.250.250.115). No separate OS IP to find. servers/sf-r630/ now clarifies it as the hardware/BMC-only inventory entry; servers/sfsrv-ana/ is the OS view. servers/ana-ml2/README.md updated with its own distinct BMC IP (10.250.250.50, Supermicro) to prevent future confusion between the two physical chassis.

4. (removed — see note above)

🟧 Real work (dedicated session each)

4b. Migrate DB data directories off NFS onto local VM disk. — done 2026-04-23/24. Postgres on pfi-postgres migrated 04-23; mongodb on esh-vm-db confirmed already local (and serves zero user data in practice — paperless uses Postgres 15 on the same VM). fstab entry + NFS mount for /mnt/db on pfi-postgres removed 2026-04-24 via playbooks/decouple-pfi-postgres-from-ana-nas.yaml. pfi-postgres has zero remaining dependency on ana-nas. Only residue: cold archive of pre-migration data still on ana-nas (/mnt/db/pfi-*); harmless, can sit indefinitely.

5. Rotate exposed secrets — done 2026-04-23. All six rotated: vaultwarden/gitea/paperless-ng Postgres passwords (hardcoded compose.yaml literals moved to gitignored .env files in the process), ana-docker + ana-ml2 + esh-docker-vm rest-server htpasswd entries, and rest-server repo passphrases for ana-docker + esh-docker-vm (ana-ml2's was already rotated during prior wipe+reinit). Also discovered along the way: paperless uses a separate esh-vm-db VM (10.0.50.60), not pfi-postgres as the old notes implied.

6. Cross-site rsync — done 2026-04-23, both directions.

   • ana → nh3 (04:00 daily): ana-nas:/mnt/backup/restic/repo/ana/ → nh3-nas:/volume1/Backup/restic-ana-mirror/. Runs on ana-nas as lkraven. 15.6 GB initial sync completed 07:18 UTC.
   • nh3 → ana (05:00 daily): nh3-nas:/volume1/Backup/restic/ → ana-nas:/mnt/backup/restic-nh3-mirror/. Runs on nh3-nas as root (rest-server-nh3 container writes mode-400 files; only root can read them on Synology). 2.31 GB initial sync completed 07:42 UTC.
   • Tracked at configs/rsync/ana-nas-to-nh3/ and configs/rsync/nh3-nas-to-ana/ respectively.

6b. PBS deployment across the fleet. Runbook at docs/runbooks/pbs-deployment.md. Phases 0–6 done (2026-04-22): PBS-ANA on pfi-pve with NFS datastore on Ana NAS (NFSv3 for ZFS case-insensitivity), PBS-NH3 on nh3-pve with NFS datastore on NH3 Synology (NFSv3 + Linux-mode POSIX 777 to avoid syno_acl interference), per-hypervisor namespaces, API tokens, verify jobs on both, one-way sync ANA → NH3 at 06:00 daily, and all 5 hypervisors (pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) onboarded to PBS-ANA. Remaining phases: - Phase 7-8 — one week burn-in, then retire legacy vzdump targets on each hypervisor (keep until 2026-04-29 earliest) - Phase 9 — homepage cards for PBS-ANA + PBS-NH3 web UIs, memory/doc updates, servers/pbs-ana + servers/pbs-nh3 dirs

7. SureFire tenant backup plan decision — resolved 2026-04-23 by folding sfsrv-ana into the fleet-wide PBS deployment (dedicated sfsrv-ana namespace on PBS-ANA, replicates to PBS-NH3 via the same sync job as the rest of the fleet). Hosting-agreement option chosen: PFI provides backup coverage as part of managed hosting.

🟨 Prereqs / polish

8. Synology SSH setup — done 2026-04-22. Dedicated syncuser account (admin-group membership) with key auth, registered as servers/nh3-nas/, reachable as ssh nh3-nas. Unlocks cross-site rsync (item 6), rest-server-nh3 healthcheck deploy, and .htpasswd edits on the NH3 side.

9. scripts/restic-prune.sh — done 2026-04-24. Quarterly disk-hygiene tool. Drops --append-only on the rest-server (via a temporary docker-compose.override.yaml — never edits the canonical compose), runs resticprofile forget --prune --verbose on each client, restores --append-only (with trap so it runs even on partial failure). ANA side fully automated (5 clients); NH3 side prints a manual ceremony because DSM Container Manager + sudo on syncuser aren't cleanly scriptable from this workstation. Run with scripts/restic-prune.sh ana|nh3|all, optionally --dry-run.

10. Retire offen/docker-volume-backup sidecars — done 2026-04-23. Removed from paperless-ngx and pgadmin composes on esh-docker-vm (only hosts in the fleet that had them). 16 GB of orphan tarballs at /mnt/backup/docker/esh-vm-docker/ reclaimed. Restic coverage verified equivalent: /var/lib/docker/volumes in source + pre-backup hook handles the DB dumps for paperless (against esh-vm-db), HA, pgadmin, CWA, uptime-kuma. Obsolete *_offen_backup_data exclude also removed from the restic profile.

11. Clean up retired mattermost dir on ana-docker — done (verified 2026-04-24: /opt/docker/compose/mattermost/ does not exist; no mattermost containers anywhere on the host).

🟩 Research / deferred / intermittent

12. Backrest UI intermittent timeout. Backend is confirmed healthy (direct curl GetConfig returns in 200ms). UI chokes on the same endpoint. Likely cause: SSE stream wedge in browser, or service worker stale. Workaround: Ctrl+Shift+R. Root cause investigation deferred.

13. UniFi controller homepage cards — done 2026-04-24. Both cards added to configs/homepage/services.yaml and pushed to esh-docker-vm: PFI-UDMSE (10.100.0.1, UDM Pro SE) under Infra - NH3 as the new edge device replacing the retired Fortigate 101F; ESH-UDMPM (10.0.0.1, UDM Pro Max) under Infra - ESH. Icon si-ubiquiti.

14. Prune + credential-rotation scripts as repeatable tooling (vs per-incident manual work).

🟦 Memory / documentation housekeeping

15. docs/ organization — done 2026-04-24. First-pass landed earlier (misfiled tea-*.sha256 removed; docs/README.md nav map added). Second pass landed same day: stripped the broken YAML frontmatter from both VM-102 Matrix docs (the path: values pointed at docs/pfi-ana/... which doesn't exist in this repo, and no toolchain consumed the metadata); deleted pfi/chromadb-setup.md (referenced configs/pfi-ana/... and scripts/setup-chromadb.sh, both nonexistent — deployment is long done and the operational truth lives in docker-stack.md). Kept the two VM-102 docs separate by design (each is right-sized; a merge would push past the 500-line guideline in the README).

16. STATUS.md drift discipline — done 2026-04-24. Refreshed with the post-tooling-day session work; status-regen.sh idea dropped — STATUS.md is intentionally narrative, not derivable from git/code, so auto-regen would lose information. Discipline rule: refresh STATUS.md at the end of any session where a 🟧 or 🟥 item closes, or three+ smaller items land.

Active migration — docker.io 20.10 → docker-ce 29.x

Bookworm's Debian-packaged docker.io is pinned at 20.10.24 with client/daemon API 1.41. Newer compose clients (1.52+) refuse to talk to it ("client version 1.52 is too new" during docker compose build). The migration moves each host to Docker's official docker-ce APT repo (29.x). Playbook: playbooks/upgrade-docker-ce.yaml.

Order (least → most blast radius):

1. nh3-docker — done 2026-04-24. docker-ce 29.4.1, two stacks (portainer agent + portainer) restarted clean. Debug surfaced two unit-file gotchas now baked into the playbook:
   • /etc/systemd/system/docker.service.d/override.conf from the docker.io era hardcoded /usr/sbin/dockerd (docker-ce installs at /usr/bin/dockerd) → daemon failed status=203/EXEC.
   • The shipped docker-ce unit's ExecStart=dockerd -H fd:// … conflicts with daemon.json hosts: (defined for the 0.0.0.0:2375 homepage-discovery binding) → "conflicting host options". Playbook now backs up any existing override, probes daemon.json for hosts:, and installs an override that strips -H when needed.
2. esh-docker-vm — pending. Run from your TTY: scripts/elway esh-docker-vm --playbook playbooks/upgrade-docker-ce.yaml
3. ana-docker — pending. Largest blast radius (homepage, miniflux, news-digest, synapse, seafile, vaultwarden, gitea, openwebui, …). Verify each card / service after.

Also during the nh3-docker debug: configs/homepage/docker.yaml commented out the irv-ml1-docker provider — the 20s-per-poll ETIMEDOUTs from the stalled host were drowning homepage's logs and appeared to block ana-pfi-docker discovery (News group's Miniflux card wouldn't render until removal). Re-enable when irv-ml1 is back.

Session milestones — 2026-04-24 (the "tooling day" + housekeeping pm)

Morning / early afternoon — the original tooling day:

• irv-ml1 AI stacks deployed as Docker: ComfyUI (port 8188, host- writable workflows), Parakeet ASR (port 8765, rewritten on sherpa-onnx after the Shadowfita FastAPI wrapper hit two unfixed upstream bugs), CosyVoice 3 (port 8190, GLaDOS voice cloned + live).
• irv-ml1 restic profile extended to cover /worktank/{comfyui/basedir/{user,custom_nodes,input},cosyvoice/voices}; bulk weights + scratch dirs stay excluded.
• scripts/elway shipped — ~800-line Python playbook runner with tier 1 + tier 2 idempotency; handlers / register / multi-host / content-hash are deferred (gitea #3–#6).
• task-board built end-to-end (separate repo, vh/task-board on gitea) and shipped as a Claude Code plugin. Green/red cards per session via UserPromptSubmit + Stop hooks; four MCP tools expose explicit activity tracking.
• pfi-postgres NFS decoupling finished item 4b — zero residual dependency on ana-nas for that VM.

Late afternoon / evening — task-board iteration + STATUS sweep:

• task-board v0.1.1 → v0.1.3 shipped over four iterations on the live ANA deployment:
  • v0.1.1 — hooks parse Claude Code's stdin JSON for session_id and append a short suffix when TASK_BOARD_SESSION isn't set, so two sessions in one project no longer collide on a single card. Dormant transition preserves cumulative idle time (state_entered_at = last_update_at instead of now).
  • v0.1.2 — favicon (3-column SVG in active/waiting/dormant state colors); served at /static/favicon.svg with a <link rel="icon"> and a /favicon.ico route returning the same SVG. Followup fix for an XML-illegal -- in a comment.
  • v0.1.2-followup — UI live-duration ticker bumped from 5 s → 1 s (humanDuration floors to integer seconds; cheap render).
  • v0.1.3 — case-insensitive session names. sessions.name COLLATE NOCASE; real ALTER migration (not a wipe) — keeps earliest- created_at row as canonical, reassigns child comments. Write path canonicalizes session label before inserting comments. Read path uses COLLATE NOCASE for safety on external API callers.
• Parakeet (irv-ml1) healthcheck fix — image ships wget not curl; healthcheck swap, 2,190 failing checks → healthy.
• AIPA-MCP project session label fix — set TASK_BOARD_SESSION= Architect in .claude/settings.json, updated the project's CLAUDE.md to specify session="Architect" for explicit MCP calls, and renamed the existing AIPA-MCP card → Architect in the live SQLite (1 session row + 25 comment rows preserved).
• STATUS items 11 / 13 / 15 / 16 closed. Mattermost dir gone on ana-docker (verified); UniFi UDM cards added to homepage (PFI-UDMSE for NH3 edge replacing the retired Fortigate 101F, ESH-UDMPM for ESH); docs nav map added + stale chromadb-setup.md removed + VM-102 Matrix docs frontmatter stripped; STATUS.md discipline rule recorded.

Session milestones — 2026-04-20 / 2026-04-21

• Homepage reorganized to function-first with tabs (Main / Infrastructure / Toolchain).
• Per-group icons + equal-height layout + 4-column grids.
• Fleet-wide label sweep (function groups, Service Networking renamed from Wiring/Plumbing due to homepage-parser slash bug).
• 6 restic clients deployed end-to-end (client creds, repo init, profile install, systemd timers, first backups verified).
• 9 hosts registered under servers/ from gap-analysis (PFI VMs + SureFire tenant, with tenancy awareness).
• Discovery scripts (FortiGate SSH + UniFi cloud API + TSV diff vs inventory).
• Proxmox inspect script + fleet-wide refresh wrapper.
• Calibre-Web-Automated migration replacing calibre + calibre-web pair.
• llama-swap Qwen3.6 stock + mradermacher abliterated, 128K ctx.
• Mattermost retired (not running, compose dir cleanup pending).
• FortiGate 101F at NH3 retired; homepage card removed.

Memory pointers (for future Claude sessions)

Relevant ~/.claude/.../memory/ entries:

• server_split.md — host placement rules
• feedback_ssh_sudo.md — use ssh -t for remote sudo (mostly superseded by elway, but still applies to ad-hoc ssh)
• feedback_git_autonomous.md — handle git commits without asking
• feedback_git_commits.md — no Claude attribution in commit messages
• feedback_use_elway.md — write elway playbooks; don't chain ssh+sudo
• feedback_backups_must_not_risk_production.md — rule adopted after the 2026-04-23 ana-nas self-backup crash
• project_backup_pipeline_gaps.md — user's explicit goal of "all hosts + configs + DBs backed up"
• project_db_migrate_off_nfs.md — DB-off-NFS decision + status
• project_surefire_tenant.md — SureFire tenancy boundary awareness
• reference_gitea_remote.md — origin is vh/esh-pfi-infrastructure on gitea.phasefinal.com; tea CLI logged in as vh
• reference_task_board.md — task-board plugin + tools contract
• storage_ana_nas.md — ana NAS is Debian LXC (CT 109), not TrueNAS
• storage_nh3_nas.md — NH3 NAS via syncuser, not admin
• incident_ana_nas_spof.md — blast-radius matrix for ana-nas outages