esh-pfi-infrastructure/STATUS.md

Status + Open Issues

Last updated: 2026-04-29

Snapshot of fleet state and open work. Refresh this file when a pass of significant work lands — don't let it drift quietly.

New session? Read docs/orientation.md first.

What's in place

Backup coverage (2-layer, fully operational)

• PBS fleet-wide. All 5 hypervisors (pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) → pbs-ana (primary, VM on pfi-pve with NFSv3 datastore on ana-nas). Per-hypervisor namespaces to avoid VMID collisions. DR mirror at pbs-nh3 (VM on nh3-pve with NFSv3 datastore on nh3-nas Synology) pulls daily at 06:00. Verify jobs armed on both sides.
• restic 8/8 hosts with resticprofile + systemd timers at 01:00:

  Host	Target rest-server	DB hooks
  ana-docker	rest-server-ana (local)	synapse, seafile, vaultwarden-pg, gitea (native dump), openwebui
  ana-ml2	rest-server-ana (cross-site)	— (no DBs)
  nh3-docker	rest-server-nh3 (local, Synology)	—
  esh-docker-vm	rest-server-ana (cross-site)	paperless-pg, HA SQLite, CWA SQLite, pgadmin SQLite, uptime-kuma SQLite
  vm-esh-nas	rest-server-ana (cross-site)	—
  nh3-dev (workstation)	rest-server-nh3 (local)	—
  irv-ml1	rest-server-nh3 (via WG)	—
  esh-vm-db	rest-server-ana (cross-site)	pg_dumpall, mongodump

• Cross-site restic rsync. ana-nas → nh3-nas at 04:00 daily (mirrors rest-server-ana data). nh3-nas → ana-nas at 05:00 daily (mirrors rest-server-nh3 data, runs as root since DSM writes files as admin mode 400). Tracked at configs/rsync/{ana-nas-to-nh3,nh3-nas-to-ana}/.

Inventory

• ~22 tracked hosts under servers/ across ANA / NH3 / ESH / IRV sites. SSH config aliases in ~/.ssh/config for every host — ssh <name> just works.
• Homepage at http://10.0.50.45:5100 — function-first layout (Main / Infrastructure / Toolchain tabs), per-group icons, four Infra sub-groups (ANA / NH3 / IRV / ESH). Docker auto-discovery active for 5 hosts (ana-docker, ana-ml2, nh3-docker, esh-docker-vm, irv-ml1).
• FortiGate + UniFi discovery scripts produce TSVs; gap analysis against servers/*/ for unmanaged IPs.

Architecture decisions (durable)

• DB data on local disk, not NFS. pfi-postgres migrated 2026-04-23; fstab + NFS mount removed 2026-04-24 (pfi-postgres now fully decoupled from ana-nas). Mongo on esh-vm-db confirmed already local. Removes the biggest ana-nas blast-radius risk. Memory: project_db_migrate_off_nfs.md.
• Backups must not risk production. Rule adopted after 2026-04-23 ana-nas self-backup crash. Memory: feedback_backups_must_not_risk_production.md.
• offen sidecars retired fleet-wide 2026-04-23. restic covers equivalent scope; reclaimed 16 GB of redundant tarballs on /mnt/backup/docker/esh-vm-docker/.
• Gitea remote for this repo (2026-04-23). origin is vh/esh-pfi-infrastructure on gitea.phasefinal.com. Also: vh/task-board (new 2026-04-24) hosts a Claude Code plugin + marketplace serving the assistant task-state dashboard.
• Prefer elway for multi-step SSH work (2026-04-24). The new scripts/elway mini-playbook runner replaces chained ssh -t sudo … commands — handles sudo once up front, structured reporting, idempotency (creates/when/changed_when). Memory: feedback_use_elway.md; template playbook: playbooks/elway-smoke.yaml.

Tooling

• scripts/server_inspect.sh + refresh-server-info.sh for Docker hosts.
• scripts/proxmox_inspect.sh + refresh-proxmox-info.sh for PVE nodes (reports VMs/LXCs/storage/backup-coverage).
• scripts/discover-fortigate.sh, scripts/discover-unifi.sh, scripts/discover-gaps.sh for network-level inventory discovery.
• scripts/deploy-stack.sh + scripts/sync-stacks.sh for compose push/pull.
• scripts/add-host.sh for new-host registration.
• scripts/elway (2026-04-24) — mini-ansible playbook runner over SSH. Playbooks live under playbooks/. Tier 1 (creates/when) + tier 2 (changed_when) idempotency; handlers + register + multi-host fan-out deferred as gitea issues #3–#5.
• tea CLI at /usr/local/bin/tea — gitea CLI, already logged in as vh. Use for issue / PR work instead of inventing URLs.

Open issues

🟥 Blocked — irv-ml1 stalled after CUDA driver upgrade (2026-04-25) — RECOVERED 2026-04-27

Resolution: the host eventually came back on its own — driver 595.58.03 (CUDA 13.2) IS working, both GPUs detected (RTX 3090 + RTX A6000), kernel modules loaded cleanly. The original "stall" symptoms must have been a one-shot post-install issue that resolved on a later boot. All 5 pre-existing GPU stacks (comfyui, cosyvoice, qwen3-tts, index-tts, parakeet) restarted clean and report healthy. Homepage's irv-ml1-docker provider re-enabled via configs/homepage/ docker.yaml. Remaining post-recovery work was the deferred deploys (Kokoro GPU, VibeVoice, Chatterbox) — see "Session milestones — 2026-04-27" below.

The original stall context is preserved below for the next time we do a CUDA driver bump on a bare-metal host:

What was attempted: scripts/upgrade-irv-ml1-cuda.sh phase1 on 2026-04-25. Phase 1 is supposed to: snapshot current nvidia packages → stop GPU containers → apt install cuda-drivers → DKMS rebuild → reboot. Whether all those steps actually completed before the unclean boot is unknown — the snapshot file at /tmp/nvidia-pre-upgrade.txt on irv-ml1 (if it survived) will say.

Recovery options (in increasing invasiveness):

1. Console diagnose first. SSH may or may not work depending on the failure mode. Check via console (irv-ml1 is bare-metal, has physical access via Lenovo IPMI). Look at journalctl -b -1 from the previous boot to see what failed: DKMS build failure, kernel module load error, nvidia-container-toolkit init regression, etc.

2. Rollback the driver if the host is reachable but GPU is broken:

   scripts/upgrade-irv-ml1-cuda.sh rollback
   

   This reinstalls the snapshot from /tmp/nvidia-pre-upgrade.txt with --allow-downgrades and reboots. Restores 570.124.06.

3. Manual driver pin if the script can't recover:

   ssh irv-ml1 sudo apt install --allow-downgrades \
     nvidia-driver=570.124.06-1 nvidia-driver-cuda=570.124.06-1 \
     nvidia-driver-libs=570.124.06-1 nvidia-kernel-open-dkms=570.124.06-1
   ssh irv-ml1 sudo apt-mark hold nvidia-driver nvidia-driver-cuda \
     nvidia-driver-libs nvidia-kernel-open-dkms
   ssh irv-ml1 sudo systemctl reboot
   

4. Try a different driver version if 595 is genuinely incompatible with kernel 6.1.0-37: 580 series may build cleanly. Check via apt-cache madison cuda-drivers after recovery.

Once irv-ml1 is healthy again, deferred work to revisit:

• Kokoro GPU variant: deployed CPU works as a fallback. Switch .env to KOKORO_VARIANT=gpu / KOKORO_USE_GPU=true / KOKORO_GPU_DEVICES=0 and redeploy via scripts/elway irv-ml1 --playbook playbooks/deploy-kokoro.yaml.
• VibeVoice 1.5B deploy (long-form / multi-speaker dialogue): scripts/elway irv-ml1 --playbook playbooks/deploy-vibevoice.yaml. Stack files at stacks/vibevoice/. ~12 min build + ~7 GB model.
• Chatterbox Turbo deploy (low-latency English voice cloning): scripts/elway irv-ml1 --playbook playbooks/deploy-chatterbox.yaml. Stack files at stacks/chatterbox/. ~10 min build + ~6 GB model.
• Verify the existing 5 GPU stacks still work post-upgrade: comfyui, cosyvoice, qwen3-tts, index-tts, parakeet. They were all Up + healthy before the driver attempt. Smoke-test each.
• Re-evaluate Kokoro variant default after recovery. If a 595+ series driver doesn't end up working, the .env.example default of KOKORO_VARIANT=cpu is correct as-is. If a working GPU driver lands, flip the default to gpu for future deploys.

Don't run on irv-ml1 until this is closed:

• Any of the three deploy playbooks above
• Any host-state-mutating playbook against irv-ml1
• The driver upgrade script itself unless intentionally retrying

Once recovered, update this section with what fixed it and remove the 🟥 marker.

🟥 Quick wins (do next)

1. Fix Backrest's esh-docker-vm URI — done.

2. Push SSH keys + pull snapshots for 6 unrefreshed hosts — done 2026-04-23. 4 VMs (pfi-ana-webhost, pfi-pteradactyl, pfi-tacticalrmm, pfi-postgres) took the workstation key via ssh-copy-id with lkraven@. 2 LXCs (ana-filebot, ana-wg) needed key installed via pct push from pfi-pve because PermitRootLogin prohibit-password blocked ssh-copy-id. Minor: pfi-pteradactyl's server_inspect Docker section is blank because lkraven is not in the docker group there — sudo usermod -aG docker lkraven + re-login to fix.

3. Patch out forget schedules from all 6 restic profiles — done.

3b. Discover sf-r630 OS-side IP — resolved 2026-04-23: the R630 with iDRAC 10.250.250.110 is the same physical box that runs sfsrv-ana (Proxmox VE at 10.250.250.115). No separate OS IP to find. servers/sf-r630/ now clarifies it as the hardware/BMC-only inventory entry; servers/sfsrv-ana/ is the OS view. servers/ana-ml2/README.md updated with its own distinct BMC IP (10.250.250.50, Supermicro) to prevent future confusion between the two physical chassis.

4. (removed — see note above)

🟧 Real work (dedicated session each)

4b. Migrate DB data directories off NFS onto local VM disk. — done 2026-04-23/24. Postgres on pfi-postgres migrated 04-23; mongodb on esh-vm-db confirmed already local (and serves zero user data in practice — paperless uses Postgres 15 on the same VM). fstab entry + NFS mount for /mnt/db on pfi-postgres removed 2026-04-24 via playbooks/decouple-pfi-postgres-from-ana-nas.yaml. pfi-postgres has zero remaining dependency on ana-nas. Only residue: cold archive of pre-migration data still on ana-nas (/mnt/db/pfi-*); harmless, can sit indefinitely.

5. Rotate exposed secrets — done 2026-04-23. All six rotated: vaultwarden/gitea/paperless-ng Postgres passwords (hardcoded compose.yaml literals moved to gitignored .env files in the process), ana-docker + ana-ml2 + esh-docker-vm rest-server htpasswd entries, and rest-server repo passphrases for ana-docker + esh-docker-vm (ana-ml2's was already rotated during prior wipe+reinit). Also discovered along the way: paperless uses a separate esh-vm-db VM (10.0.50.60), not pfi-postgres as the old notes implied.

6. Cross-site rsync — done 2026-04-23, both directions.

   • ana → nh3 (04:00 daily): ana-nas:/mnt/backup/restic/repo/ana/ → nh3-nas:/volume1/Backup/restic-ana-mirror/. Runs on ana-nas as lkraven. 15.6 GB initial sync completed 07:18 UTC.
   • nh3 → ana (05:00 daily): nh3-nas:/volume1/Backup/restic/ → ana-nas:/mnt/backup/restic-nh3-mirror/. Runs on nh3-nas as root (rest-server-nh3 container writes mode-400 files; only root can read them on Synology). 2.31 GB initial sync completed 07:42 UTC.
   • Tracked at configs/rsync/ana-nas-to-nh3/ and configs/rsync/nh3-nas-to-ana/ respectively.

6b. PBS deployment across the fleet. Runbook at docs/runbooks/pbs-deployment.md. Phases 0–6 done (2026-04-22): PBS-ANA on pfi-pve with NFS datastore on Ana NAS (NFSv3 for ZFS case-insensitivity), PBS-NH3 on nh3-pve with NFS datastore on NH3 Synology (NFSv3 + Linux-mode POSIX 777 to avoid syno_acl interference), per-hypervisor namespaces, API tokens, verify jobs on both, one-way sync ANA → NH3 at 06:00 daily, and all 5 hypervisors (pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) onboarded to PBS-ANA. Remaining phases: - Phase 7-8 — one week burn-in, then retire legacy vzdump targets on each hypervisor (keep until 2026-04-29 earliest) - Phase 9 — homepage cards for PBS-ANA + PBS-NH3 web UIs, memory/doc updates, servers/pbs-ana + servers/pbs-nh3 dirs

7. SureFire tenant backup plan decision — resolved 2026-04-23 by folding sfsrv-ana into the fleet-wide PBS deployment (dedicated sfsrv-ana namespace on PBS-ANA, replicates to PBS-NH3 via the same sync job as the rest of the fleet). Hosting-agreement option chosen: PFI provides backup coverage as part of managed hosting.

🟨 Prereqs / polish

8. Synology SSH setup — done 2026-04-22. Dedicated syncuser account (admin-group membership) with key auth, registered as servers/nh3-nas/, reachable as ssh nh3-nas. Unlocks cross-site rsync (item 6), rest-server-nh3 healthcheck deploy, and .htpasswd edits on the NH3 side.

9. scripts/restic-prune.sh — done 2026-04-24. Quarterly disk-hygiene tool. Drops --append-only on the rest-server (via a temporary docker-compose.override.yaml — never edits the canonical compose), runs resticprofile forget --prune --verbose on each client, restores --append-only (with trap so it runs even on partial failure). ANA side fully automated (5 clients); NH3 side prints a manual ceremony because DSM Container Manager + sudo on syncuser aren't cleanly scriptable from this workstation. Run with scripts/restic-prune.sh ana|nh3|all, optionally --dry-run.

   First NH3 ritual run: 2026-04-27 — full process documented in docs/runbooks/nh3-prune-ritual.md (Phase 1 DSM toggle off → Phase 2 forget+prune on 3 clients → Phase 3 DSM toggle back on). Today's run was a no-op everywhere (every snapshot still in keep window). Next round: ~2026-07-27.

10. Retire offen/docker-volume-backup sidecars — done 2026-04-23. Removed from paperless-ngx and pgadmin composes on esh-docker-vm (only hosts in the fleet that had them). 16 GB of orphan tarballs at /mnt/backup/docker/esh-vm-docker/ reclaimed. Restic coverage verified equivalent: /var/lib/docker/volumes in source + pre-backup hook handles the DB dumps for paperless (against esh-vm-db), HA, pgadmin, CWA, uptime-kuma. Obsolete *_offen_backup_data exclude also removed from the restic profile.

11. Clean up retired mattermost dir on ana-docker — done (verified 2026-04-24: /opt/docker/compose/mattermost/ does not exist; no mattermost containers anywhere on the host).

🟩 Research / deferred / intermittent

12. Backrest UI intermittent timeout. Backend is confirmed healthy (direct curl GetConfig returns in 200ms). UI chokes on the same endpoint. Likely cause: SSE stream wedge in browser, or service worker stale. Workaround: Ctrl+Shift+R. Root cause investigation deferred.

13. UniFi controller homepage cards — done 2026-04-24. Both cards added to configs/homepage/services.yaml and pushed to esh-docker-vm: PFI-UDMSE (10.100.0.1, UDM Pro SE) under Infra - NH3 as the new edge device replacing the retired Fortigate 101F; ESH-UDMPM (10.0.0.1, UDM Pro Max) under Infra - ESH. Icon si-ubiquiti.

14. Prune + credential-rotation scripts as repeatable tooling (vs per-incident manual work).

🟦 Memory / documentation housekeeping

15. docs/ organization — done 2026-04-24. First-pass landed earlier (misfiled tea-*.sha256 removed; docs/README.md nav map added). Second pass landed same day: stripped the broken YAML frontmatter from both VM-102 Matrix docs (the path: values pointed at docs/pfi-ana/... which doesn't exist in this repo, and no toolchain consumed the metadata); deleted pfi/chromadb-setup.md (referenced configs/pfi-ana/... and scripts/setup-chromadb.sh, both nonexistent — deployment is long done and the operational truth lives in docker-stack.md). Kept the two VM-102 docs separate by design (each is right-sized; a merge would push past the 500-line guideline in the README).

16. STATUS.md drift discipline — done 2026-04-24. Refreshed with the post-tooling-day session work; status-regen.sh idea dropped — STATUS.md is intentionally narrative, not derivable from git/code, so auto-regen would lose information. Discipline rule: refresh STATUS.md at the end of any session where a 🟧 or 🟥 item closes, or three+ smaller items land.

Active migration — docker.io 20.10 → docker-ce 29.x

Bookworm's Debian-packaged docker.io is pinned at 20.10.24 with client/daemon API 1.41. Newer compose clients (1.52+) refuse to talk to it ("client version 1.52 is too new" during docker compose build). The migration moves each host to Docker's official docker-ce APT repo (29.x). Playbook: playbooks/upgrade-docker-ce.yaml.

Order (least → most blast radius):

1. nh3-docker — done 2026-04-24. docker-ce 29.4.1, two stacks (portainer agent + portainer) restarted clean. Debug surfaced two unit-file gotchas now baked into the playbook:
   • /etc/systemd/system/docker.service.d/override.conf from the docker.io era hardcoded /usr/sbin/dockerd (docker-ce installs at /usr/bin/dockerd) → daemon failed status=203/EXEC.
   • The shipped docker-ce unit's ExecStart=dockerd -H fd:// … conflicts with daemon.json hosts: (defined for the 0.0.0.0:2375 homepage-discovery binding) → "conflicting host options". Playbook now backs up any existing override, probes daemon.json for hosts:, and installs an override that strips -H when needed.
2. esh-docker-vm — done 2026-04-24. docker-ce 29.4.1, all 16 stacks back up clean (adguard, beszel-agent, calibre-web, cloudflare-ddns, dockge, dozzle-agent, drawio, esphome, homeassistant, homepage, mosquitto, paperless-ngx, pgadmin, traefik, uptimekuma). Surfaced one verify-only nit: the docker-compose-plugin shipped with docker-ce 29 is renumbered v5.x (was v2.x with 26-28); same Compose v2 codebase, just realigned major. Playbook's compose-version verify was hardcoded to v2.X.Y and false-failed — now accepts any major ≥ 2.
3. ana-docker — done 2026-04-24. docker-ce 29.4.1, all 29 containers back up clean. Traefik routing live (200s for matrix.phasefinal.com presence + seafile.phasefinal.com syncs verified post-restart). Sub-finding (since fixed, see below): seafile's internal seahub raced mysql on startup and wedged.

Also during the nh3-docker debug: configs/homepage/docker.yaml commented out the irv-ml1-docker provider — the 20s-per-poll ETIMEDOUTs from the stalled host were drowning homepage's logs and appeared to block ana-pfi-docker discovery (News group's Miniflux card wouldn't render until removal). Re-enable when irv-ml1 is back.

traefik-postboot — host-level recovery for daemon-restart routing races

Installed 2026-04-24 on both hosts that run traefik (esh-docker-vm, ana-docker). One-shot systemd unit (traefik-postboot.service, After=docker.service, Type=oneshot, sleep 60 → docker restart traefik) that fires once per boot. Forces a clean cold scan after the topology has settled, fixing the long-standing "traefik misses backends after host reboot" symptom (events-stream debounce + network- attach race when 30+ containers start in a burst).

Playbook: playbooks/install-traefik-postboot.yaml. Tunables: --var delay_seconds=N (default 60), --var container=NAME (default traefik). Runs systemd-analyze verify non-disruptively so installing on a healthy host doesn't blip traefik.

autorestic decommission — redundant with the new two-layer pipeline

Hosts with autorestic detected 2026-04-24: esh-docker-vm, esh-vm-db. Both are pre-pipeline ESH manual installs at /usr/local/bin/autorestic

• four systemd units backing up to /mnt/backup/restic/repo/esh. The new two-layer pipeline (PBS for VM images, structured restic profiles via configs/restic/<host>/profiles.yaml) covers everything autorestic was doing. Trigger: a typo on esh-docker-vm's autorestic-backup.timer line 2 (D:escription instead of Description) surfaced during the traefik-postboot install verify.

Playbook: playbooks/remove-autorestic.yaml. Done 2026-04-24 on both hosts after two playbook bug-fixes surfaced by the first run: (a) rm path1 \ + newline + path2 was bitten by YAML plain-scalar folding — the backslash-newline collapsed to a literal \ , so only the first path got passed to rm. Fixed by switching to a glob. (b) shell: ! command -v autorestic >/dev/null was parsed by YAML as a tag indicator with the ! stripped, inverting the verify's exit code. Fixed by quoting the value.

Deliberately untouched (decide separately):

• /srv/backups/autorestic/.autorestic.yml — config; archival value
• /mnt/backup/restic/repo/esh — historical autorestic snapshots; prune via restic forget once comfortable with new pipeline coverage.

seafile seahub startup race — fixed 2026-04-24

When ana-docker rebooted (or any time the seafile container was recreated alongside seafile-mysql), seahub (the python frontend at port 8000 inside the seafile container) raced mariadb's startup. Short-form depends_on: [db, memcached] in compose only waits for the dependency container to be CREATED, not healthy — so seafile's start.py would hit mysql Connection refused and give up, leaving nginx serving seafile but proxying to a dead python backend. Symptom in traefik logs: ~7 of 30 requests returning 502 on seafile@docker dynamic endpoints; static assets fine.

Fix applied directly to ana-docker's /opt/docker/compose/seafile/ compose.yaml (and into stacks-mirror/, which is gitignored — so not in the repo, but the file on the server is the source of truth):

• Added healthcheck to the db service using the bundled /usr/local/bin/healthcheck.sh --connect --innodb_initialized (5s interval, 30s start_period, 30 retries).
• Converted seafile's depends_on to long-form with condition: service_healthy for db (and service_started for memcached).

Verified post-recreate: seahub log shows Seahub is started ... Done. without traceback; traefik seafile@docker dropped to 0 502s in the next 20s window.

news-digest — per-item × button + cross-device hidden tray (2026-04-24)

Daily Digest pages now have a small × on each item that hides it from the page. State persists server-side at /output/hidden.json, so the same hidden set follows the user across devices (home, ipad, laptop, work). A "Hidden (N)" tray at the bottom of each page lists items hidden FROM THE CURRENT PAGE with a restore button per row; older hidden ids that aren't on this page sit silently in hidden.json and continue to filter future editions that include the same article.

Architecture change: news-digest-web swapped from nginx:alpine to a FastAPI app on uvicorn, built from the same Dockerfile as the worker. One image, two containers — the worker's cron-driven ENTRYPOINT runs in news-digest-worker, while compose overrides entrypoint+command to launch uvicorn web:app in news-digest-web. Drops one image dep, adds three endpoints:

• GET /api/hidden — JSON array of hidden item ids
• POST /api/hide — {id} → adds to hidden.json
• POST /api/restore — {id} → removes from hidden.json

Item ids are stable 12-char sha1 prefixes (reddit:<post_id> / miniflux:<entry_id>) computed in digest.py at render time so hide-once means hide-forever-for-that-article. Storage: single JSON array, atomic writes via tempfile+rename, threading.Lock inside the single uvicorn worker. No auth — same trust boundary as the digest itself (LAN-only on port 8181).

The deploy playbook also dropped its DOCKER_BUILDKIT=0 fallback (no longer needed post docker-ce 29 migration) and gained three verify steps: /api/hidden returns a JSON array, app.js is reachable, and a synthetic smoke-<ts> id round-trips through hide → list → restore → list.

llama-swap — added two vision-capable Qwen 3.6 entries (2026-04-24)

• qwen3.6-35-a3b-heretic — llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF:Q6_K (~28 GB) + auto-loaded BF16 mmproj for vision.
• qwen3.6-27b — unsloth/Qwen3.6-27B-GGUF:UD-Q6_K_XL (~26 GB) + auto-loaded BF16 mmproj for vision (dense 27B, not MoE).

Both pre-pulled into /tank/aimodels/huggingface (HF_HOME=/hfcache in the container) via a one-shot python:3.12-slim container running huggingface_hub.snapshot_download with hf_transfer for parallel chunked downloads — heretic's 29 GB landed in 4:10, unsloth's 26.5 GB in 3:46 (~118 MB/s each). Both repos include mmproj-BF16.gguf alongside the main GGUF and llama-server's -hf flag auto-loads it when present, so vision works without an explicit --mmproj path.

Stack tree convention (canonical vs mirror) — clarified 2026-04-24

The workspace has two stack trees and they look similar enough that deploy-stack.sh was reading from the wrong one (stacks-mirror/ instead of stacks/). Edits to stacks/llama-swap/config.yaml for the two new model entries above never reached ana-ml2 on the first deploy attempt because the diff was canonical-vs-mirror, not canonical-vs-server. Roles now nailed down explicitly in CLAUDE.md and memory/convention_stacks_vs_mirror.md:

tree	role	git	who writes	who reads
stacks/<stack>/	canonical / intent	tracked	you / Claude	deploy-stack.sh
stacks-mirror/<host>/<stack>/	snapshot / reality	gitignored	sync-stacks.sh	drift inspection

Side fix: moved stacks/llama-swap/config.yaml → stacks/llama-swap/conf/config.yaml so the deploy mapping matches the server layout (stacks/<stack>/conf/<file> → /opt/docker/conf/<stack>/<file>).

Session milestones — 2026-04-29 (Nevermore extraction)

The news-digest applet was extracted into its own repo at vh/nevermore — clean break, fresh git history, renamed throughout (DIGEST_* → NEVERMORE_, news-digest- containers → nevermore-*, masthead → "Nevermore"). The existing news-digest-* deployment on ana-docker continues to run unchanged on its baked image; future updates come from the new repo. Migration path documented in nevermore's README.

Removed from this workspace: stacks/news-digest/, playbooks/deploy-news-digest.yaml, scripts/add-digest-user.sh.

Session milestones — 2026-04-28..29 (audio gen, digest applet, task-board polish, tooling)

Audio generation on irv-ml1 (A6000)

• ACE-Step 1.5 (music gen, Apache 2.0) — port 8210. Hybrid diffusion + LLM, ~10-12 GB VRAM at inference. Custom Dockerfile patches upstream's torch/cu126 resolution bug; patched infer-api.py ships in the image (upstream's positional tuple is missing 6 of 24 pipeline args). Verified bench: 60 s clip in 14.8 s (~4× realtime).
• Stable Audio Open 1.0 (SFX/ambience, Stability Community license) — port 8211. Custom FastAPI shim wraps diffusers.StableAudioPipeline. ~4-6 GB VRAM in fp16, up to 47 s clips at 44.1 kHz.

Daily Digest applet

• World + Local headline sections above the existing tech/reddit cards. seed-headlines.py provisions the categories and curated feeds (BBC + NPR + Al Jazeera; LA Times + Voice of OC) in miniflux.
• Article-extracted summaries via trafilatura — 2-3 sentence summaries grounded in real article content (was: title-paraphrase). Cached at /output/.article-cache.json with 7-day TTL. Adds ~30 s per digest run, parallel-prefetched.
• Per-desk collapse chevron, localStorage-persisted per device.
• Mobile-responsive — true 480 px breakpoint, fixed masthead overlap, cache-busted asset URLs.
• Multi-tenant via scripts/add-digest-user.sh <name> — shared miniflux + per-user digest stack (own port, hide-state, cron). DIGEST_CRON_AM / DIGEST_CRON_PM are now env-driven; each teammate's stack fires on their own hours.

task-board

• v0.1.11 pending_shells / orange treatment. New task_set_shells MCP tool + pending_shells field on the Task model (schema v6, ALTER ADD COLUMN). When a card is state=waiting AND has non-empty shells, it flips from blue (waiting on user) to orange (waiting on background work). Cards with non-empty shells are excluded from auto-dormant promotion.
• v0.1.12 card-bar rhythms — three distinct pulse cadences (active 2.4 s, waiting 3.6 s patient breath, attention 1.6 s louder) so column scan reads at a glance.

Tooling

• scripts/fork-fleet.sh — bootstrap a new fleet-management workspace from this one. Mirrors reusable tooling, strips fleet-specific content, regenerates skeleton CLAUDE.md / README / STATUS, fresh git history.
• CLAUDE.md task_set_shells discipline — assistant must call task_set_shells on any Bash run_in_background (and clear when done) so the orange treatment fires; hooks have no way to enumerate Claude Code's bg tasks externally.

New stack on ana-docker

• memos (port 5230) — stock neosmemo/memos:stable, SQLite at /opt/docker/conf/memos/data/, homepage label Notes (auto-discovered).

Session milestones — 2026-04-27 (irv-ml1 recovery + new GPU deploys)

irv-ml1 fully recovered

• Host alive, driver 595.58.03 / CUDA 13.2, both GPUs detected, kernel modules loaded.
• 5 pre-existing GPU stacks (comfyui, cosyvoice, qwen3-tts, index-tts, parakeet) restarted clean and report healthy.
• Homepage irv-ml1-docker provider re-enabled in configs/homepage/docker.yaml, pushed via the rsync workflow.

Three new GPU stacks deployed on irv-ml1

• Kokoro GPU variant — first try succeeded once the .env.example default was flipped from cpu → gpu (matching the now-working driver). KOKORO_VARIANT=gpu / KOKORO_USE_GPU=true / KOKORO_GPU_DEVICES=0 (pinned to RTX 3090; ~1 GB VRAM, doesn't need the A6000). Serving on irv-ml1:8193.

• VibeVoice 1.5B — needed two iterations:

  • VIBEVOICE_SHA=7614c469a145 (12-char short) made docker buildx's git source resolver error with "repository does not contain ref 7614c469a145" — same commit IS still HEAD of main, but buildx requires the FULL 40-char SHA even when the short is unambiguous. Now 7614c469a145dafc30b1fd5f6a6046ea831dbd6a.
  • The verify step greped /v1/audio/voices for voices/voice/ alloy/Carter literals, but VibeVoice's actual response shape is OpenAI list-format {"object":"list","data":[]}. Switched to JSON-parse + assert object="list" and data is a list. Robust against an empty voices set on a fresh install.
  • Serving on irv-ml1:8194.

• Chatterbox Turbo (devnen wrapper) — needed three iterations:

  • Old playbook used dockerfile: docker/Dockerfile.gpu; devnen restructured the repo to put Dockerfiles at root, renamed by CUDA version (Dockerfile.cu128, .cpu, .rocm). Switched to Dockerfile.cu128 (GPU build for CUDA 12.8 toolkit; works with irv-ml1's 595.58.03 driver via backward-compat).
  • Pinned CHATTERBOX_SHA from main to the current SHA 68213e75c248f4f4bbd185f7c20ec02722fdc79b so future upstream churn doesn't break us without warning.
  • Wait + verify steps probed /health, but devnen's wrapper doesn't expose that route (404). Switched to /api/model-info which returns {"loaded":true,...} only after the model finishes loading — doubles as liveness + readiness. Compose healthcheck
    • verify both updated. Verify of /v1/audio/voices also fixed to parse the wrapper's actual response shape {"status":"ok","voices":[...]} (note: NOT the same OpenAI list shape vibevoice uses).
  • Serving on irv-ml1:8196.

llama-swap pin/unload cleanup

• Removed ttl: 600 from qwen3.6-35-a3b-abliterated, qwen3.6-35-a3b-heretic, qwen3.6-27b so they stay loaded across idle (still evictable when another non-coexistent model is requested).
• qwen3.6-35-a3b removed from the pinned group — its persistent: true flag was preventing eviction under memory pressure, OOM-ing other model loads. Now sits with ttl: 0 outside any group: never idle-unloads but is evictable.
• granite-4-small added to the pinned group alongside qwen3.5-9b so it stops swapping with qwen3.6-27b when both are used concurrently. ~5-6 GB at Q4_K_M + 120K KV ≈ comfortable inside the existing pin budget.

Backup verification post-recovery

All three layers green as of 2026-04-27:

• Per-host restic (resticprofile-backup@profile-default.timer): all 6 hosts (ana-docker, esh-docker-vm, nh3-docker, esh-vm-db, ana-ml2, irv-ml1) ran successfully at 01:00 today. The migration to docker-ce on three of these hosts didn't disrupt the timers. irv-ml1's timer fired right after recovery.
• PBS-ANA: every namespace (ana-pve, esh-pve, nh3-pve, sfsrv-pve) has fresh 2026-04-27 snapshots for every VM/CT.
• PBS-NH3 (DR mirror): synced — 2026-04-27 backups present in /mnt/pbs-datastore/ns/. NH3 NAS at 27TB / 42TB used (64%, healthy headroom).

Note: backrest's UI shows recentBackups: {} for all repos because backrest in this setup is a viewer with zero plans configured — the actual orchestration is the per-host resticprofile timers. Backrest's only configured operations are _unassociated_ (manual one-off pushes from the UI) and _system_ housekeeping. Don't be alarmed by the empty dashboard — check the per-host timers instead.

Session milestones — 2026-04-24 (the "tooling day" + housekeeping pm)

Morning / early afternoon — the original tooling day:

• irv-ml1 AI stacks deployed as Docker: ComfyUI (port 8188, host- writable workflows), Parakeet ASR (port 8765, rewritten on sherpa-onnx after the Shadowfita FastAPI wrapper hit two unfixed upstream bugs), CosyVoice 3 (port 8190, GLaDOS voice cloned + live).
• irv-ml1 restic profile extended to cover /worktank/{comfyui/basedir/{user,custom_nodes,input},cosyvoice/voices}; bulk weights + scratch dirs stay excluded.
• scripts/elway shipped — ~800-line Python playbook runner with tier 1 + tier 2 idempotency; handlers / register / multi-host / content-hash are deferred (gitea #3–#6).
• task-board built end-to-end (separate repo, vh/task-board on gitea) and shipped as a Claude Code plugin. Green/red cards per session via UserPromptSubmit + Stop hooks; four MCP tools expose explicit activity tracking.
• pfi-postgres NFS decoupling finished item 4b — zero residual dependency on ana-nas for that VM.

Late afternoon / evening — task-board iteration + STATUS sweep:

• task-board v0.1.1 → v0.1.3 shipped over four iterations on the live ANA deployment:
  • v0.1.1 — hooks parse Claude Code's stdin JSON for session_id and append a short suffix when TASK_BOARD_SESSION isn't set, so two sessions in one project no longer collide on a single card. Dormant transition preserves cumulative idle time (state_entered_at = last_update_at instead of now).
  • v0.1.2 — favicon (3-column SVG in active/waiting/dormant state colors); served at /static/favicon.svg with a <link rel="icon"> and a /favicon.ico route returning the same SVG. Followup fix for an XML-illegal -- in a comment.
  • v0.1.2-followup — UI live-duration ticker bumped from 5 s → 1 s (humanDuration floors to integer seconds; cheap render).
  • v0.1.3 — case-insensitive session names. sessions.name COLLATE NOCASE; real ALTER migration (not a wipe) — keeps earliest- created_at row as canonical, reassigns child comments. Write path canonicalizes session label before inserting comments. Read path uses COLLATE NOCASE for safety on external API callers.
• Parakeet (irv-ml1) healthcheck fix — image ships wget not curl; healthcheck swap, 2,190 failing checks → healthy.
• AIPA-MCP project session label fix — set TASK_BOARD_SESSION= Architect in .claude/settings.json, updated the project's CLAUDE.md to specify session="Architect" for explicit MCP calls, and renamed the existing AIPA-MCP card → Architect in the live SQLite (1 session row + 25 comment rows preserved).
• STATUS items 11 / 13 / 15 / 16 closed. Mattermost dir gone on ana-docker (verified); UniFi UDM cards added to homepage (PFI-UDMSE for NH3 edge replacing the retired Fortigate 101F, ESH-UDMPM for ESH); docs nav map added + stale chromadb-setup.md removed + VM-102 Matrix docs frontmatter stripped; STATUS.md discipline rule recorded.

Session milestones — 2026-04-20 / 2026-04-21

• Homepage reorganized to function-first with tabs (Main / Infrastructure / Toolchain).
• Per-group icons + equal-height layout + 4-column grids.
• Fleet-wide label sweep (function groups, Service Networking renamed from Wiring/Plumbing due to homepage-parser slash bug).
• 6 restic clients deployed end-to-end (client creds, repo init, profile install, systemd timers, first backups verified).
• 9 hosts registered under servers/ from gap-analysis (PFI VMs + SureFire tenant, with tenancy awareness).
• Discovery scripts (FortiGate SSH + UniFi cloud API + TSV diff vs inventory).
• Proxmox inspect script + fleet-wide refresh wrapper.
• Calibre-Web-Automated migration replacing calibre + calibre-web pair.
• llama-swap Qwen3.6 stock + mradermacher abliterated, 128K ctx.
• Mattermost retired (not running, compose dir cleanup pending).
• FortiGate 101F at NH3 retired; homepage card removed.

Memory pointers (for future Claude sessions)

Relevant ~/.claude/.../memory/ entries:

• server_split.md — host placement rules
• feedback_ssh_sudo.md — use ssh -t for remote sudo (mostly superseded by elway, but still applies to ad-hoc ssh)
• feedback_git_autonomous.md — handle git commits without asking
• feedback_git_commits.md — no Claude attribution in commit messages
• feedback_use_elway.md — write elway playbooks; don't chain ssh+sudo
• feedback_backups_must_not_risk_production.md — rule adopted after the 2026-04-23 ana-nas self-backup crash
• project_backup_pipeline_gaps.md — user's explicit goal of "all hosts + configs + DBs backed up"
• project_db_migrate_off_nfs.md — DB-off-NFS decision + status
• project_surefire_tenant.md — SureFire tenancy boundary awareness
• reference_gitea_remote.md — origin is vh/esh-pfi-infrastructure on gitea.phasefinal.com; tea CLI logged in as vh
• reference_task_board.md — task-board plugin + tools contract
• storage_ana_nas.md — ana NAS is Debian LXC (CT 109), not TrueNAS
• storage_nh3_nas.md — NH3 NAS via syncuser, not admin
• incident_ana_nas_spof.md — blast-radius matrix for ana-nas outages