# Status + Open Issues Last updated: 2026-04-29 Snapshot of fleet state and open work. Refresh this file when a pass of significant work lands — don't let it drift quietly. **New session? Read [`docs/orientation.md`](docs/orientation.md) first.** ## What's in place ### Backup coverage (2-layer, fully operational) - **PBS fleet-wide.** All 5 hypervisors (pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) → `pbs-ana` (primary, VM on pfi-pve with NFSv3 datastore on ana-nas). Per-hypervisor namespaces to avoid VMID collisions. DR mirror at `pbs-nh3` (VM on nh3-pve with NFSv3 datastore on nh3-nas Synology) pulls daily at 06:00. Verify jobs armed on both sides. - **restic 8/8 hosts** with resticprofile + systemd timers at 01:00: | Host | Target rest-server | DB hooks | |---|---|---| | ana-docker | rest-server-ana (local) | synapse, seafile, vaultwarden-pg, gitea (native dump), openwebui | | ana-ml2 | rest-server-ana (cross-site) | — (no DBs) | | nh3-docker | rest-server-nh3 (local, Synology) | — | | esh-docker-vm | rest-server-ana (cross-site) | paperless-pg, HA SQLite, CWA SQLite, pgadmin SQLite, uptime-kuma SQLite | | vm-esh-nas | rest-server-ana (cross-site) | — | | nh3-dev (workstation) | rest-server-nh3 (local) | — | | irv-ml1 | rest-server-nh3 (via WG) | — | | esh-vm-db | rest-server-ana (cross-site) | pg_dumpall, mongodump | - **Cross-site restic rsync.** `ana-nas → nh3-nas` at 04:00 daily (mirrors rest-server-ana data). `nh3-nas → ana-nas` at 05:00 daily (mirrors rest-server-nh3 data, runs as root since DSM writes files as admin mode 400). Tracked at `configs/rsync/{ana-nas-to-nh3,nh3-nas-to-ana}/`. ### Inventory - **~22 tracked hosts** under `servers/` across ANA / NH3 / ESH / IRV sites. SSH config aliases in `~/.ssh/config` for every host — `ssh ` just works. - **Homepage** at — function-first layout (Main / Infrastructure / Toolchain tabs), per-group icons, four Infra sub-groups (ANA / NH3 / IRV / ESH). Docker auto-discovery active for 5 hosts (ana-docker, ana-ml2, nh3-docker, esh-docker-vm, irv-ml1). - **FortiGate + UniFi discovery scripts** produce TSVs; gap analysis against `servers/*/` for unmanaged IPs. ### Architecture decisions (durable) - **DB data on local disk, not NFS.** pfi-postgres migrated 2026-04-23; fstab + NFS mount removed 2026-04-24 (pfi-postgres now fully decoupled from ana-nas). Mongo on esh-vm-db confirmed already local. Removes the biggest ana-nas blast-radius risk. Memory: `project_db_migrate_off_nfs.md`. - **Backups must not risk production.** Rule adopted after 2026-04-23 ana-nas self-backup crash. Memory: `feedback_backups_must_not_risk_production.md`. - **offen sidecars retired fleet-wide 2026-04-23.** restic covers equivalent scope; reclaimed 16 GB of redundant tarballs on `/mnt/backup/docker/esh-vm-docker/`. - **Gitea remote for this repo (2026-04-23).** `origin` is `vh/esh-pfi-infrastructure` on `gitea.phasefinal.com`. Also: `vh/task-board` (new 2026-04-24) hosts a Claude Code plugin + marketplace serving the assistant task-state dashboard. - **Prefer elway for multi-step SSH work (2026-04-24).** The new `scripts/elway` mini-playbook runner replaces chained `ssh -t sudo …` commands — handles sudo once up front, structured reporting, idempotency (creates/when/changed_when). Memory: `feedback_use_elway.md`; template playbook: `playbooks/elway-smoke.yaml`. ### Tooling - `scripts/server_inspect.sh` + `refresh-server-info.sh` for Docker hosts. - `scripts/proxmox_inspect.sh` + `refresh-proxmox-info.sh` for PVE nodes (reports VMs/LXCs/storage/backup-coverage). - `scripts/discover-fortigate.sh`, `scripts/discover-unifi.sh`, `scripts/discover-gaps.sh` for network-level inventory discovery. - `scripts/deploy-stack.sh` + `scripts/sync-stacks.sh` for compose push/pull. - `scripts/add-host.sh` for new-host registration. - **`scripts/elway`** (2026-04-24) — mini-ansible playbook runner over SSH. Playbooks live under `playbooks/`. Tier 1 (creates/when) + tier 2 (changed_when) idempotency; handlers + register + multi-host fan-out deferred as gitea issues #3–#5. - **`tea` CLI** at `/usr/local/bin/tea` — gitea CLI, already logged in as `vh`. Use for issue / PR work instead of inventing URLs. ## Open issues ### ~~🟥 Blocked — irv-ml1 stalled after CUDA driver upgrade (2026-04-25)~~ — RECOVERED 2026-04-27 **Resolution**: the host eventually came back on its own — driver 595.58.03 (CUDA 13.2) IS working, both GPUs detected (RTX 3090 + RTX A6000), kernel modules loaded cleanly. The original "stall" symptoms must have been a one-shot post-install issue that resolved on a later boot. All 5 pre-existing GPU stacks (comfyui, cosyvoice, qwen3-tts, index-tts, parakeet) restarted clean and report healthy. Homepage's irv-ml1-docker provider re-enabled via `configs/homepage/ docker.yaml`. Remaining post-recovery work was the deferred deploys (Kokoro GPU, VibeVoice, Chatterbox) — see "Session milestones — 2026-04-27" below. The original stall context is preserved below for the next time we do a CUDA driver bump on a bare-metal host: **What was attempted**: `scripts/upgrade-irv-ml1-cuda.sh phase1` on 2026-04-25. Phase 1 is supposed to: snapshot current nvidia packages → stop GPU containers → `apt install cuda-drivers` → DKMS rebuild → reboot. Whether all those steps actually completed before the unclean boot is unknown — the snapshot file at `/tmp/nvidia-pre-upgrade.txt` on irv-ml1 (if it survived) will say. **Recovery options** (in increasing invasiveness): 1. **Console diagnose first.** SSH may or may not work depending on the failure mode. Check via console (irv-ml1 is bare-metal, has physical access via Lenovo IPMI). Look at `journalctl -b -1` from the previous boot to see what failed: DKMS build failure, kernel module load error, nvidia-container-toolkit init regression, etc. 2. **Rollback the driver** if the host is reachable but GPU is broken: ``` scripts/upgrade-irv-ml1-cuda.sh rollback ``` This reinstalls the snapshot from `/tmp/nvidia-pre-upgrade.txt` with `--allow-downgrades` and reboots. Restores 570.124.06. 3. **Manual driver pin** if the script can't recover: ``` ssh irv-ml1 sudo apt install --allow-downgrades \ nvidia-driver=570.124.06-1 nvidia-driver-cuda=570.124.06-1 \ nvidia-driver-libs=570.124.06-1 nvidia-kernel-open-dkms=570.124.06-1 ssh irv-ml1 sudo apt-mark hold nvidia-driver nvidia-driver-cuda \ nvidia-driver-libs nvidia-kernel-open-dkms ssh irv-ml1 sudo systemctl reboot ``` 4. **Try a different driver version** if 595 is genuinely incompatible with kernel 6.1.0-37: 580 series may build cleanly. Check via `apt-cache madison cuda-drivers` after recovery. **Once irv-ml1 is healthy again**, deferred work to revisit: - **Kokoro GPU variant**: deployed CPU works as a fallback. Switch `.env` to `KOKORO_VARIANT=gpu` / `KOKORO_USE_GPU=true` / `KOKORO_GPU_DEVICES=0` and redeploy via `scripts/elway irv-ml1 --playbook playbooks/deploy-kokoro.yaml`. - **VibeVoice 1.5B deploy** (long-form / multi-speaker dialogue): `scripts/elway irv-ml1 --playbook playbooks/deploy-vibevoice.yaml`. Stack files at `stacks/vibevoice/`. ~12 min build + ~7 GB model. - **Chatterbox Turbo deploy** (low-latency English voice cloning): `scripts/elway irv-ml1 --playbook playbooks/deploy-chatterbox.yaml`. Stack files at `stacks/chatterbox/`. ~10 min build + ~6 GB model. - **Verify the existing 5 GPU stacks still work** post-upgrade: comfyui, cosyvoice, qwen3-tts, index-tts, parakeet. They were all Up + healthy before the driver attempt. Smoke-test each. - **Re-evaluate Kokoro variant default** after recovery. If a 595+ series driver doesn't end up working, the `.env.example` default of `KOKORO_VARIANT=cpu` is correct as-is. If a working GPU driver lands, flip the default to `gpu` for future deploys. **Don't run on irv-ml1 until this is closed**: - Any of the three deploy playbooks above - Any host-state-mutating playbook against irv-ml1 - The driver upgrade script itself unless intentionally retrying Once recovered, update this section with what fixed it and remove the 🟥 marker. ### 🟥 Quick wins (do next) 1. ~~**Fix Backrest's `esh-docker-vm` URI**~~ — done. 2. ~~**Push SSH keys + pull snapshots for 6 unrefreshed hosts**~~ — **done 2026-04-23**. 4 VMs (pfi-ana-webhost, pfi-pteradactyl, pfi-tacticalrmm, pfi-postgres) took the workstation key via `ssh-copy-id` with `lkraven@`. 2 LXCs (ana-filebot, ana-wg) needed key installed via `pct push` from pfi-pve because `PermitRootLogin prohibit-password` blocked `ssh-copy-id`. Minor: pfi-pteradactyl's `server_inspect` Docker section is blank because lkraven is not in the `docker` group there — `sudo usermod -aG docker lkraven` + re-login to fix. 3. ~~**Patch out `forget` schedules from all 6 restic profiles**~~ — done. 3b. ~~**Discover sf-r630 OS-side IP**~~ — **resolved 2026-04-23**: the R630 with iDRAC `10.250.250.110` is the same physical box that runs `sfsrv-ana` (Proxmox VE at `10.250.250.115`). No separate OS IP to find. `servers/sf-r630/` now clarifies it as the hardware/BMC-only inventory entry; `servers/sfsrv-ana/` is the OS view. `servers/ana-ml2/README.md` updated with its own distinct BMC IP (`10.250.250.50`, Supermicro) to prevent future confusion between the two physical chassis. 4. _(removed — see note above)_ ### 🟧 Real work (dedicated session each) 4b. ~~**Migrate DB data directories off NFS onto local VM disk.**~~ — **done 2026-04-23/24.** Postgres on pfi-postgres migrated 04-23; mongodb on esh-vm-db confirmed already local (and serves zero user data in practice — paperless uses Postgres 15 on the same VM). fstab entry + NFS mount for `/mnt/db` on pfi-postgres removed 2026-04-24 via `playbooks/decouple-pfi-postgres-from-ana-nas.yaml`. pfi-postgres has zero remaining dependency on ana-nas. Only residue: cold archive of pre-migration data still on ana-nas (`/mnt/db/pfi-*`); harmless, can sit indefinitely. 5. ~~**Rotate exposed secrets**~~ — **done 2026-04-23**. All six rotated: vaultwarden/gitea/paperless-ng Postgres passwords (hardcoded `compose.yaml` literals moved to gitignored `.env` files in the process), ana-docker + ana-ml2 + esh-docker-vm rest-server htpasswd entries, and rest-server repo passphrases for ana-docker + esh-docker-vm (ana-ml2's was already rotated during prior wipe+reinit). Also discovered along the way: paperless uses a separate `esh-vm-db` VM (10.0.50.60), not pfi-postgres as the old notes implied. 6. ~~**Cross-site rsync**~~ — **done 2026-04-23, both directions**. - **ana → nh3** (04:00 daily): `ana-nas:/mnt/backup/restic/repo/ana/` → `nh3-nas:/volume1/Backup/restic-ana-mirror/`. Runs on ana-nas as lkraven. 15.6 GB initial sync completed 07:18 UTC. - **nh3 → ana** (05:00 daily): `nh3-nas:/volume1/Backup/restic/` → `ana-nas:/mnt/backup/restic-nh3-mirror/`. Runs on nh3-nas as root (rest-server-nh3 container writes mode-400 files; only root can read them on Synology). 2.31 GB initial sync completed 07:42 UTC. - Tracked at `configs/rsync/ana-nas-to-nh3/` and `configs/rsync/nh3-nas-to-ana/` respectively. 6b. **PBS deployment across the fleet.** Runbook at `docs/runbooks/pbs-deployment.md`. Phases 0–6 **done** (2026-04-22): PBS-ANA on pfi-pve with NFS datastore on Ana NAS (NFSv3 for ZFS case-insensitivity), PBS-NH3 on nh3-pve with NFS datastore on NH3 Synology (NFSv3 + Linux-mode POSIX 777 to avoid syno_acl interference), per-hypervisor namespaces, API tokens, verify jobs on both, one-way sync ANA → NH3 at 06:00 daily, and all 5 hypervisors (pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) onboarded to PBS-ANA. Remaining phases: - Phase 7-8 — one week burn-in, then retire legacy vzdump targets on each hypervisor (keep until 2026-04-29 earliest) - Phase 9 — homepage cards for PBS-ANA + PBS-NH3 web UIs, memory/doc updates, `servers/pbs-ana` + `servers/pbs-nh3` dirs 7. ~~**SureFire tenant backup plan decision**~~ — **resolved 2026-04-23** by folding sfsrv-ana into the fleet-wide PBS deployment (dedicated `sfsrv-ana` namespace on PBS-ANA, replicates to PBS-NH3 via the same sync job as the rest of the fleet). Hosting-agreement option chosen: PFI provides backup coverage as part of managed hosting. ### 🟨 Prereqs / polish 8. ~~**Synology SSH setup**~~ — **done 2026-04-22**. Dedicated `syncuser` account (admin-group membership) with key auth, registered as `servers/nh3-nas/`, reachable as `ssh nh3-nas`. Unlocks cross-site rsync (item 6), rest-server-nh3 healthcheck deploy, and `.htpasswd` edits on the NH3 side. 9. ~~**`scripts/restic-prune.sh`**~~ — **done 2026-04-24.** Quarterly disk-hygiene tool. Drops `--append-only` on the rest-server (via a temporary `docker-compose.override.yaml` — never edits the canonical compose), runs `resticprofile forget --prune --verbose` on each client, restores `--append-only` (with `trap` so it runs even on partial failure). ANA side fully automated (5 clients); NH3 side prints a manual ceremony because DSM Container Manager + sudo on syncuser aren't cleanly scriptable from this workstation. Run with `scripts/restic-prune.sh ana|nh3|all`, optionally `--dry-run`. **First NH3 ritual run: 2026-04-27** — full process documented in [`docs/runbooks/nh3-prune-ritual.md`](docs/runbooks/nh3-prune-ritual.md) (Phase 1 DSM toggle off → Phase 2 forget+prune on 3 clients → Phase 3 DSM toggle back on). Today's run was a no-op everywhere (every snapshot still in keep window). Next round: ~2026-07-27. 10. ~~**Retire `offen/docker-volume-backup` sidecars**~~ — **done 2026-04-23**. Removed from paperless-ngx and pgadmin composes on esh-docker-vm (only hosts in the fleet that had them). 16 GB of orphan tarballs at `/mnt/backup/docker/esh-vm-docker/` reclaimed. Restic coverage verified equivalent: `/var/lib/docker/volumes` in source + pre-backup hook handles the DB dumps for paperless (against esh-vm-db), HA, pgadmin, CWA, uptime-kuma. Obsolete `*_offen_backup_data` exclude also removed from the restic profile. 11. ~~**Clean up retired mattermost dir** on ana-docker~~ — **done** (verified 2026-04-24: `/opt/docker/compose/mattermost/` does not exist; no mattermost containers anywhere on the host). ### 🟩 Research / deferred / intermittent 12. **Backrest UI intermittent timeout.** Backend is confirmed healthy (direct `curl GetConfig` returns in 200ms). UI chokes on the same endpoint. Likely cause: SSE stream wedge in browser, or service worker stale. Workaround: Ctrl+Shift+R. Root cause investigation deferred. 13. ~~**UniFi controller homepage cards**~~ — **done 2026-04-24.** Both cards added to `configs/homepage/services.yaml` and pushed to esh-docker-vm: `PFI-UDMSE` (10.100.0.1, UDM Pro SE) under `Infra - NH3` as the new edge device replacing the retired Fortigate 101F; `ESH-UDMPM` (10.0.0.1, UDM Pro Max) under `Infra - ESH`. Icon `si-ubiquiti`. 14. **Prune + credential-rotation scripts** as repeatable tooling (vs per-incident manual work). ### 🟦 Memory / documentation housekeeping 15. ~~**`docs/` organization**~~ — **done 2026-04-24.** First-pass landed earlier (misfiled `tea-*.sha256` removed; `docs/README.md` nav map added). Second pass landed same day: stripped the broken YAML frontmatter from both VM-102 Matrix docs (the `path:` values pointed at `docs/pfi-ana/...` which doesn't exist in this repo, and no toolchain consumed the metadata); deleted `pfi/chromadb-setup.md` (referenced `configs/pfi-ana/...` and `scripts/setup-chromadb.sh`, both nonexistent — deployment is long done and the operational truth lives in `docker-stack.md`). Kept the two VM-102 docs separate by design (each is right-sized; a merge would push past the 500-line guideline in the README). 16. ~~**`STATUS.md` drift discipline**~~ — **done 2026-04-24.** Refreshed with the post-tooling-day session work; `status-regen.sh` idea dropped — STATUS.md is intentionally narrative, not derivable from git/code, so auto-regen would lose information. Discipline rule: refresh STATUS.md at the end of any session where a 🟧 or 🟥 item closes, or three+ smaller items land. ## Active migration — docker.io 20.10 → docker-ce 29.x Bookworm's Debian-packaged `docker.io` is pinned at 20.10.24 with client/daemon API 1.41. Newer compose clients (1.52+) refuse to talk to it ("client version 1.52 is too new" during `docker compose build`). The migration moves each host to Docker's official `docker-ce` APT repo (29.x). Playbook: `playbooks/upgrade-docker-ce.yaml`. Order (least → most blast radius): 1. ~~**nh3-docker**~~ — **done 2026-04-24.** docker-ce 29.4.1, two stacks (portainer agent + portainer) restarted clean. Debug surfaced two unit-file gotchas now baked into the playbook: - `/etc/systemd/system/docker.service.d/override.conf` from the docker.io era hardcoded `/usr/sbin/dockerd` (docker-ce installs at `/usr/bin/dockerd`) → daemon failed status=203/EXEC. - The shipped docker-ce unit's `ExecStart=dockerd -H fd:// …` conflicts with `daemon.json` `hosts:` (defined for the 0.0.0.0:2375 homepage-discovery binding) → "conflicting host options". Playbook now backs up any existing override, probes `daemon.json` for `hosts:`, and installs an override that strips `-H` when needed. 2. ~~**esh-docker-vm**~~ — **done 2026-04-24.** docker-ce 29.4.1, all 16 stacks back up clean (adguard, beszel-agent, calibre-web, cloudflare-ddns, dockge, dozzle-agent, drawio, esphome, homeassistant, homepage, mosquitto, paperless-ngx, pgadmin, traefik, uptimekuma). Surfaced one verify-only nit: the docker-compose-plugin shipped with docker-ce 29 is renumbered v5.x (was v2.x with 26-28); same Compose v2 codebase, just realigned major. Playbook's compose-version verify was hardcoded to `v2.X.Y` and false-failed — now accepts any major ≥ 2. 3. ~~**ana-docker**~~ — **done 2026-04-24.** docker-ce 29.4.1, all 29 containers back up clean. Traefik routing live (200s for matrix.phasefinal.com presence + seafile.phasefinal.com syncs verified post-restart). Sub-finding (since fixed, see below): seafile's internal `seahub` raced mysql on startup and wedged. Also during the nh3-docker debug: `configs/homepage/docker.yaml` commented out the `irv-ml1-docker` provider — the 20s-per-poll ETIMEDOUTs from the stalled host were drowning homepage's logs and appeared to block ana-pfi-docker discovery (News group's Miniflux card wouldn't render until removal). Re-enable when irv-ml1 is back. ## traefik-postboot — host-level recovery for daemon-restart routing races Installed 2026-04-24 on both hosts that run traefik (esh-docker-vm, ana-docker). One-shot systemd unit (`traefik-postboot.service`, After=docker.service, Type=oneshot, sleep 60 → `docker restart traefik`) that fires once per boot. Forces a clean cold scan after the topology has settled, fixing the long-standing "traefik misses backends after host reboot" symptom (events-stream debounce + network- attach race when 30+ containers start in a burst). Playbook: `playbooks/install-traefik-postboot.yaml`. Tunables: `--var delay_seconds=N` (default 60), `--var container=NAME` (default `traefik`). Runs `systemd-analyze verify` non-disruptively so installing on a healthy host doesn't blip traefik. ## autorestic decommission — redundant with the new two-layer pipeline Hosts with autorestic detected 2026-04-24: `esh-docker-vm`, `esh-vm-db`. Both are pre-pipeline ESH manual installs at `/usr/local/bin/autorestic` + four systemd units backing up to `/mnt/backup/restic/repo/esh`. The new two-layer pipeline (PBS for VM images, structured restic profiles via `configs/restic//profiles.yaml`) covers everything autorestic was doing. Trigger: a typo on esh-docker-vm's `autorestic-backup.timer` line 2 (`D:escription` instead of `Description`) surfaced during the traefik-postboot install verify. Playbook: `playbooks/remove-autorestic.yaml`. **Done 2026-04-24** on both hosts after two playbook bug-fixes surfaced by the first run: (a) `rm path1 \` + newline + `path2` was bitten by YAML plain-scalar folding — the backslash-newline collapsed to a literal `\ `, so only the first path got passed to rm. Fixed by switching to a glob. (b) `shell: ! command -v autorestic >/dev/null` was parsed by YAML as a tag indicator with the `!` stripped, inverting the verify's exit code. Fixed by quoting the value. Deliberately untouched (decide separately): - `/srv/backups/autorestic/.autorestic.yml` — config; archival value - `/mnt/backup/restic/repo/esh` — historical autorestic snapshots; prune via `restic forget` once comfortable with new pipeline coverage. ## seafile seahub startup race — fixed 2026-04-24 When ana-docker rebooted (or any time the seafile container was recreated alongside seafile-mysql), seahub (the python frontend at port 8000 inside the seafile container) raced mariadb's startup. Short-form `depends_on: [db, memcached]` in compose only waits for the dependency container to be CREATED, not healthy — so seafile's `start.py` would hit `mysql Connection refused` and give up, leaving nginx serving seafile but proxying to a dead python backend. Symptom in traefik logs: ~7 of 30 requests returning 502 on seafile@docker dynamic endpoints; static assets fine. Fix applied directly to ana-docker's `/opt/docker/compose/seafile/ compose.yaml` (and into `stacks-mirror/`, which is gitignored — so not in the repo, but the file on the server is the source of truth): - Added `healthcheck` to the `db` service using the bundled `/usr/local/bin/healthcheck.sh --connect --innodb_initialized` (5s interval, 30s start_period, 30 retries). - Converted seafile's `depends_on` to long-form with `condition: service_healthy` for db (and `service_started` for memcached). Verified post-recreate: seahub log shows `Seahub is started ... Done.` without traceback; traefik `seafile@docker` dropped to 0 502s in the next 20s window. ## news-digest — per-item × button + cross-device hidden tray (2026-04-24) Daily Digest pages now have a small × on each item that hides it from the page. State persists server-side at `/output/hidden.json`, so the same hidden set follows the user across devices (home, ipad, laptop, work). A "Hidden (N)" tray at the bottom of each page lists items hidden FROM THE CURRENT PAGE with a restore button per row; older hidden ids that aren't on this page sit silently in `hidden.json` and continue to filter future editions that include the same article. Architecture change: `news-digest-web` swapped from `nginx:alpine` to a FastAPI app on uvicorn, built from the **same Dockerfile** as the worker. One image, two containers — the worker's cron-driven ENTRYPOINT runs in `news-digest-worker`, while compose overrides entrypoint+command to launch `uvicorn web:app` in `news-digest-web`. Drops one image dep, adds three endpoints: - `GET /api/hidden` — JSON array of hidden item ids - `POST /api/hide` — `{id}` → adds to hidden.json - `POST /api/restore` — `{id}` → removes from hidden.json Item ids are stable 12-char sha1 prefixes (`reddit:` / `miniflux:`) computed in `digest.py` at render time so hide-once means hide-forever-for-that-article. Storage: single JSON array, atomic writes via tempfile+rename, threading.Lock inside the single uvicorn worker. No auth — same trust boundary as the digest itself (LAN-only on port 8181). The deploy playbook also dropped its `DOCKER_BUILDKIT=0` fallback (no longer needed post docker-ce 29 migration) and gained three verify steps: `/api/hidden` returns a JSON array, `app.js` is reachable, and a synthetic `smoke-` id round-trips through hide → list → restore → list. ## llama-swap — added two vision-capable Qwen 3.6 entries (2026-04-24) - `qwen3.6-35-a3b-heretic` — `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF:Q6_K` (~28 GB) + auto-loaded BF16 mmproj for vision. - `qwen3.6-27b` — `unsloth/Qwen3.6-27B-GGUF:UD-Q6_K_XL` (~26 GB) + auto-loaded BF16 mmproj for vision (dense 27B, not MoE). Both pre-pulled into `/tank/aimodels/huggingface` (HF_HOME=/hfcache in the container) via a one-shot `python:3.12-slim` container running `huggingface_hub.snapshot_download` with `hf_transfer` for parallel chunked downloads — heretic's 29 GB landed in 4:10, unsloth's 26.5 GB in 3:46 (~118 MB/s each). Both repos include `mmproj-BF16.gguf` alongside the main GGUF and llama-server's `-hf` flag auto-loads it when present, so vision works without an explicit `--mmproj` path. ## Stack tree convention (canonical vs mirror) — clarified 2026-04-24 The workspace has two stack trees and they look similar enough that deploy-stack.sh was reading from the wrong one (`stacks-mirror/` instead of `stacks/`). Edits to `stacks/llama-swap/config.yaml` for the two new model entries above never reached ana-ml2 on the first deploy attempt because the diff was canonical-vs-mirror, not canonical-vs-server. Roles now nailed down explicitly in CLAUDE.md and `memory/convention_stacks_vs_mirror.md`: | tree | role | git | who writes | who reads | |---|---|---|---|---| | `stacks//` | canonical / intent | tracked | you / Claude | `deploy-stack.sh` | | `stacks-mirror///` | snapshot / reality | gitignored | `sync-stacks.sh` | drift inspection | Side fix: moved `stacks/llama-swap/config.yaml` → `stacks/llama-swap/conf/config.yaml` so the deploy mapping matches the server layout (`stacks//conf/` → `/opt/docker/conf//`). ## Session milestones — 2026-04-29 (Nevermore extraction) The news-digest applet was extracted into its own repo at [`vh/nevermore`](https://gitea.phasefinal.com/vh/nevermore) — clean break, fresh git history, renamed throughout (DIGEST_* → NEVERMORE_*, news-digest-* containers → nevermore-*, masthead → "Nevermore"). The existing `news-digest-*` deployment on ana-docker continues to run unchanged on its baked image; future updates come from the new repo. Migration path documented in nevermore's README. Removed from this workspace: stacks/news-digest/, playbooks/deploy-news-digest.yaml, scripts/add-digest-user.sh. ## Session milestones — 2026-04-28..29 (audio gen, digest applet, task-board polish, tooling) ### Audio generation on irv-ml1 (A6000) - **ACE-Step 1.5** (music gen, Apache 2.0) — port 8210. Hybrid diffusion + LLM, ~10-12 GB VRAM at inference. Custom `Dockerfile` patches upstream's torch/cu126 resolution bug; patched `infer-api.py` ships in the image (upstream's positional tuple is missing 6 of 24 pipeline args). Verified bench: 60 s clip in 14.8 s (~4× realtime). - **Stable Audio Open 1.0** (SFX/ambience, Stability Community license) — port 8211. Custom FastAPI shim wraps `diffusers.StableAudioPipeline`. ~4-6 GB VRAM in fp16, up to 47 s clips at 44.1 kHz. ### Daily Digest applet - **World + Local headline sections** above the existing tech/reddit cards. `seed-headlines.py` provisions the categories and curated feeds (BBC + NPR + Al Jazeera; LA Times + Voice of OC) in miniflux. - **Article-extracted summaries** via `trafilatura` — 2-3 sentence summaries grounded in real article content (was: title-paraphrase). Cached at `/output/.article-cache.json` with 7-day TTL. Adds ~30 s per digest run, parallel-prefetched. - **Per-desk collapse** chevron, localStorage-persisted per device. - **Mobile-responsive** — true 480 px breakpoint, fixed masthead overlap, cache-busted asset URLs. - **Multi-tenant** via `scripts/add-digest-user.sh ` — shared miniflux + per-user digest stack (own port, hide-state, cron). `DIGEST_CRON_AM` / `DIGEST_CRON_PM` are now env-driven; each teammate's stack fires on their own hours. ### task-board - **v0.1.11 pending_shells / orange treatment.** New `task_set_shells` MCP tool + `pending_shells` field on the Task model (schema v6, ALTER ADD COLUMN). When a card is `state=waiting` AND has non-empty shells, it flips from blue (waiting on user) to orange (waiting on background work). Cards with non-empty shells are excluded from auto-dormant promotion. - **v0.1.12 card-bar rhythms** — three distinct pulse cadences (active 2.4 s, waiting 3.6 s patient breath, attention 1.6 s louder) so column scan reads at a glance. ### Tooling - **`scripts/fork-fleet.sh`** — bootstrap a new fleet-management workspace from this one. Mirrors reusable tooling, strips fleet-specific content, regenerates skeleton CLAUDE.md / README / STATUS, fresh git history. - **CLAUDE.md `task_set_shells` discipline** — assistant must call `task_set_shells` on any `Bash run_in_background` (and clear when done) so the orange treatment fires; hooks have no way to enumerate Claude Code's bg tasks externally. ### New stack on ana-docker - **memos** (port 5230) — stock `neosmemo/memos:stable`, SQLite at `/opt/docker/conf/memos/data/`, homepage label `Notes` (auto-discovered). ## Session milestones — 2026-04-27 (irv-ml1 recovery + new GPU deploys) ### irv-ml1 fully recovered - Host alive, driver 595.58.03 / CUDA 13.2, both GPUs detected, kernel modules loaded. - 5 pre-existing GPU stacks (comfyui, cosyvoice, qwen3-tts, index-tts, parakeet) restarted clean and report healthy. - Homepage `irv-ml1-docker` provider re-enabled in `configs/homepage/docker.yaml`, pushed via the rsync workflow. ### Three new GPU stacks deployed on irv-ml1 - **Kokoro GPU variant** — first try succeeded once the `.env.example` default was flipped from `cpu` → `gpu` (matching the now-working driver). `KOKORO_VARIANT=gpu` / `KOKORO_USE_GPU=true` / `KOKORO_GPU_DEVICES=0` (pinned to RTX 3090; ~1 GB VRAM, doesn't need the A6000). Serving on irv-ml1:8193. - **VibeVoice 1.5B** — needed two iterations: - `VIBEVOICE_SHA=7614c469a145` (12-char short) made docker buildx's git source resolver error with "repository does not contain ref 7614c469a145" — same commit IS still HEAD of main, but buildx requires the FULL 40-char SHA even when the short is unambiguous. Now `7614c469a145dafc30b1fd5f6a6046ea831dbd6a`. - The verify step greped `/v1/audio/voices` for `voices`/`voice`/ `alloy`/`Carter` literals, but VibeVoice's actual response shape is OpenAI list-format `{"object":"list","data":[]}`. Switched to JSON-parse + assert `object="list"` and `data` is a list. Robust against an empty voices set on a fresh install. - Serving on irv-ml1:8194. - **Chatterbox Turbo** (devnen wrapper) — needed three iterations: - Old playbook used `dockerfile: docker/Dockerfile.gpu`; devnen restructured the repo to put Dockerfiles at root, renamed by CUDA version (`Dockerfile.cu128`, `.cpu`, `.rocm`). Switched to `Dockerfile.cu128` (GPU build for CUDA 12.8 toolkit; works with irv-ml1's 595.58.03 driver via backward-compat). - Pinned `CHATTERBOX_SHA` from `main` to the current SHA `68213e75c248f4f4bbd185f7c20ec02722fdc79b` so future upstream churn doesn't break us without warning. - Wait + verify steps probed `/health`, but devnen's wrapper doesn't expose that route (404). Switched to `/api/model-info` which returns `{"loaded":true,...}` only after the model finishes loading — doubles as liveness + readiness. Compose healthcheck + verify both updated. Verify of `/v1/audio/voices` also fixed to parse the wrapper's actual response shape `{"status":"ok","voices":[...]}` (note: NOT the same OpenAI list shape vibevoice uses). - Serving on irv-ml1:8196. ### llama-swap pin/unload cleanup - Removed `ttl: 600` from `qwen3.6-35-a3b-abliterated`, `qwen3.6-35-a3b-heretic`, `qwen3.6-27b` so they stay loaded across idle (still evictable when another non-coexistent model is requested). - `qwen3.6-35-a3b` removed from the `pinned` group — its `persistent: true` flag was preventing eviction under memory pressure, OOM-ing other model loads. Now sits with `ttl: 0` outside any group: never idle-unloads but is evictable. - `granite-4-small` added to the `pinned` group alongside `qwen3.5-9b` so it stops swapping with `qwen3.6-27b` when both are used concurrently. ~5-6 GB at Q4_K_M + 120K KV ≈ comfortable inside the existing pin budget. ### Backup verification post-recovery All three layers green as of 2026-04-27: - **Per-host restic** (`resticprofile-backup@profile-default.timer`): all 6 hosts (ana-docker, esh-docker-vm, nh3-docker, esh-vm-db, ana-ml2, irv-ml1) ran successfully at 01:00 today. The migration to docker-ce on three of these hosts didn't disrupt the timers. irv-ml1's timer fired right after recovery. - **PBS-ANA**: every namespace (`ana-pve`, `esh-pve`, `nh3-pve`, `sfsrv-pve`) has fresh 2026-04-27 snapshots for every VM/CT. - **PBS-NH3 (DR mirror)**: synced — 2026-04-27 backups present in /mnt/pbs-datastore/ns/. NH3 NAS at 27TB / 42TB used (64%, healthy headroom). Note: backrest's UI shows `recentBackups: {}` for all repos because backrest in this setup is a viewer with **zero plans configured** — the actual orchestration is the per-host resticprofile timers. Backrest's only configured operations are `_unassociated_` (manual one-off pushes from the UI) and `_system_` housekeeping. Don't be alarmed by the empty dashboard — check the per-host timers instead. ## Session milestones — 2026-04-24 (the "tooling day" + housekeeping pm) Morning / early afternoon — the original tooling day: - **irv-ml1 AI stacks deployed** as Docker: ComfyUI (port 8188, host- writable workflows), Parakeet ASR (port 8765, rewritten on sherpa-onnx after the Shadowfita FastAPI wrapper hit two unfixed upstream bugs), CosyVoice 3 (port 8190, GLaDOS voice cloned + live). - **irv-ml1 restic profile extended** to cover `/worktank/{comfyui/basedir/{user,custom_nodes,input},cosyvoice/voices}`; bulk weights + scratch dirs stay excluded. - **`scripts/elway`** shipped — ~800-line Python playbook runner with tier 1 + tier 2 idempotency; handlers / register / multi-host / content-hash are deferred (gitea #3–#6). - **task-board** built end-to-end (separate repo, `vh/task-board` on gitea) and shipped as a Claude Code plugin. Green/red cards per session via UserPromptSubmit + Stop hooks; four MCP tools expose explicit activity tracking. - **pfi-postgres NFS decoupling** finished item 4b — zero residual dependency on ana-nas for that VM. Late afternoon / evening — task-board iteration + STATUS sweep: - **task-board v0.1.1 → v0.1.3** shipped over four iterations on the live ANA deployment: - **v0.1.1** — hooks parse Claude Code's stdin JSON for `session_id` and append a short suffix when `TASK_BOARD_SESSION` isn't set, so two sessions in one project no longer collide on a single card. Dormant transition preserves cumulative idle time (`state_entered_at = last_update_at` instead of `now`). - **v0.1.2** — favicon (3-column SVG in active/waiting/dormant state colors); served at `/static/favicon.svg` with a `` and a `/favicon.ico` route returning the same SVG. Followup fix for an XML-illegal `--` in a comment. - **v0.1.2-followup** — UI live-duration ticker bumped from 5 s → 1 s (humanDuration floors to integer seconds; cheap render). - **v0.1.3** — case-insensitive session names. `sessions.name COLLATE NOCASE`; real ALTER migration (not a wipe) — keeps earliest- `created_at` row as canonical, reassigns child comments. Write path canonicalizes session label before inserting comments. Read path uses `COLLATE NOCASE` for safety on external API callers. - **Parakeet (irv-ml1) healthcheck fix** — image ships `wget` not `curl`; healthcheck swap, 2,190 failing checks → healthy. - **AIPA-MCP project session label fix** — set `TASK_BOARD_SESSION= Architect` in `.claude/settings.json`, updated the project's `CLAUDE.md` to specify `session="Architect"` for explicit MCP calls, and renamed the existing `AIPA-MCP` card → `Architect` in the live SQLite (1 session row + 25 comment rows preserved). - **STATUS items 11 / 13 / 15 / 16 closed.** Mattermost dir gone on ana-docker (verified); UniFi UDM cards added to homepage (PFI-UDMSE for NH3 edge replacing the retired Fortigate 101F, ESH-UDMPM for ESH); docs nav map added + stale chromadb-setup.md removed + VM-102 Matrix docs frontmatter stripped; STATUS.md discipline rule recorded. ## Session milestones — 2026-04-20 / 2026-04-21 - Homepage reorganized to function-first with tabs (Main / Infrastructure / Toolchain). - Per-group icons + equal-height layout + 4-column grids. - Fleet-wide label sweep (function groups, Service Networking renamed from Wiring/Plumbing due to homepage-parser slash bug). - 6 restic clients deployed end-to-end (client creds, repo init, profile install, systemd timers, first backups verified). - 9 hosts registered under `servers/` from gap-analysis (PFI VMs + SureFire tenant, with tenancy awareness). - Discovery scripts (FortiGate SSH + UniFi cloud API + TSV diff vs inventory). - Proxmox inspect script + fleet-wide refresh wrapper. - Calibre-Web-Automated migration replacing calibre + calibre-web pair. - llama-swap Qwen3.6 stock + mradermacher abliterated, 128K ctx. - Mattermost retired (not running, compose dir cleanup pending). - FortiGate 101F at NH3 retired; homepage card removed. ## Memory pointers (for future Claude sessions) Relevant `~/.claude/.../memory/` entries: - `server_split.md` — host placement rules - `feedback_ssh_sudo.md` — use `ssh -t` for remote sudo (mostly superseded by elway, but still applies to ad-hoc ssh) - `feedback_git_autonomous.md` — handle git commits without asking - `feedback_git_commits.md` — no Claude attribution in commit messages - `feedback_use_elway.md` — write elway playbooks; don't chain ssh+sudo - `feedback_backups_must_not_risk_production.md` — rule adopted after the 2026-04-23 ana-nas self-backup crash - `project_backup_pipeline_gaps.md` — user's explicit goal of "all hosts + configs + DBs backed up" - `project_db_migrate_off_nfs.md` — DB-off-NFS decision + status - `project_surefire_tenant.md` — SureFire tenancy boundary awareness - `reference_gitea_remote.md` — origin is `vh/esh-pfi-infrastructure` on gitea.phasefinal.com; `tea` CLI logged in as `vh` - `reference_task_board.md` — task-board plugin + tools contract - `storage_ana_nas.md` — ana NAS is Debian LXC (CT 109), not TrueNAS - `storage_nh3_nas.md` — NH3 NAS via `syncuser`, not `admin` - `incident_ana_nas_spof.md` — blast-radius matrix for ana-nas outages