# Status + Open Issues Last updated: 2026-04-24 Snapshot of fleet state and open work. Refresh this file when a pass of significant work lands — don't let it drift quietly. **New session? Read [`docs/orientation.md`](docs/orientation.md) first.** ## What's in place ### Backup coverage (2-layer, fully operational) - **PBS fleet-wide.** All 5 hypervisors (pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) → `pbs-ana` (primary, VM on pfi-pve with NFSv3 datastore on ana-nas). Per-hypervisor namespaces to avoid VMID collisions. DR mirror at `pbs-nh3` (VM on nh3-pve with NFSv3 datastore on nh3-nas Synology) pulls daily at 06:00. Verify jobs armed on both sides. - **restic 8/8 hosts** with resticprofile + systemd timers at 01:00: | Host | Target rest-server | DB hooks | |---|---|---| | ana-docker | rest-server-ana (local) | synapse, seafile, vaultwarden-pg, gitea (native dump), openwebui | | ana-ml2 | rest-server-ana (cross-site) | — (no DBs) | | nh3-docker | rest-server-nh3 (local, Synology) | — | | esh-docker-vm | rest-server-ana (cross-site) | paperless-pg, HA SQLite, CWA SQLite, pgadmin SQLite, uptime-kuma SQLite | | vm-esh-nas | rest-server-ana (cross-site) | — | | nh3-dev (workstation) | rest-server-nh3 (local) | — | | irv-ml1 | rest-server-nh3 (via WG) | — | | esh-vm-db | rest-server-ana (cross-site) | pg_dumpall, mongodump | - **Cross-site restic rsync.** `ana-nas → nh3-nas` at 04:00 daily (mirrors rest-server-ana data). `nh3-nas → ana-nas` at 05:00 daily (mirrors rest-server-nh3 data, runs as root since DSM writes files as admin mode 400). Tracked at `configs/rsync/{ana-nas-to-nh3,nh3-nas-to-ana}/`. ### Inventory - **~22 tracked hosts** under `servers/` across ANA / NH3 / ESH / IRV sites. SSH config aliases in `~/.ssh/config` for every host — `ssh ` just works. - **Homepage** at — function-first layout (Main / Infrastructure / Toolchain tabs), per-group icons, four Infra sub-groups (ANA / NH3 / IRV / ESH). Docker auto-discovery active for 5 hosts (ana-docker, ana-ml2, nh3-docker, esh-docker-vm, irv-ml1). - **FortiGate + UniFi discovery scripts** produce TSVs; gap analysis against `servers/*/` for unmanaged IPs. ### Architecture decisions (durable) - **DB data on local disk, not NFS.** pfi-postgres migrated 2026-04-23; fstab + NFS mount removed 2026-04-24 (pfi-postgres now fully decoupled from ana-nas). Mongo on esh-vm-db confirmed already local. Removes the biggest ana-nas blast-radius risk. Memory: `project_db_migrate_off_nfs.md`. - **Backups must not risk production.** Rule adopted after 2026-04-23 ana-nas self-backup crash. Memory: `feedback_backups_must_not_risk_production.md`. - **offen sidecars retired fleet-wide 2026-04-23.** restic covers equivalent scope; reclaimed 16 GB of redundant tarballs on `/mnt/backup/docker/esh-vm-docker/`. - **Gitea remote for this repo (2026-04-23).** `origin` is `vh/esh-pfi-infrastructure` on `gitea.phasefinal.com`. Also: `vh/task-board` (new 2026-04-24) hosts a Claude Code plugin + marketplace serving the assistant task-state dashboard. - **Prefer elway for multi-step SSH work (2026-04-24).** The new `scripts/elway` mini-playbook runner replaces chained `ssh -t sudo …` commands — handles sudo once up front, structured reporting, idempotency (creates/when/changed_when). Memory: `feedback_use_elway.md`; template playbook: `playbooks/elway-smoke.yaml`. ### Tooling - `scripts/server_inspect.sh` + `refresh-server-info.sh` for Docker hosts. - `scripts/proxmox_inspect.sh` + `refresh-proxmox-info.sh` for PVE nodes (reports VMs/LXCs/storage/backup-coverage). - `scripts/discover-fortigate.sh`, `scripts/discover-unifi.sh`, `scripts/discover-gaps.sh` for network-level inventory discovery. - `scripts/deploy-stack.sh` + `scripts/sync-stacks.sh` for compose push/pull. - `scripts/add-host.sh` for new-host registration. - **`scripts/elway`** (2026-04-24) — mini-ansible playbook runner over SSH. Playbooks live under `playbooks/`. Tier 1 (creates/when) + tier 2 (changed_when) idempotency; handlers + register + multi-host fan-out deferred as gitea issues #3–#5. - **`tea` CLI** at `/usr/local/bin/tea` — gitea CLI, already logged in as `vh`. Use for issue / PR work instead of inventing URLs. ## Open issues ### 🟥 Blocked — irv-ml1 stalled after CUDA driver upgrade (2026-04-25) Driver bump on irv-ml1 (570.124.06 → 595.58.03 via NVIDIA's CUDA APT repo) was attempted to unblock Kokoro's GPU image (which requires CUDA ≥ 12.9; old driver capped at 12.8). The host did not come up cleanly after the post-install reboot. **Stop touching irv-ml1 until this is resolved.** **What was attempted**: `scripts/upgrade-irv-ml1-cuda.sh phase1` on 2026-04-25. Phase 1 is supposed to: snapshot current nvidia packages → stop GPU containers → `apt install cuda-drivers` → DKMS rebuild → reboot. Whether all those steps actually completed before the unclean boot is unknown — the snapshot file at `/tmp/nvidia-pre-upgrade.txt` on irv-ml1 (if it survived) will say. **Recovery options** (in increasing invasiveness): 1. **Console diagnose first.** SSH may or may not work depending on the failure mode. Check via console (irv-ml1 is bare-metal, has physical access via Lenovo IPMI). Look at `journalctl -b -1` from the previous boot to see what failed: DKMS build failure, kernel module load error, nvidia-container-toolkit init regression, etc. 2. **Rollback the driver** if the host is reachable but GPU is broken: ``` scripts/upgrade-irv-ml1-cuda.sh rollback ``` This reinstalls the snapshot from `/tmp/nvidia-pre-upgrade.txt` with `--allow-downgrades` and reboots. Restores 570.124.06. 3. **Manual driver pin** if the script can't recover: ``` ssh irv-ml1 sudo apt install --allow-downgrades \ nvidia-driver=570.124.06-1 nvidia-driver-cuda=570.124.06-1 \ nvidia-driver-libs=570.124.06-1 nvidia-kernel-open-dkms=570.124.06-1 ssh irv-ml1 sudo apt-mark hold nvidia-driver nvidia-driver-cuda \ nvidia-driver-libs nvidia-kernel-open-dkms ssh irv-ml1 sudo systemctl reboot ``` 4. **Try a different driver version** if 595 is genuinely incompatible with kernel 6.1.0-37: 580 series may build cleanly. Check via `apt-cache madison cuda-drivers` after recovery. **Once irv-ml1 is healthy again**, deferred work to revisit: - **Kokoro GPU variant**: deployed CPU works as a fallback. Switch `.env` to `KOKORO_VARIANT=gpu` / `KOKORO_USE_GPU=true` / `KOKORO_GPU_DEVICES=0` and redeploy via `scripts/elway irv-ml1 --playbook playbooks/deploy-kokoro.yaml`. - **VibeVoice 1.5B deploy** (long-form / multi-speaker dialogue): `scripts/elway irv-ml1 --playbook playbooks/deploy-vibevoice.yaml`. Stack files at `stacks/vibevoice/`. ~12 min build + ~7 GB model. - **Chatterbox Turbo deploy** (low-latency English voice cloning): `scripts/elway irv-ml1 --playbook playbooks/deploy-chatterbox.yaml`. Stack files at `stacks/chatterbox/`. ~10 min build + ~6 GB model. - **Verify the existing 5 GPU stacks still work** post-upgrade: comfyui, cosyvoice, qwen3-tts, index-tts, parakeet. They were all Up + healthy before the driver attempt. Smoke-test each. - **Re-evaluate Kokoro variant default** after recovery. If a 595+ series driver doesn't end up working, the `.env.example` default of `KOKORO_VARIANT=cpu` is correct as-is. If a working GPU driver lands, flip the default to `gpu` for future deploys. **Don't run on irv-ml1 until this is closed**: - Any of the three deploy playbooks above - Any host-state-mutating playbook against irv-ml1 - The driver upgrade script itself unless intentionally retrying Once recovered, update this section with what fixed it and remove the 🟥 marker. ### 🟥 Quick wins (do next) 1. ~~**Fix Backrest's `esh-docker-vm` URI**~~ — done. 2. ~~**Push SSH keys + pull snapshots for 6 unrefreshed hosts**~~ — **done 2026-04-23**. 4 VMs (pfi-ana-webhost, pfi-pteradactyl, pfi-tacticalrmm, pfi-postgres) took the workstation key via `ssh-copy-id` with `lkraven@`. 2 LXCs (ana-filebot, ana-wg) needed key installed via `pct push` from pfi-pve because `PermitRootLogin prohibit-password` blocked `ssh-copy-id`. Minor: pfi-pteradactyl's `server_inspect` Docker section is blank because lkraven is not in the `docker` group there — `sudo usermod -aG docker lkraven` + re-login to fix. 3. ~~**Patch out `forget` schedules from all 6 restic profiles**~~ — done. 3b. ~~**Discover sf-r630 OS-side IP**~~ — **resolved 2026-04-23**: the R630 with iDRAC `10.250.250.110` is the same physical box that runs `sfsrv-ana` (Proxmox VE at `10.250.250.115`). No separate OS IP to find. `servers/sf-r630/` now clarifies it as the hardware/BMC-only inventory entry; `servers/sfsrv-ana/` is the OS view. `servers/ana-ml2/README.md` updated with its own distinct BMC IP (`10.250.250.50`, Supermicro) to prevent future confusion between the two physical chassis. 4. _(removed — see note above)_ ### 🟧 Real work (dedicated session each) 4b. ~~**Migrate DB data directories off NFS onto local VM disk.**~~ — **done 2026-04-23/24.** Postgres on pfi-postgres migrated 04-23; mongodb on esh-vm-db confirmed already local (and serves zero user data in practice — paperless uses Postgres 15 on the same VM). fstab entry + NFS mount for `/mnt/db` on pfi-postgres removed 2026-04-24 via `playbooks/decouple-pfi-postgres-from-ana-nas.yaml`. pfi-postgres has zero remaining dependency on ana-nas. Only residue: cold archive of pre-migration data still on ana-nas (`/mnt/db/pfi-*`); harmless, can sit indefinitely. 5. ~~**Rotate exposed secrets**~~ — **done 2026-04-23**. All six rotated: vaultwarden/gitea/paperless-ng Postgres passwords (hardcoded `compose.yaml` literals moved to gitignored `.env` files in the process), ana-docker + ana-ml2 + esh-docker-vm rest-server htpasswd entries, and rest-server repo passphrases for ana-docker + esh-docker-vm (ana-ml2's was already rotated during prior wipe+reinit). Also discovered along the way: paperless uses a separate `esh-vm-db` VM (10.0.50.60), not pfi-postgres as the old notes implied. 6. ~~**Cross-site rsync**~~ — **done 2026-04-23, both directions**. - **ana → nh3** (04:00 daily): `ana-nas:/mnt/backup/restic/repo/ana/` → `nh3-nas:/volume1/Backup/restic-ana-mirror/`. Runs on ana-nas as lkraven. 15.6 GB initial sync completed 07:18 UTC. - **nh3 → ana** (05:00 daily): `nh3-nas:/volume1/Backup/restic/` → `ana-nas:/mnt/backup/restic-nh3-mirror/`. Runs on nh3-nas as root (rest-server-nh3 container writes mode-400 files; only root can read them on Synology). 2.31 GB initial sync completed 07:42 UTC. - Tracked at `configs/rsync/ana-nas-to-nh3/` and `configs/rsync/nh3-nas-to-ana/` respectively. 6b. **PBS deployment across the fleet.** Runbook at `docs/runbooks/pbs-deployment.md`. Phases 0–6 **done** (2026-04-22): PBS-ANA on pfi-pve with NFS datastore on Ana NAS (NFSv3 for ZFS case-insensitivity), PBS-NH3 on nh3-pve with NFS datastore on NH3 Synology (NFSv3 + Linux-mode POSIX 777 to avoid syno_acl interference), per-hypervisor namespaces, API tokens, verify jobs on both, one-way sync ANA → NH3 at 06:00 daily, and all 5 hypervisors (pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) onboarded to PBS-ANA. Remaining phases: - Phase 7-8 — one week burn-in, then retire legacy vzdump targets on each hypervisor (keep until 2026-04-29 earliest) - Phase 9 — homepage cards for PBS-ANA + PBS-NH3 web UIs, memory/doc updates, `servers/pbs-ana` + `servers/pbs-nh3` dirs 7. ~~**SureFire tenant backup plan decision**~~ — **resolved 2026-04-23** by folding sfsrv-ana into the fleet-wide PBS deployment (dedicated `sfsrv-ana` namespace on PBS-ANA, replicates to PBS-NH3 via the same sync job as the rest of the fleet). Hosting-agreement option chosen: PFI provides backup coverage as part of managed hosting. ### 🟨 Prereqs / polish 8. ~~**Synology SSH setup**~~ — **done 2026-04-22**. Dedicated `syncuser` account (admin-group membership) with key auth, registered as `servers/nh3-nas/`, reachable as `ssh nh3-nas`. Unlocks cross-site rsync (item 6), rest-server-nh3 healthcheck deploy, and `.htpasswd` edits on the NH3 side. 9. ~~**`scripts/restic-prune.sh`**~~ — **done 2026-04-24.** Quarterly disk-hygiene tool. Drops `--append-only` on the rest-server (via a temporary `docker-compose.override.yaml` — never edits the canonical compose), runs `resticprofile forget --prune --verbose` on each client, restores `--append-only` (with `trap` so it runs even on partial failure). ANA side fully automated (5 clients); NH3 side prints a manual ceremony because DSM Container Manager + sudo on syncuser aren't cleanly scriptable from this workstation. Run with `scripts/restic-prune.sh ana|nh3|all`, optionally `--dry-run`. 10. ~~**Retire `offen/docker-volume-backup` sidecars**~~ — **done 2026-04-23**. Removed from paperless-ngx and pgadmin composes on esh-docker-vm (only hosts in the fleet that had them). 16 GB of orphan tarballs at `/mnt/backup/docker/esh-vm-docker/` reclaimed. Restic coverage verified equivalent: `/var/lib/docker/volumes` in source + pre-backup hook handles the DB dumps for paperless (against esh-vm-db), HA, pgadmin, CWA, uptime-kuma. Obsolete `*_offen_backup_data` exclude also removed from the restic profile. 11. ~~**Clean up retired mattermost dir** on ana-docker~~ — **done** (verified 2026-04-24: `/opt/docker/compose/mattermost/` does not exist; no mattermost containers anywhere on the host). ### 🟩 Research / deferred / intermittent 12. **Backrest UI intermittent timeout.** Backend is confirmed healthy (direct `curl GetConfig` returns in 200ms). UI chokes on the same endpoint. Likely cause: SSE stream wedge in browser, or service worker stale. Workaround: Ctrl+Shift+R. Root cause investigation deferred. 13. ~~**UniFi controller homepage cards**~~ — **done 2026-04-24.** Both cards added to `configs/homepage/services.yaml` and pushed to esh-docker-vm: `PFI-UDMSE` (10.100.0.1, UDM Pro SE) under `Infra - NH3` as the new edge device replacing the retired Fortigate 101F; `ESH-UDMPM` (10.0.0.1, UDM Pro Max) under `Infra - ESH`. Icon `si-ubiquiti`. 14. **Prune + credential-rotation scripts** as repeatable tooling (vs per-incident manual work). ### 🟦 Memory / documentation housekeeping 15. ~~**`docs/` organization**~~ — **done 2026-04-24.** First-pass landed earlier (misfiled `tea-*.sha256` removed; `docs/README.md` nav map added). Second pass landed same day: stripped the broken YAML frontmatter from both VM-102 Matrix docs (the `path:` values pointed at `docs/pfi-ana/...` which doesn't exist in this repo, and no toolchain consumed the metadata); deleted `pfi/chromadb-setup.md` (referenced `configs/pfi-ana/...` and `scripts/setup-chromadb.sh`, both nonexistent — deployment is long done and the operational truth lives in `docker-stack.md`). Kept the two VM-102 docs separate by design (each is right-sized; a merge would push past the 500-line guideline in the README). 16. ~~**`STATUS.md` drift discipline**~~ — **done 2026-04-24.** Refreshed with the post-tooling-day session work; `status-regen.sh` idea dropped — STATUS.md is intentionally narrative, not derivable from git/code, so auto-regen would lose information. Discipline rule: refresh STATUS.md at the end of any session where a 🟧 or 🟥 item closes, or three+ smaller items land. ## Active migration — docker.io 20.10 → docker-ce 29.x Bookworm's Debian-packaged `docker.io` is pinned at 20.10.24 with client/daemon API 1.41. Newer compose clients (1.52+) refuse to talk to it ("client version 1.52 is too new" during `docker compose build`). The migration moves each host to Docker's official `docker-ce` APT repo (29.x). Playbook: `playbooks/upgrade-docker-ce.yaml`. Order (least → most blast radius): 1. ~~**nh3-docker**~~ — **done 2026-04-24.** docker-ce 29.4.1, two stacks (portainer agent + portainer) restarted clean. Debug surfaced two unit-file gotchas now baked into the playbook: - `/etc/systemd/system/docker.service.d/override.conf` from the docker.io era hardcoded `/usr/sbin/dockerd` (docker-ce installs at `/usr/bin/dockerd`) → daemon failed status=203/EXEC. - The shipped docker-ce unit's `ExecStart=dockerd -H fd:// …` conflicts with `daemon.json` `hosts:` (defined for the 0.0.0.0:2375 homepage-discovery binding) → "conflicting host options". Playbook now backs up any existing override, probes `daemon.json` for `hosts:`, and installs an override that strips `-H` when needed. 2. ~~**esh-docker-vm**~~ — **done 2026-04-24.** docker-ce 29.4.1, all 16 stacks back up clean (adguard, beszel-agent, calibre-web, cloudflare-ddns, dockge, dozzle-agent, drawio, esphome, homeassistant, homepage, mosquitto, paperless-ngx, pgadmin, traefik, uptimekuma). Surfaced one verify-only nit: the docker-compose-plugin shipped with docker-ce 29 is renumbered v5.x (was v2.x with 26-28); same Compose v2 codebase, just realigned major. Playbook's compose-version verify was hardcoded to `v2.X.Y` and false-failed — now accepts any major ≥ 2. 3. ~~**ana-docker**~~ — **done 2026-04-24.** docker-ce 29.4.1, all 29 containers back up clean. Traefik routing live (200s for matrix.phasefinal.com presence + seafile.phasefinal.com syncs verified post-restart). Sub-finding (since fixed, see below): seafile's internal `seahub` raced mysql on startup and wedged. Also during the nh3-docker debug: `configs/homepage/docker.yaml` commented out the `irv-ml1-docker` provider — the 20s-per-poll ETIMEDOUTs from the stalled host were drowning homepage's logs and appeared to block ana-pfi-docker discovery (News group's Miniflux card wouldn't render until removal). Re-enable when irv-ml1 is back. ## traefik-postboot — host-level recovery for daemon-restart routing races Installed 2026-04-24 on both hosts that run traefik (esh-docker-vm, ana-docker). One-shot systemd unit (`traefik-postboot.service`, After=docker.service, Type=oneshot, sleep 60 → `docker restart traefik`) that fires once per boot. Forces a clean cold scan after the topology has settled, fixing the long-standing "traefik misses backends after host reboot" symptom (events-stream debounce + network- attach race when 30+ containers start in a burst). Playbook: `playbooks/install-traefik-postboot.yaml`. Tunables: `--var delay_seconds=N` (default 60), `--var container=NAME` (default `traefik`). Runs `systemd-analyze verify` non-disruptively so installing on a healthy host doesn't blip traefik. ## autorestic decommission — redundant with the new two-layer pipeline Hosts with autorestic detected 2026-04-24: `esh-docker-vm`, `esh-vm-db`. Both are pre-pipeline ESH manual installs at `/usr/local/bin/autorestic` + four systemd units backing up to `/mnt/backup/restic/repo/esh`. The new two-layer pipeline (PBS for VM images, structured restic profiles via `configs/restic//profiles.yaml`) covers everything autorestic was doing. Trigger: a typo on esh-docker-vm's `autorestic-backup.timer` line 2 (`D:escription` instead of `Description`) surfaced during the traefik-postboot install verify. Playbook: `playbooks/remove-autorestic.yaml`. **Done 2026-04-24** on both hosts after two playbook bug-fixes surfaced by the first run: (a) `rm path1 \` + newline + `path2` was bitten by YAML plain-scalar folding — the backslash-newline collapsed to a literal `\ `, so only the first path got passed to rm. Fixed by switching to a glob. (b) `shell: ! command -v autorestic >/dev/null` was parsed by YAML as a tag indicator with the `!` stripped, inverting the verify's exit code. Fixed by quoting the value. Deliberately untouched (decide separately): - `/srv/backups/autorestic/.autorestic.yml` — config; archival value - `/mnt/backup/restic/repo/esh` — historical autorestic snapshots; prune via `restic forget` once comfortable with new pipeline coverage. ## seafile seahub startup race — fixed 2026-04-24 When ana-docker rebooted (or any time the seafile container was recreated alongside seafile-mysql), seahub (the python frontend at port 8000 inside the seafile container) raced mariadb's startup. Short-form `depends_on: [db, memcached]` in compose only waits for the dependency container to be CREATED, not healthy — so seafile's `start.py` would hit `mysql Connection refused` and give up, leaving nginx serving seafile but proxying to a dead python backend. Symptom in traefik logs: ~7 of 30 requests returning 502 on seafile@docker dynamic endpoints; static assets fine. Fix applied directly to ana-docker's `/opt/docker/compose/seafile/ compose.yaml` (and into `stacks-mirror/`, which is gitignored — so not in the repo, but the file on the server is the source of truth): - Added `healthcheck` to the `db` service using the bundled `/usr/local/bin/healthcheck.sh --connect --innodb_initialized` (5s interval, 30s start_period, 30 retries). - Converted seafile's `depends_on` to long-form with `condition: service_healthy` for db (and `service_started` for memcached). Verified post-recreate: seahub log shows `Seahub is started ... Done.` without traceback; traefik `seafile@docker` dropped to 0 502s in the next 20s window. ## news-digest — per-item × button + cross-device hidden tray (2026-04-24) Daily Digest pages now have a small × on each item that hides it from the page. State persists server-side at `/output/hidden.json`, so the same hidden set follows the user across devices (home, ipad, laptop, work). A "Hidden (N)" tray at the bottom of each page lists items hidden FROM THE CURRENT PAGE with a restore button per row; older hidden ids that aren't on this page sit silently in `hidden.json` and continue to filter future editions that include the same article. Architecture change: `news-digest-web` swapped from `nginx:alpine` to a FastAPI app on uvicorn, built from the **same Dockerfile** as the worker. One image, two containers — the worker's cron-driven ENTRYPOINT runs in `news-digest-worker`, while compose overrides entrypoint+command to launch `uvicorn web:app` in `news-digest-web`. Drops one image dep, adds three endpoints: - `GET /api/hidden` — JSON array of hidden item ids - `POST /api/hide` — `{id}` → adds to hidden.json - `POST /api/restore` — `{id}` → removes from hidden.json Item ids are stable 12-char sha1 prefixes (`reddit:` / `miniflux:`) computed in `digest.py` at render time so hide-once means hide-forever-for-that-article. Storage: single JSON array, atomic writes via tempfile+rename, threading.Lock inside the single uvicorn worker. No auth — same trust boundary as the digest itself (LAN-only on port 8181). The deploy playbook also dropped its `DOCKER_BUILDKIT=0` fallback (no longer needed post docker-ce 29 migration) and gained three verify steps: `/api/hidden` returns a JSON array, `app.js` is reachable, and a synthetic `smoke-` id round-trips through hide → list → restore → list. ## llama-swap — added two vision-capable Qwen 3.6 entries (2026-04-24) - `qwen3.6-35-a3b-heretic` — `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF:Q6_K` (~28 GB) + auto-loaded BF16 mmproj for vision. - `qwen3.6-27b` — `unsloth/Qwen3.6-27B-GGUF:UD-Q6_K_XL` (~26 GB) + auto-loaded BF16 mmproj for vision (dense 27B, not MoE). Both pre-pulled into `/tank/aimodels/huggingface` (HF_HOME=/hfcache in the container) via a one-shot `python:3.12-slim` container running `huggingface_hub.snapshot_download` with `hf_transfer` for parallel chunked downloads — heretic's 29 GB landed in 4:10, unsloth's 26.5 GB in 3:46 (~118 MB/s each). Both repos include `mmproj-BF16.gguf` alongside the main GGUF and llama-server's `-hf` flag auto-loads it when present, so vision works without an explicit `--mmproj` path. ## Stack tree convention (canonical vs mirror) — clarified 2026-04-24 The workspace has two stack trees and they look similar enough that deploy-stack.sh was reading from the wrong one (`stacks-mirror/` instead of `stacks/`). Edits to `stacks/llama-swap/config.yaml` for the two new model entries above never reached ana-ml2 on the first deploy attempt because the diff was canonical-vs-mirror, not canonical-vs-server. Roles now nailed down explicitly in CLAUDE.md and `memory/convention_stacks_vs_mirror.md`: | tree | role | git | who writes | who reads | |---|---|---|---|---| | `stacks//` | canonical / intent | tracked | you / Claude | `deploy-stack.sh` | | `stacks-mirror///` | snapshot / reality | gitignored | `sync-stacks.sh` | drift inspection | Side fix: moved `stacks/llama-swap/config.yaml` → `stacks/llama-swap/conf/config.yaml` so the deploy mapping matches the server layout (`stacks//conf/` → `/opt/docker/conf//`). ## Session milestones — 2026-04-24 (the "tooling day" + housekeeping pm) Morning / early afternoon — the original tooling day: - **irv-ml1 AI stacks deployed** as Docker: ComfyUI (port 8188, host- writable workflows), Parakeet ASR (port 8765, rewritten on sherpa-onnx after the Shadowfita FastAPI wrapper hit two unfixed upstream bugs), CosyVoice 3 (port 8190, GLaDOS voice cloned + live). - **irv-ml1 restic profile extended** to cover `/worktank/{comfyui/basedir/{user,custom_nodes,input},cosyvoice/voices}`; bulk weights + scratch dirs stay excluded. - **`scripts/elway`** shipped — ~800-line Python playbook runner with tier 1 + tier 2 idempotency; handlers / register / multi-host / content-hash are deferred (gitea #3–#6). - **task-board** built end-to-end (separate repo, `vh/task-board` on gitea) and shipped as a Claude Code plugin. Green/red cards per session via UserPromptSubmit + Stop hooks; four MCP tools expose explicit activity tracking. - **pfi-postgres NFS decoupling** finished item 4b — zero residual dependency on ana-nas for that VM. Late afternoon / evening — task-board iteration + STATUS sweep: - **task-board v0.1.1 → v0.1.3** shipped over four iterations on the live ANA deployment: - **v0.1.1** — hooks parse Claude Code's stdin JSON for `session_id` and append a short suffix when `TASK_BOARD_SESSION` isn't set, so two sessions in one project no longer collide on a single card. Dormant transition preserves cumulative idle time (`state_entered_at = last_update_at` instead of `now`). - **v0.1.2** — favicon (3-column SVG in active/waiting/dormant state colors); served at `/static/favicon.svg` with a `` and a `/favicon.ico` route returning the same SVG. Followup fix for an XML-illegal `--` in a comment. - **v0.1.2-followup** — UI live-duration ticker bumped from 5 s → 1 s (humanDuration floors to integer seconds; cheap render). - **v0.1.3** — case-insensitive session names. `sessions.name COLLATE NOCASE`; real ALTER migration (not a wipe) — keeps earliest- `created_at` row as canonical, reassigns child comments. Write path canonicalizes session label before inserting comments. Read path uses `COLLATE NOCASE` for safety on external API callers. - **Parakeet (irv-ml1) healthcheck fix** — image ships `wget` not `curl`; healthcheck swap, 2,190 failing checks → healthy. - **AIPA-MCP project session label fix** — set `TASK_BOARD_SESSION= Architect` in `.claude/settings.json`, updated the project's `CLAUDE.md` to specify `session="Architect"` for explicit MCP calls, and renamed the existing `AIPA-MCP` card → `Architect` in the live SQLite (1 session row + 25 comment rows preserved). - **STATUS items 11 / 13 / 15 / 16 closed.** Mattermost dir gone on ana-docker (verified); UniFi UDM cards added to homepage (PFI-UDMSE for NH3 edge replacing the retired Fortigate 101F, ESH-UDMPM for ESH); docs nav map added + stale chromadb-setup.md removed + VM-102 Matrix docs frontmatter stripped; STATUS.md discipline rule recorded. ## Session milestones — 2026-04-20 / 2026-04-21 - Homepage reorganized to function-first with tabs (Main / Infrastructure / Toolchain). - Per-group icons + equal-height layout + 4-column grids. - Fleet-wide label sweep (function groups, Service Networking renamed from Wiring/Plumbing due to homepage-parser slash bug). - 6 restic clients deployed end-to-end (client creds, repo init, profile install, systemd timers, first backups verified). - 9 hosts registered under `servers/` from gap-analysis (PFI VMs + SureFire tenant, with tenancy awareness). - Discovery scripts (FortiGate SSH + UniFi cloud API + TSV diff vs inventory). - Proxmox inspect script + fleet-wide refresh wrapper. - Calibre-Web-Automated migration replacing calibre + calibre-web pair. - llama-swap Qwen3.6 stock + mradermacher abliterated, 128K ctx. - Mattermost retired (not running, compose dir cleanup pending). - FortiGate 101F at NH3 retired; homepage card removed. ## Memory pointers (for future Claude sessions) Relevant `~/.claude/.../memory/` entries: - `server_split.md` — host placement rules - `feedback_ssh_sudo.md` — use `ssh -t` for remote sudo (mostly superseded by elway, but still applies to ad-hoc ssh) - `feedback_git_autonomous.md` — handle git commits without asking - `feedback_git_commits.md` — no Claude attribution in commit messages - `feedback_use_elway.md` — write elway playbooks; don't chain ssh+sudo - `feedback_backups_must_not_risk_production.md` — rule adopted after the 2026-04-23 ana-nas self-backup crash - `project_backup_pipeline_gaps.md` — user's explicit goal of "all hosts + configs + DBs backed up" - `project_db_migrate_off_nfs.md` — DB-off-NFS decision + status - `project_surefire_tenant.md` — SureFire tenancy boundary awareness - `reference_gitea_remote.md` — origin is `vh/esh-pfi-infrastructure` on gitea.phasefinal.com; `tea` CLI logged in as `vh` - `reference_task_board.md` — task-board plugin + tools contract - `storage_ana_nas.md` — ana NAS is Debian LXC (CT 109), not TrueNAS - `storage_nh3_nas.md` — NH3 NAS via `syncuser`, not `admin` - `incident_ana_nas_spof.md` — blast-radius matrix for ana-nas outages