esh-pfi-infrastructure/STATUS.md

# Status + Open Issues

Last updated: 2026-04-29

Snapshot of fleet state and open work. Refresh this file when a pass of
significant work lands — don't let it drift quietly.

**New session? Read [`docs/orientation.md`](docs/orientation.md) first.**

## What's in place

### Backup coverage (2-layer, fully operational)

- **PBS fleet-wide.** All 5 hypervisors (pfi-pve, nh3-pve, esh-pve,
  esh-pve-nas, sfsrv-ana) → `pbs-ana` (primary, VM on pfi-pve with
  NFSv3 datastore on ana-nas). Per-hypervisor namespaces to avoid
  VMID collisions. DR mirror at `pbs-nh3` (VM on nh3-pve with NFSv3
  datastore on nh3-nas Synology) pulls daily at 06:00. Verify jobs
  armed on both sides.
- **restic 8/8 hosts** with resticprofile + systemd timers at 01:00:
  | Host | Target rest-server | DB hooks |
  |---|---|---|
  | ana-docker | rest-server-ana (local) | synapse, seafile, vaultwarden-pg, gitea (native dump), openwebui |
  | ana-ml2 | rest-server-ana (cross-site) | — (no DBs) |
  | nh3-docker | rest-server-nh3 (local, Synology) | — |
  | esh-docker-vm | rest-server-ana (cross-site) | paperless-pg, HA SQLite, CWA SQLite, pgadmin SQLite, uptime-kuma SQLite |
  | vm-esh-nas | rest-server-ana (cross-site) | — |
  | nh3-dev (workstation) | rest-server-nh3 (local) | — |
  | irv-ml1 | rest-server-nh3 (via WG) | — |
  | esh-vm-db | rest-server-ana (cross-site) | pg_dumpall, mongodump |
- **Cross-site restic rsync.** `ana-nas → nh3-nas` at 04:00 daily
  (mirrors rest-server-ana data). `nh3-nas → ana-nas` at 05:00 daily
  (mirrors rest-server-nh3 data, runs as root since DSM writes files
  as admin mode 400). Tracked at `configs/rsync/{ana-nas-to-nh3,nh3-nas-to-ana}/`.

### Inventory

- **~22 tracked hosts** under `servers/` across ANA / NH3 / ESH / IRV sites.
  SSH config aliases in `~/.ssh/config` for every host — `ssh <name>` just works.
- **Homepage** at <http://10.0.50.45:5100> — function-first layout
  (Main / Infrastructure / Toolchain tabs), per-group icons, four
  Infra sub-groups (ANA / NH3 / IRV / ESH). Docker auto-discovery
  active for 5 hosts (ana-docker, ana-ml2, nh3-docker, esh-docker-vm,
  irv-ml1).
- **FortiGate + UniFi discovery scripts** produce TSVs; gap analysis
  against `servers/*/` for unmanaged IPs.

### Architecture decisions (durable)

- **DB data on local disk, not NFS.** pfi-postgres migrated 2026-04-23;
  fstab + NFS mount removed 2026-04-24 (pfi-postgres now fully
  decoupled from ana-nas). Mongo on esh-vm-db confirmed already local.
  Removes the biggest ana-nas blast-radius risk. Memory:
  `project_db_migrate_off_nfs.md`.
- **Backups must not risk production.** Rule adopted after
  2026-04-23 ana-nas self-backup crash. Memory:
  `feedback_backups_must_not_risk_production.md`.
- **offen sidecars retired fleet-wide 2026-04-23.** restic covers
  equivalent scope; reclaimed 16 GB of redundant tarballs on
  `/mnt/backup/docker/esh-vm-docker/`.
- **Gitea remote for this repo (2026-04-23).** `origin` is
  `vh/esh-pfi-infrastructure` on `gitea.phasefinal.com`. Also:
  `vh/task-board` (new 2026-04-24) hosts a Claude Code plugin +
  marketplace serving the assistant task-state dashboard.
- **Prefer elway for multi-step SSH work (2026-04-24).** The new
  `scripts/elway` mini-playbook runner replaces chained `ssh -t sudo …`
  commands — handles sudo once up front, structured reporting,
  idempotency (creates/when/changed_when). Memory:
  `feedback_use_elway.md`; template playbook:
  `playbooks/elway-smoke.yaml`.

### Tooling

- `scripts/server_inspect.sh` + `refresh-server-info.sh` for Docker hosts.
- `scripts/proxmox_inspect.sh` + `refresh-proxmox-info.sh` for PVE nodes
  (reports VMs/LXCs/storage/backup-coverage).
- `scripts/discover-fortigate.sh`, `scripts/discover-unifi.sh`,
  `scripts/discover-gaps.sh` for network-level inventory discovery.
- `scripts/deploy-stack.sh` + `scripts/sync-stacks.sh` for compose push/pull.
- `scripts/add-host.sh` for new-host registration.
- **`scripts/elway`** (2026-04-24) — mini-ansible playbook runner over
  SSH. Playbooks live under `playbooks/`. Tier 1 (creates/when) + tier
  2 (changed_when) idempotency; handlers + register + multi-host
  fan-out deferred as gitea issues #3–#5.
- **`tea` CLI** at `/usr/local/bin/tea` — gitea CLI, already logged in
  as `vh`. Use for issue / PR work instead of inventing URLs.

## Open issues

### ~~🟥 Blocked — irv-ml1 stalled after CUDA driver upgrade (2026-04-25)~~ — RECOVERED 2026-04-27

**Resolution**: the host eventually came back on its own — driver
595.58.03 (CUDA 13.2) IS working, both GPUs detected (RTX 3090 + RTX
A6000), kernel modules loaded cleanly. The original "stall" symptoms
must have been a one-shot post-install issue that resolved on a
later boot. All 5 pre-existing GPU stacks (comfyui, cosyvoice,
qwen3-tts, index-tts, parakeet) restarted clean and report healthy.
Homepage's irv-ml1-docker provider re-enabled via `configs/homepage/
docker.yaml`. Remaining post-recovery work was the deferred deploys
(Kokoro GPU, VibeVoice, Chatterbox) — see "Session milestones —
2026-04-27" below.

The original stall context is preserved below for the next time we
do a CUDA driver bump on a bare-metal host:

**What was attempted**: `scripts/upgrade-irv-ml1-cuda.sh phase1`
on 2026-04-25. Phase 1 is supposed to: snapshot current nvidia
packages → stop GPU containers → `apt install cuda-drivers` →
DKMS rebuild → reboot. Whether all those steps actually completed
before the unclean boot is unknown — the snapshot file at
`/tmp/nvidia-pre-upgrade.txt` on irv-ml1 (if it survived) will say.

**Recovery options** (in increasing invasiveness):

1. **Console diagnose first.** SSH may or may not work depending on
   the failure mode. Check via console (irv-ml1 is bare-metal, has
   physical access via Lenovo IPMI). Look at `journalctl -b -1` from
   the previous boot to see what failed: DKMS build failure, kernel
   module load error, nvidia-container-toolkit init regression, etc.

2. **Rollback the driver** if the host is reachable but GPU is broken:
   ```
   scripts/upgrade-irv-ml1-cuda.sh rollback
   ```
   This reinstalls the snapshot from `/tmp/nvidia-pre-upgrade.txt`
   with `--allow-downgrades` and reboots. Restores 570.124.06.

3. **Manual driver pin** if the script can't recover:
   ```
   ssh irv-ml1 sudo apt install --allow-downgrades \
     nvidia-driver=570.124.06-1 nvidia-driver-cuda=570.124.06-1 \
     nvidia-driver-libs=570.124.06-1 nvidia-kernel-open-dkms=570.124.06-1
   ssh irv-ml1 sudo apt-mark hold nvidia-driver nvidia-driver-cuda \
     nvidia-driver-libs nvidia-kernel-open-dkms
   ssh irv-ml1 sudo systemctl reboot
   ```

4. **Try a different driver version** if 595 is genuinely incompatible
   with kernel 6.1.0-37: 580 series may build cleanly. Check via
   `apt-cache madison cuda-drivers` after recovery.

**Once irv-ml1 is healthy again**, deferred work to revisit:

- **Kokoro GPU variant**: deployed CPU works as a fallback. Switch
  `.env` to `KOKORO_VARIANT=gpu` / `KOKORO_USE_GPU=true` /
  `KOKORO_GPU_DEVICES=0` and redeploy via
  `scripts/elway irv-ml1 --playbook playbooks/deploy-kokoro.yaml`.
- **VibeVoice 1.5B deploy** (long-form / multi-speaker dialogue):
  `scripts/elway irv-ml1 --playbook playbooks/deploy-vibevoice.yaml`.
  Stack files at `stacks/vibevoice/`. ~12 min build + ~7 GB model.
- **Chatterbox Turbo deploy** (low-latency English voice cloning):
  `scripts/elway irv-ml1 --playbook playbooks/deploy-chatterbox.yaml`.
  Stack files at `stacks/chatterbox/`. ~10 min build + ~6 GB model.
- **Verify the existing 5 GPU stacks still work** post-upgrade:
  comfyui, cosyvoice, qwen3-tts, index-tts, parakeet. They were all
  Up + healthy before the driver attempt. Smoke-test each.
- **Re-evaluate Kokoro variant default** after recovery. If a 595+
  series driver doesn't end up working, the `.env.example` default
  of `KOKORO_VARIANT=cpu` is correct as-is. If a working GPU driver
  lands, flip the default to `gpu` for future deploys.

**Don't run on irv-ml1 until this is closed**:
- Any of the three deploy playbooks above
- Any host-state-mutating playbook against irv-ml1
- The driver upgrade script itself unless intentionally retrying

Once recovered, update this section with what fixed it and remove the
🟥 marker.

### 🟥 Quick wins (do next)

1. ~~**Fix Backrest's `esh-docker-vm` URI**~~ — done.

2. ~~**Push SSH keys + pull snapshots for 6 unrefreshed hosts**~~ —
   **done 2026-04-23**. 4 VMs (pfi-ana-webhost, pfi-pteradactyl,
   pfi-tacticalrmm, pfi-postgres) took the workstation key via
   `ssh-copy-id` with `lkraven@`. 2 LXCs (ana-filebot, ana-wg)
   needed key installed via `pct push` from pfi-pve because
   `PermitRootLogin prohibit-password` blocked `ssh-copy-id`.
   Minor: pfi-pteradactyl's `server_inspect` Docker section is blank
   because lkraven is not in the `docker` group there — `sudo usermod
   -aG docker lkraven` + re-login to fix.

3. ~~**Patch out `forget` schedules from all 6 restic profiles**~~ — done.

3b. ~~**Discover sf-r630 OS-side IP**~~ — **resolved 2026-04-23**:
    the R630 with iDRAC `10.250.250.110` is the same physical box
    that runs `sfsrv-ana` (Proxmox VE at `10.250.250.115`). No
    separate OS IP to find. `servers/sf-r630/` now clarifies it as
    the hardware/BMC-only inventory entry; `servers/sfsrv-ana/` is
    the OS view. `servers/ana-ml2/README.md` updated with its own
    distinct BMC IP (`10.250.250.50`, Supermicro) to prevent future
    confusion between the two physical chassis.

<!-- 2026-04-21: Dropped the "nightly-restart safety net" idea — user
     confirmed Backrest UI chokes even right after startup, so periodic
     restart wouldn't reliably help. Root cause still unknown; see
     item under "Research / deferred" below. -->

4. _(removed — see note above)_

### 🟧 Real work (dedicated session each)

4b. ~~**Migrate DB data directories off NFS onto local VM disk.**~~ —
    **done 2026-04-23/24.** Postgres on pfi-postgres migrated
    04-23; mongodb on esh-vm-db confirmed already local (and serves
    zero user data in practice — paperless uses Postgres 15 on the
    same VM). fstab entry + NFS mount for `/mnt/db` on pfi-postgres
    removed 2026-04-24 via
    `playbooks/decouple-pfi-postgres-from-ana-nas.yaml`. pfi-postgres
    has zero remaining dependency on ana-nas. Only residue: cold
    archive of pre-migration data still on ana-nas (`/mnt/db/pfi-*`);
    harmless, can sit indefinitely.

5. ~~**Rotate exposed secrets**~~ — **done 2026-04-23**. All six
   rotated: vaultwarden/gitea/paperless-ng Postgres passwords
   (hardcoded `compose.yaml` literals moved to gitignored `.env`
   files in the process), ana-docker + ana-ml2 + esh-docker-vm
   rest-server htpasswd entries, and rest-server repo passphrases for
   ana-docker + esh-docker-vm (ana-ml2's was already rotated during
   prior wipe+reinit). Also discovered along the way: paperless uses
   a separate `esh-vm-db` VM (10.0.50.60), not pfi-postgres as the
   old notes implied.

6. ~~**Cross-site rsync**~~ — **done 2026-04-23, both directions**.
   - **ana → nh3** (04:00 daily): `ana-nas:/mnt/backup/restic/repo/ana/`
     → `nh3-nas:/volume1/Backup/restic-ana-mirror/`. Runs on ana-nas
     as lkraven. 15.6 GB initial sync completed 07:18 UTC.
   - **nh3 → ana** (05:00 daily): `nh3-nas:/volume1/Backup/restic/`
     → `ana-nas:/mnt/backup/restic-nh3-mirror/`. Runs on nh3-nas as
     root (rest-server-nh3 container writes mode-400 files; only
     root can read them on Synology). 2.31 GB initial sync completed
     07:42 UTC.
   - Tracked at `configs/rsync/ana-nas-to-nh3/` and
     `configs/rsync/nh3-nas-to-ana/` respectively.

6b. **PBS deployment across the fleet.** Runbook at
    `docs/runbooks/pbs-deployment.md`. Phases 0–6 **done** (2026-04-22):
    PBS-ANA on pfi-pve with NFS datastore on Ana NAS (NFSv3 for ZFS
    case-insensitivity), PBS-NH3 on nh3-pve with NFS datastore on NH3
    Synology (NFSv3 + Linux-mode POSIX 777 to avoid syno_acl
    interference), per-hypervisor namespaces, API tokens, verify jobs
    on both, one-way sync ANA → NH3 at 06:00 daily, and all 5
    hypervisors (pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana)
    onboarded to PBS-ANA.
    Remaining phases:
    - Phase 7-8 — one week burn-in, then retire legacy vzdump targets
      on each hypervisor (keep until 2026-04-29 earliest)
    - Phase 9 — homepage cards for PBS-ANA + PBS-NH3 web UIs,
      memory/doc updates, `servers/pbs-ana` + `servers/pbs-nh3` dirs

7. ~~**SureFire tenant backup plan decision**~~ — **resolved 2026-04-23**
   by folding sfsrv-ana into the fleet-wide PBS deployment (dedicated
   `sfsrv-ana` namespace on PBS-ANA, replicates to PBS-NH3 via the
   same sync job as the rest of the fleet). Hosting-agreement option
   chosen: PFI provides backup coverage as part of managed hosting.

### 🟨 Prereqs / polish

8. ~~**Synology SSH setup**~~ — **done 2026-04-22**. Dedicated
   `syncuser` account (admin-group membership) with key auth,
   registered as `servers/nh3-nas/`, reachable as `ssh nh3-nas`.
   Unlocks cross-site rsync (item 6), rest-server-nh3 healthcheck
   deploy, and `.htpasswd` edits on the NH3 side.

9. ~~**`scripts/restic-prune.sh`**~~ — **done 2026-04-24.** Quarterly
   disk-hygiene tool. Drops `--append-only` on the rest-server (via a
   temporary `docker-compose.override.yaml` — never edits the canonical
   compose), runs `resticprofile forget --prune --verbose` on each
   client, restores `--append-only` (with `trap` so it runs even on
   partial failure). ANA side fully automated (5 clients); NH3 side
   prints a manual ceremony because DSM Container Manager + sudo on
   syncuser aren't cleanly scriptable from this workstation. Run with
   `scripts/restic-prune.sh ana|nh3|all`, optionally `--dry-run`.

   **First NH3 ritual run: 2026-04-27** — full process documented in
   [`docs/runbooks/nh3-prune-ritual.md`](docs/runbooks/nh3-prune-ritual.md)
   (Phase 1 DSM toggle off → Phase 2 forget+prune on 3 clients →
   Phase 3 DSM toggle back on). Today's run was a no-op everywhere
   (every snapshot still in keep window). Next round: ~2026-07-27.

10. ~~**Retire `offen/docker-volume-backup` sidecars**~~ — **done
    2026-04-23**. Removed from paperless-ngx and pgadmin composes on
    esh-docker-vm (only hosts in the fleet that had them). 16 GB of
    orphan tarballs at `/mnt/backup/docker/esh-vm-docker/` reclaimed.
    Restic coverage verified equivalent: `/var/lib/docker/volumes` in
    source + pre-backup hook handles the DB dumps for paperless
    (against esh-vm-db), HA, pgadmin, CWA, uptime-kuma. Obsolete
    `*_offen_backup_data` exclude also removed from the restic profile.

11. ~~**Clean up retired mattermost dir** on ana-docker~~ — **done**
    (verified 2026-04-24: `/opt/docker/compose/mattermost/` does not
    exist; no mattermost containers anywhere on the host).

### 🟩 Research / deferred / intermittent

12. **Backrest UI intermittent timeout.** Backend is confirmed healthy
    (direct `curl GetConfig` returns in 200ms). UI chokes on the same
    endpoint. Likely cause: SSE stream wedge in browser, or service
    worker stale. Workaround: Ctrl+Shift+R. Root cause investigation
    deferred.

13. ~~**UniFi controller homepage cards**~~ — **done 2026-04-24.** Both
    cards added to `configs/homepage/services.yaml` and pushed to
    esh-docker-vm: `PFI-UDMSE` (10.100.0.1, UDM Pro SE) under `Infra -
    NH3` as the new edge device replacing the retired Fortigate 101F;
    `ESH-UDMPM` (10.0.0.1, UDM Pro Max) under `Infra - ESH`. Icon
    `si-ubiquiti`.

14. **Prune + credential-rotation scripts** as repeatable tooling (vs
    per-incident manual work).

### 🟦 Memory / documentation housekeeping

15. ~~**`docs/` organization**~~ — **done 2026-04-24.** First-pass
    landed earlier (misfiled `tea-*.sha256` removed; `docs/README.md`
    nav map added). Second pass landed same day: stripped the broken
    YAML frontmatter from both VM-102 Matrix docs (the `path:` values
    pointed at `docs/pfi-ana/...` which doesn't exist in this repo,
    and no toolchain consumed the metadata); deleted
    `pfi/chromadb-setup.md` (referenced `configs/pfi-ana/...` and
    `scripts/setup-chromadb.sh`, both nonexistent — deployment is
    long done and the operational truth lives in `docker-stack.md`).
    Kept the two VM-102 docs separate by design (each is right-sized;
    a merge would push past the 500-line guideline in the README).

16. ~~**`STATUS.md` drift discipline**~~ — **done 2026-04-24.**
    Refreshed with the post-tooling-day session work; `status-regen.sh`
    idea dropped — STATUS.md is intentionally narrative, not derivable
    from git/code, so auto-regen would lose information. Discipline
    rule: refresh STATUS.md at the end of any session where a 🟧 or 🟥
    item closes, or three+ smaller items land.

## Active migration — docker.io 20.10 → docker-ce 29.x

Bookworm's Debian-packaged `docker.io` is pinned at 20.10.24 with
client/daemon API 1.41. Newer compose clients (1.52+) refuse to talk
to it ("client version 1.52 is too new" during `docker compose
build`). The migration moves each host to Docker's official `docker-ce`
APT repo (29.x). Playbook: `playbooks/upgrade-docker-ce.yaml`.

Order (least → most blast radius):

1. ~~**nh3-docker**~~ — **done 2026-04-24.** docker-ce 29.4.1, two
   stacks (portainer agent + portainer) restarted clean. Debug surfaced
   two unit-file gotchas now baked into the playbook:
   - `/etc/systemd/system/docker.service.d/override.conf` from the
     docker.io era hardcoded `/usr/sbin/dockerd` (docker-ce installs
     at `/usr/bin/dockerd`) → daemon failed status=203/EXEC.
   - The shipped docker-ce unit's `ExecStart=dockerd -H fd:// …`
     conflicts with `daemon.json` `hosts:` (defined for the 0.0.0.0:2375
     homepage-discovery binding) → "conflicting host options".
   Playbook now backs up any existing override, probes `daemon.json`
   for `hosts:`, and installs an override that strips `-H` when needed.
2. ~~**esh-docker-vm**~~ — **done 2026-04-24.** docker-ce 29.4.1, all
   16 stacks back up clean (adguard, beszel-agent, calibre-web,
   cloudflare-ddns, dockge, dozzle-agent, drawio, esphome, homeassistant,
   homepage, mosquitto, paperless-ngx, pgadmin, traefik, uptimekuma).
   Surfaced one verify-only nit: the docker-compose-plugin shipped with
   docker-ce 29 is renumbered v5.x (was v2.x with 26-28); same Compose
   v2 codebase, just realigned major. Playbook's compose-version verify
   was hardcoded to `v2.X.Y` and false-failed — now accepts any major ≥ 2.
3. ~~**ana-docker**~~ — **done 2026-04-24.** docker-ce 29.4.1, all 29
   containers back up clean. Traefik routing live (200s for
   matrix.phasefinal.com presence + seafile.phasefinal.com syncs
   verified post-restart). Sub-finding (since fixed, see below):
   seafile's internal `seahub` raced mysql on startup and wedged.

Also during the nh3-docker debug: `configs/homepage/docker.yaml`
commented out the `irv-ml1-docker` provider — the 20s-per-poll
ETIMEDOUTs from the stalled host were drowning homepage's logs and
appeared to block ana-pfi-docker discovery (News group's Miniflux card
wouldn't render until removal). Re-enable when irv-ml1 is back.

## traefik-postboot — host-level recovery for daemon-restart routing races

Installed 2026-04-24 on both hosts that run traefik (esh-docker-vm,
ana-docker). One-shot systemd unit (`traefik-postboot.service`,
After=docker.service, Type=oneshot, sleep 60 → `docker restart
traefik`) that fires once per boot. Forces a clean cold scan after
the topology has settled, fixing the long-standing "traefik misses
backends after host reboot" symptom (events-stream debounce + network-
attach race when 30+ containers start in a burst).

Playbook: `playbooks/install-traefik-postboot.yaml`. Tunables: 
`--var delay_seconds=N` (default 60), `--var container=NAME`
(default `traefik`). Runs `systemd-analyze verify` non-disruptively
so installing on a healthy host doesn't blip traefik.

## autorestic decommission — redundant with the new two-layer pipeline

Hosts with autorestic detected 2026-04-24: `esh-docker-vm`, `esh-vm-db`.
Both are pre-pipeline ESH manual installs at `/usr/local/bin/autorestic`
+ four systemd units backing up to `/mnt/backup/restic/repo/esh`.
The new two-layer pipeline (PBS for VM images, structured restic
profiles via `configs/restic/<host>/profiles.yaml`) covers everything
autorestic was doing. Trigger: a typo on esh-docker-vm's
`autorestic-backup.timer` line 2 (`D:escription` instead of
`Description`) surfaced during the traefik-postboot install verify.

Playbook: `playbooks/remove-autorestic.yaml`. **Done 2026-04-24** on
both hosts after two playbook bug-fixes surfaced by the first run:
(a) `rm path1 \` + newline + `path2` was bitten by YAML plain-scalar
folding — the backslash-newline collapsed to a literal `\ `, so only
the first path got passed to rm. Fixed by switching to a glob.
(b) `shell: ! command -v autorestic >/dev/null` was parsed by YAML as
a tag indicator with the `!` stripped, inverting the verify's exit
code. Fixed by quoting the value.

Deliberately untouched (decide separately):
- `/srv/backups/autorestic/.autorestic.yml` — config; archival value
- `/mnt/backup/restic/repo/esh` — historical autorestic snapshots;
  prune via `restic forget` once comfortable with new pipeline coverage.

## seafile seahub startup race — fixed 2026-04-24

When ana-docker rebooted (or any time the seafile container was
recreated alongside seafile-mysql), seahub (the python frontend at
port 8000 inside the seafile container) raced mariadb's startup.
Short-form `depends_on: [db, memcached]` in compose only waits for
the dependency container to be CREATED, not healthy — so seafile's
`start.py` would hit `mysql Connection refused` and give up,
leaving nginx serving seafile but proxying to a dead python backend.
Symptom in traefik logs: ~7 of 30 requests returning 502 on
seafile@docker dynamic endpoints; static assets fine.

Fix applied directly to ana-docker's `/opt/docker/compose/seafile/
compose.yaml` (and into `stacks-mirror/`, which is gitignored — so
not in the repo, but the file on the server is the source of truth):
- Added `healthcheck` to the `db` service using the bundled
  `/usr/local/bin/healthcheck.sh --connect --innodb_initialized`
  (5s interval, 30s start_period, 30 retries).
- Converted seafile's `depends_on` to long-form with
  `condition: service_healthy` for db (and `service_started` for
  memcached).

Verified post-recreate: seahub log shows `Seahub is started ... Done.`
without traceback; traefik `seafile@docker` dropped to 0 502s in
the next 20s window.

## news-digest — per-item × button + cross-device hidden tray (2026-04-24)

Daily Digest pages now have a small × on each item that hides it
from the page. State persists server-side at `/output/hidden.json`,
so the same hidden set follows the user across devices (home, ipad,
laptop, work). A "Hidden (N)" tray at the bottom of each page lists
items hidden FROM THE CURRENT PAGE with a restore button per row;
older hidden ids that aren't on this page sit silently in
`hidden.json` and continue to filter future editions that include
the same article.

Architecture change: `news-digest-web` swapped from `nginx:alpine`
to a FastAPI app on uvicorn, built from the **same Dockerfile** as
the worker. One image, two containers — the worker's cron-driven
ENTRYPOINT runs in `news-digest-worker`, while compose overrides
entrypoint+command to launch `uvicorn web:app` in `news-digest-web`.
Drops one image dep, adds three endpoints:
- `GET  /api/hidden`   — JSON array of hidden item ids
- `POST /api/hide`     — `{id}` → adds to hidden.json
- `POST /api/restore`  — `{id}` → removes from hidden.json

Item ids are stable 12-char sha1 prefixes (`reddit:<post_id>` /
`miniflux:<entry_id>`) computed in `digest.py` at render time so
hide-once means hide-forever-for-that-article. Storage: single
JSON array, atomic writes via tempfile+rename, threading.Lock
inside the single uvicorn worker. No auth — same trust boundary
as the digest itself (LAN-only on port 8181).

The deploy playbook also dropped its `DOCKER_BUILDKIT=0` fallback
(no longer needed post docker-ce 29 migration) and gained three
verify steps: `/api/hidden` returns a JSON array, `app.js` is
reachable, and a synthetic `smoke-<ts>` id round-trips through
hide → list → restore → list.

## llama-swap — added two vision-capable Qwen 3.6 entries (2026-04-24)

- `qwen3.6-35-a3b-heretic` — `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF:Q6_K`
  (~28 GB) + auto-loaded BF16 mmproj for vision.
- `qwen3.6-27b` — `unsloth/Qwen3.6-27B-GGUF:UD-Q6_K_XL` (~26 GB) +
  auto-loaded BF16 mmproj for vision (dense 27B, not MoE).

Both pre-pulled into `/tank/aimodels/huggingface` (HF_HOME=/hfcache
in the container) via a one-shot `python:3.12-slim` container running
`huggingface_hub.snapshot_download` with `hf_transfer` for parallel
chunked downloads — heretic's 29 GB landed in 4:10, unsloth's 26.5 GB
in 3:46 (~118 MB/s each). Both repos include `mmproj-BF16.gguf`
alongside the main GGUF and llama-server's `-hf` flag auto-loads it
when present, so vision works without an explicit `--mmproj` path.

## Stack tree convention (canonical vs mirror) — clarified 2026-04-24

The workspace has two stack trees and they look similar enough that
deploy-stack.sh was reading from the wrong one (`stacks-mirror/`
instead of `stacks/`). Edits to `stacks/llama-swap/config.yaml` for
the two new model entries above never reached ana-ml2 on the first
deploy attempt because the diff was canonical-vs-mirror, not
canonical-vs-server. Roles now nailed down explicitly in CLAUDE.md
and `memory/convention_stacks_vs_mirror.md`:

| tree | role | git | who writes | who reads |
|---|---|---|---|---|
| `stacks/<stack>/` | canonical / intent | tracked | you / Claude | `deploy-stack.sh` |
| `stacks-mirror/<host>/<stack>/` | snapshot / reality | gitignored | `sync-stacks.sh` | drift inspection |

Side fix: moved `stacks/llama-swap/config.yaml` →
`stacks/llama-swap/conf/config.yaml` so the deploy mapping matches
the server layout (`stacks/<stack>/conf/<file>` →
`/opt/docker/conf/<stack>/<file>`).

## Session milestones — 2026-04-29 (Nevermore extraction)

The news-digest applet was extracted into its own repo at
[`vh/nevermore`](https://gitea.phasefinal.com/vh/nevermore) — clean
break, fresh git history, renamed throughout (DIGEST_* → NEVERMORE_*,
news-digest-* containers → nevermore-*, masthead → "Nevermore"). The
existing `news-digest-*` deployment on ana-docker continues to run
unchanged on its baked image; future updates come from the new repo.
Migration path documented in nevermore's README.

Removed from this workspace:
  stacks/news-digest/, playbooks/deploy-news-digest.yaml,
  scripts/add-digest-user.sh.

## Session milestones — 2026-04-28..29 (audio gen, digest applet, task-board polish, tooling)

### Audio generation on irv-ml1 (A6000)

- **ACE-Step 1.5** (music gen, Apache 2.0) — port 8210. Hybrid
  diffusion + LLM, ~10-12 GB VRAM at inference. Custom `Dockerfile`
  patches upstream's torch/cu126 resolution bug; patched
  `infer-api.py` ships in the image (upstream's positional tuple is
  missing 6 of 24 pipeline args). Verified bench: 60 s clip in
  14.8 s (~4× realtime).
- **Stable Audio Open 1.0** (SFX/ambience, Stability Community license)
  — port 8211. Custom FastAPI shim wraps `diffusers.StableAudioPipeline`.
  ~4-6 GB VRAM in fp16, up to 47 s clips at 44.1 kHz.

### Daily Digest applet

- **World + Local headline sections** above the existing tech/reddit
  cards. `seed-headlines.py` provisions the categories and curated
  feeds (BBC + NPR + Al Jazeera; LA Times + Voice of OC) in miniflux.
- **Article-extracted summaries** via `trafilatura` — 2-3 sentence
  summaries grounded in real article content (was: title-paraphrase).
  Cached at `/output/.article-cache.json` with 7-day TTL. Adds ~30 s
  per digest run, parallel-prefetched.
- **Per-desk collapse** chevron, localStorage-persisted per device.
- **Mobile-responsive** — true 480 px breakpoint, fixed masthead
  overlap, cache-busted asset URLs.
- **Multi-tenant** via `scripts/add-digest-user.sh <name>` — shared
  miniflux + per-user digest stack (own port, hide-state, cron).
  `DIGEST_CRON_AM` / `DIGEST_CRON_PM` are now env-driven; each
  teammate's stack fires on their own hours.

### task-board

- **v0.1.11 pending_shells / orange treatment.** New `task_set_shells`
  MCP tool + `pending_shells` field on the Task model (schema v6,
  ALTER ADD COLUMN). When a card is `state=waiting` AND has
  non-empty shells, it flips from blue (waiting on user) to orange
  (waiting on background work). Cards with non-empty shells are
  excluded from auto-dormant promotion.
- **v0.1.12 card-bar rhythms** — three distinct pulse cadences
  (active 2.4 s, waiting 3.6 s patient breath, attention 1.6 s
  louder) so column scan reads at a glance.

### Tooling

- **`scripts/fork-fleet.sh`** — bootstrap a new fleet-management
  workspace from this one. Mirrors reusable tooling, strips
  fleet-specific content, regenerates skeleton CLAUDE.md / README /
  STATUS, fresh git history.
- **CLAUDE.md `task_set_shells` discipline** — assistant must call
  `task_set_shells` on any `Bash run_in_background` (and clear when
  done) so the orange treatment fires; hooks have no way to
  enumerate Claude Code's bg tasks externally.

### New stack on ana-docker

- **memos** (port 5230) — stock `neosmemo/memos:stable`, SQLite at
  `/opt/docker/conf/memos/data/`, homepage label `Notes` (auto-discovered).

## Session milestones — 2026-04-27 (irv-ml1 recovery + new GPU deploys)

### irv-ml1 fully recovered
- Host alive, driver 595.58.03 / CUDA 13.2, both GPUs detected,
  kernel modules loaded.
- 5 pre-existing GPU stacks (comfyui, cosyvoice, qwen3-tts,
  index-tts, parakeet) restarted clean and report healthy.
- Homepage `irv-ml1-docker` provider re-enabled in
  `configs/homepage/docker.yaml`, pushed via the rsync workflow.

### Three new GPU stacks deployed on irv-ml1

- **Kokoro GPU variant** — first try succeeded once the `.env.example`
  default was flipped from `cpu` → `gpu` (matching the now-working
  driver). `KOKORO_VARIANT=gpu` / `KOKORO_USE_GPU=true` /
  `KOKORO_GPU_DEVICES=0` (pinned to RTX 3090; ~1 GB VRAM, doesn't
  need the A6000). Serving on irv-ml1:8193.

- **VibeVoice 1.5B** — needed two iterations:
  - `VIBEVOICE_SHA=7614c469a145` (12-char short) made docker buildx's
    git source resolver error with "repository does not contain ref
    7614c469a145" — same commit IS still HEAD of main, but buildx
    requires the FULL 40-char SHA even when the short is unambiguous.
    Now `7614c469a145dafc30b1fd5f6a6046ea831dbd6a`.
  - The verify step greped `/v1/audio/voices` for `voices`/`voice`/
    `alloy`/`Carter` literals, but VibeVoice's actual response shape
    is OpenAI list-format `{"object":"list","data":[]}`. Switched to
    JSON-parse + assert `object="list"` and `data` is a list. Robust
    against an empty voices set on a fresh install.
  - Serving on irv-ml1:8194.

- **Chatterbox Turbo** (devnen wrapper) — needed three iterations:
  - Old playbook used `dockerfile: docker/Dockerfile.gpu`; devnen
    restructured the repo to put Dockerfiles at root, renamed by CUDA
    version (`Dockerfile.cu128`, `.cpu`, `.rocm`). Switched to
    `Dockerfile.cu128` (GPU build for CUDA 12.8 toolkit; works with
    irv-ml1's 595.58.03 driver via backward-compat).
  - Pinned `CHATTERBOX_SHA` from `main` to the current SHA
    `68213e75c248f4f4bbd185f7c20ec02722fdc79b` so future upstream
    churn doesn't break us without warning.
  - Wait + verify steps probed `/health`, but devnen's wrapper
    doesn't expose that route (404). Switched to `/api/model-info`
    which returns `{"loaded":true,...}` only after the model finishes
    loading — doubles as liveness + readiness. Compose healthcheck
    + verify both updated. Verify of `/v1/audio/voices` also fixed
    to parse the wrapper's actual response shape
    `{"status":"ok","voices":[...]}` (note: NOT the same OpenAI list
    shape vibevoice uses).
  - Serving on irv-ml1:8196.

### llama-swap pin/unload cleanup

- Removed `ttl: 600` from `qwen3.6-35-a3b-abliterated`,
  `qwen3.6-35-a3b-heretic`, `qwen3.6-27b` so they stay loaded across
  idle (still evictable when another non-coexistent model is
  requested).
- `qwen3.6-35-a3b` removed from the `pinned` group — its
  `persistent: true` flag was preventing eviction under memory
  pressure, OOM-ing other model loads. Now sits with `ttl: 0`
  outside any group: never idle-unloads but is evictable.
- `granite-4-small` added to the `pinned` group alongside
  `qwen3.5-9b` so it stops swapping with `qwen3.6-27b` when both are
  used concurrently. ~5-6 GB at Q4_K_M + 120K KV ≈ comfortable inside
  the existing pin budget.

### Backup verification post-recovery

All three layers green as of 2026-04-27:
- **Per-host restic** (`resticprofile-backup@profile-default.timer`):
  all 6 hosts (ana-docker, esh-docker-vm, nh3-docker, esh-vm-db,
  ana-ml2, irv-ml1) ran successfully at 01:00 today. The migration
  to docker-ce on three of these hosts didn't disrupt the timers.
  irv-ml1's timer fired right after recovery.
- **PBS-ANA**: every namespace (`ana-pve`, `esh-pve`, `nh3-pve`,
  `sfsrv-pve`) has fresh 2026-04-27 snapshots for every VM/CT.
- **PBS-NH3 (DR mirror)**: synced — 2026-04-27 backups present in
  /mnt/pbs-datastore/ns/. NH3 NAS at 27TB / 42TB used (64%, healthy
  headroom).

Note: backrest's UI shows `recentBackups: {}` for all repos because
backrest in this setup is a viewer with **zero plans configured** —
the actual orchestration is the per-host resticprofile timers.
Backrest's only configured operations are `_unassociated_` (manual
one-off pushes from the UI) and `_system_` housekeeping. Don't be
alarmed by the empty dashboard — check the per-host timers instead.

## Session milestones — 2026-04-24 (the "tooling day" + housekeeping pm)

Morning / early afternoon — the original tooling day:

- **irv-ml1 AI stacks deployed** as Docker: ComfyUI (port 8188, host-
  writable workflows), Parakeet ASR (port 8765, rewritten on
  sherpa-onnx after the Shadowfita FastAPI wrapper hit two unfixed
  upstream bugs), CosyVoice 3 (port 8190, GLaDOS voice cloned + live).
- **irv-ml1 restic profile extended** to cover
  `/worktank/{comfyui/basedir/{user,custom_nodes,input},cosyvoice/voices}`;
  bulk weights + scratch dirs stay excluded.
- **`scripts/elway`** shipped — ~800-line Python playbook runner with
  tier 1 + tier 2 idempotency; handlers / register / multi-host /
  content-hash are deferred (gitea #3–#6).
- **task-board** built end-to-end (separate repo, `vh/task-board` on
  gitea) and shipped as a Claude Code plugin. Green/red cards per
  session via UserPromptSubmit + Stop hooks; four MCP tools expose
  explicit activity tracking.
- **pfi-postgres NFS decoupling** finished item 4b — zero residual
  dependency on ana-nas for that VM.

Late afternoon / evening — task-board iteration + STATUS sweep:

- **task-board v0.1.1 → v0.1.3** shipped over four iterations on the
  live ANA deployment:
  - **v0.1.1** — hooks parse Claude Code's stdin JSON for `session_id`
    and append a short suffix when `TASK_BOARD_SESSION` isn't set, so
    two sessions in one project no longer collide on a single card.
    Dormant transition preserves cumulative idle time
    (`state_entered_at = last_update_at` instead of `now`).
  - **v0.1.2** — favicon (3-column SVG in active/waiting/dormant state
    colors); served at `/static/favicon.svg` with a `<link rel="icon">`
    and a `/favicon.ico` route returning the same SVG. Followup fix
    for an XML-illegal `--` in a comment.
  - **v0.1.2-followup** — UI live-duration ticker bumped from 5 s →
    1 s (humanDuration floors to integer seconds; cheap render).
  - **v0.1.3** — case-insensitive session names. `sessions.name COLLATE
    NOCASE`; real ALTER migration (not a wipe) — keeps earliest-
    `created_at` row as canonical, reassigns child comments. Write
    path canonicalizes session label before inserting comments. Read
    path uses `COLLATE NOCASE` for safety on external API callers.
- **Parakeet (irv-ml1) healthcheck fix** — image ships `wget` not
  `curl`; healthcheck swap, 2,190 failing checks → healthy.
- **AIPA-MCP project session label fix** — set `TASK_BOARD_SESSION=
  Architect` in `.claude/settings.json`, updated the project's
  `CLAUDE.md` to specify `session="Architect"` for explicit MCP
  calls, and renamed the existing `AIPA-MCP` card → `Architect` in
  the live SQLite (1 session row + 25 comment rows preserved).
- **STATUS items 11 / 13 / 15 / 16 closed.** Mattermost dir gone on
  ana-docker (verified); UniFi UDM cards added to homepage (PFI-UDMSE
  for NH3 edge replacing the retired Fortigate 101F, ESH-UDMPM for
  ESH); docs nav map added + stale chromadb-setup.md removed +
  VM-102 Matrix docs frontmatter stripped; STATUS.md discipline rule
  recorded.

## Session milestones — 2026-04-20 / 2026-04-21

- Homepage reorganized to function-first with tabs (Main / Infrastructure / Toolchain).
- Per-group icons + equal-height layout + 4-column grids.
- Fleet-wide label sweep (function groups, Service Networking renamed from
  Wiring/Plumbing due to homepage-parser slash bug).
- 6 restic clients deployed end-to-end (client creds, repo init, profile
  install, systemd timers, first backups verified).
- 9 hosts registered under `servers/` from gap-analysis (PFI VMs + SureFire
  tenant, with tenancy awareness).
- Discovery scripts (FortiGate SSH + UniFi cloud API + TSV diff vs inventory).
- Proxmox inspect script + fleet-wide refresh wrapper.
- Calibre-Web-Automated migration replacing calibre + calibre-web pair.
- llama-swap Qwen3.6 stock + mradermacher abliterated, 128K ctx.
- Mattermost retired (not running, compose dir cleanup pending).
- FortiGate 101F at NH3 retired; homepage card removed.

## Memory pointers (for future Claude sessions)

Relevant `~/.claude/.../memory/` entries:

- `server_split.md` — host placement rules
- `feedback_ssh_sudo.md` — use `ssh -t` for remote sudo (mostly
  superseded by elway, but still applies to ad-hoc ssh)
- `feedback_git_autonomous.md` — handle git commits without asking
- `feedback_git_commits.md` — no Claude attribution in commit messages
- `feedback_use_elway.md` — write elway playbooks; don't chain ssh+sudo
- `feedback_backups_must_not_risk_production.md` — rule adopted after
  the 2026-04-23 ana-nas self-backup crash
- `project_backup_pipeline_gaps.md` — user's explicit goal of "all
  hosts + configs + DBs backed up"
- `project_db_migrate_off_nfs.md` — DB-off-NFS decision + status
- `project_surefire_tenant.md` — SureFire tenancy boundary awareness
- `reference_gitea_remote.md` — origin is `vh/esh-pfi-infrastructure`
  on gitea.phasefinal.com; `tea` CLI logged in as `vh`
- `reference_task_board.md` — task-board plugin + tools contract
- `storage_ana_nas.md` — ana NAS is Debian LXC (CT 109), not TrueNAS
- `storage_nh3_nas.md` — NH3 NAS via `syncuser`, not `admin`
- `incident_ana_nas_spof.md` — blast-radius matrix for ana-nas outages