memory: capture 2026-06-20 session — Worldtree v0.37.7 demo fix, gitea notifier recovery, backup diagnosis

persistent-memory.md

@@ -126,19 +126,70 @@ _As of 2026-06-20:_
   demo+personal bind-mounts and triggered build-and-deploy via claude-bot.
   Pinned rides CI. (d372103, d0eb09c, 7819f96)
 
-- **🔴 BACKUP GAP — `rest-server-ana` DOWN ~months on a failed ana-nas NFS
+- **Worldtree demo → v0.37.7 GREEN (2026-06-20) — two-half fix.** worldtree-dev's
-  mount (NOT a decommission — docs corrected the recollection).** The
+  v0.37.5 config (3fbf338) carried a dangling `granite-4.1-8b` catalog ref →
-  active ana-side restic file+DB target (ana-docker/ana-ml2/esh-docker-vm/
+  demo crash-blocked at startup validation. Fix needed BOTH halves: (a) gitea
-  vm-esh-nas) is crash-looping: `/etc/fstab` has `10.250.50.50:/mnt/backup`
+  was silently not creating Actions runs so no deploy could fire (the earlier
-  as an NFS mount, but `mnt-backup.mount` is in a `failed` state → /mnt/backup
+  disk-full wedged its notifier — `docker restart gitea` cleared it; see
-  is the empty local ghost dir → `.htpasswd permission denied` (the
+  Tried-and-abandoned); (b) the v0.37.7 IMAGE alone wouldn't fix demo because
-  documented disaster-recovery.md failure). ana-nas is pingable; the mount
+  demo+personal read BIND-MOUNTED config — I re-staged `config/providers.yaml`
-  just isn't attached (bare-`defaults` fstab, no auto-retry). So file+DB
+  + `config/model_roles.yaml` from gitea at **439bebf** byte-identical into BOTH
-  restic for 4 hosts has been failing for ~months (PBS VM-image backups are
+  `/opt/worldtree{,-personal}/config` (sha256-verified across source + both
-  the only cover). **DEFERRED to clean context:** the documented recovery
+  instances; `granite-4.1-8b`→`summarizer`, `granite-classifier/summarizer`→
-  (clear failed mount → rm ghost → `mount /mnt/backup` → restart rest-server)
+  `classifier/summarizer`; old configs saved `.bak-pre-439bebf`; NO container
-  needs root → infra-ops; plus prevention (fstab `_netdev`/retry + a
+  recreate — avoided the stale-`:latest` footgun). worldtree-dev self-served the
-  backup-freshness alert — the gap went unnoticed for months).
+  CI re-run via the claude-bot Actions token → **run 1290 GREEN, demo on
+  v0.37.7**, capability routing live. claude-bot token migration DONE for
+  worldtree-dev (off vh's personal PAT). **QUEUED (NON-URGENT):** the ratatoskr
+  readonly-admin tier lives in `config/policies.yaml` (006d233, in 439bebf) —
+  re-stage that file to personal's bind-mount + reload personal Heimdall, then
+  mint a `ratatoskr-readonly` key (tier=readonly-admin, distinct from their
+  send-only c990f0be) + deliver.
+
+- **🟠 BACKUP DIAGNOSIS (2026-06-20, full probe) — REVISED from "all ana
+  backups down": PBS + nh3-restic are HEALTHY & CURRENT; only the ANA-side
+  restic file+DB layer is dead, and the one host with NO other cover is
+  ana-ml2.** Three layers:
+  - ✅ **PBS-ANA (VM images)** — healthy, snapshots TODAY. All PFI guests
+    backed up under PBS namespaces `ana-pve` / `esh-pve` / `nh3-pve` (+ tenant
+    `sfsrv-pve`); PBS-ANA's own NFS datastore mounts ana-nas fine (1.2T/20T).
+    Gaps to note: `ana-pve/vm/100` has no snapshots; **there is NO `esh-pve-nas`
+    PBS namespace** → vm-esh-nas + esh-vm-db may be PBS-UNCOVERED (VERIFY).
+  - ✅ **restic file+DB nh3-side** (`rest-server-nh3` @ nh3-nas:8000, UP→401) —
+    irv-ml1 + nh3-docker, last snapshot TODAY 01:00. Repos at
+    `nh3-nas:/volume1/Backup/restic/{irv-ml1,nh3-docker}`.
+  - 🔴 **restic file+DB ANA-side** (`rest-server-ana` @ ana-docker:8000) — DOWN,
+    crash-looping. Last successful snapshot **2026-05-06 (~6.5 weeks)** across
+    ALL 5 clients (ana-docker, ana-ml2, esh-docker-vm, esh-vm-db, vm-esh-nas).
+  - **Scheduler:** NOT Backrest (it has **0 plans** — it's just a repo viewer).
+    Backups run via **`resticprofile` systemd timers** (`resticprofile-backup@
+    profile-default.timer`, daily 01:00; `…-check@…` Sun 05:00). The ana timer
+    fires daily and **FAILS SILENTLY** (no alerting → why the gap went unnoticed
+    6.5 wks).
+  **Root cause:** ana-nas NFS fstab bare-`defaults` (no `_netdev`/retry) →
+  `mnt-backup.mount` stuck `failed` → rest-server-ana's bind source
+  `/mnt/backup/restic/repo/ana` resolves to an empty LOCAL ghost dir (no
+  `.htpasswd`) → crash-loop (`cannot load /data/.htpasswd: permission denied`).
+  The REAL repos + `.htpasswd` are intact on ana-nas at
+  `/mnt/backup/restic/repo/ana/{<client>/,.htpasswd}`. ana-nas itself is FINE.
+  **Real exposure, ranked:** (1) 🔴 **ana-ml2** — BARE METAL, no PBS coverage;
+  ana-side restic is its ONLY layer → ~6.5 wks fully unprotected (bespoke
+  at-risk: in-house NVFP4 quants / configs / scripts; bulk `/tank` models are
+  re-downloadable). (2) 🟠 **esh-vm-db** (a DB VM: postgres+mongo) + vm-esh-nas
+  IF PBS-uncovered (verify the esh-pve-nas gap). (3) 🟡 **ana-docker /
+  esh-docker-vm** — VMs, PBS image-CURRENT today, only the granular restic
+  layer is stale (full-VM restore available; no recent file/DB-granular point).
+  **RECOVERY (needs root on ana-docker; the session there = `lkraven`, NO
+  NOPASSWD sudo → operator `!`-sudo or bootstrap infra-ops there):** `sudo
+  mount -a` → `sudo docker restart rest-server` (recreate if the bind doesn't
+  re-resolve onto the NFS; if the ghost persists: stop container → umount → rm
+  the ghost `repo/ana` → remount → start) → verify → next 01:00 succeeds.
+  **PREVENTION (the systemic fixes):** fstab `_netdev`+retry so it self-heals
+  on reboot; a **backup-freshness alert** (the real miss — 6.5 wks unnoticed).
+  Recovery steps mirror `docs/runbooks/disaster-recovery.md` Tier-0.
+  **⚠️ Disclosed creds:** diagnosing dumped the Backrest config, exposing the 5
+  per-repo rest-server basic-auth passwords into the session transcript →
+  rotate after recovery (add to the disclosed-keys rotation queue).
 
 - **ana-docker disk incident RESOLVED — root cause was a 94 GB unrotated
   `langfuse-clickhouse` log, NOT (just) image bloat.** ClickHouse logs
@@ -268,6 +319,33 @@ _101 older entries archived to archival-memory.md._
 
 ## Tried and abandoned
 
+- `[2026-06-20]` **A full disk wedges gitea's Actions notifier SILENTLY — and it
+  STAYS wedged after disk recovers; only a `docker restart gitea` clears it.** After
+  the 94 GB clickhouse disk-full event, gitea stopped CREATING workflow runs on push
+  (push succeeds, ref lands, but NO run spawns — newest run stale, queue empty), while
+  the act_runner stayed registered. NOT run-pickup, NOT disk-now (52% free), NO gitea
+  errors/panics — a silent notifier stall. Hit BOTH vh/arbo (comfy-dev) and vh/Worldtree
+  (worldtree-dev's deploys) → it's global, not repo-specific. Fix: `docker restart gitea`
+  on ana-docker (it had been up 2 weeks, never bounced). A restart does NOT retroactively
+  fire pushes made during the outage — re-push/re-trigger after. Reinforces the deferred
+  docker-daemon-default log-cap fix (the disk-full that started it can't be allowed to recur).
+
+- `[2026-06-20]` **Mistral-Medium-3.5-128B NVFP4 (RecViking) on ana-ml2 GPU 0 —
+  ABANDONED, too slow (operator confirmed).** Stood up as a temporary speed-check
+  displacing mistral-small-4; measured base **~19.2 tok/s** decode on one Blackwell
+  (vs `gen`/Qwopus3.5-122B ~92.7 tok/s warm) → confirmed the operator's "too slow"
+  suspicion. KEEP these serving findings: (1) vLLM **nightly** loads the HF-format
+  NVFP4 weights DIRECTLY — no Mistral native-convert (unlike Small 4, same `mistral3`
+  arch) — via the FlashInfer Cutlass NVFP4 kernel + `turboquant_4bit_nc` KV; TP=1
+  fits the ~70 GB NVFP4 on one 96 GB Blackwell. (2) **EAGLE was a dead end**: the
+  official native FP8 head (`mistralai/Mistral-Medium-3.5-128B-EAGLE`) crashes in
+  CUDA-graph replay with graphs on, and nets NO speedup with `--enforce-eager`
+  (~18.5 vs ~19.2 base) — likely low draft acceptance against the NVFP4
+  (non-official-FP8) base. Untracked workspace files (`stacks/mistral-medium-3.5/` +
+  `playbooks/serve-mistral-medium-3.5.yaml`) removed. ⚠️ Weights still staged on
+  ana-ml2 `/tank/aimodels/mistral-medium-3.5-{nvfp4,eagle}` (~70 GB+) — pending
+  operator OK to free.
+
 - `[2026-06-20]` **A 94 GB unrotated `langfuse-clickhouse` json.log was the
   REAL disk-filler (not image bloat) — there was NO log rotation anywhere.**
   No per-stack `logging:` limit and no docker-daemon default → ClickHouse's