From 8f15f6bb0dbee2f85b8627a0d193ec50d3a458ff Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sat, 20 Jun 2026 15:46:14 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20capture=202026-06-20=20session=20?= =?UTF-8?q?=E2=80=94=20Worldtree=20v0.37.7=20demo=20fix,=20gitea=20notifie?= =?UTF-8?q?r=20recovery,=20backup=20diagnosis?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- persistent-memory.md | 104 +++++++++++++++++++++++++++++++++++++------ 1 file changed, 91 insertions(+), 13 deletions(-) diff --git a/persistent-memory.md b/persistent-memory.md index a8bd9ad..69c8144 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -126,19 +126,70 @@ _As of 2026-06-20:_ demo+personal bind-mounts and triggered build-and-deploy via claude-bot. Pinned rides CI. (d372103, d0eb09c, 7819f96) -- **🔴 BACKUP GAP — `rest-server-ana` DOWN ~months on a failed ana-nas NFS - mount (NOT a decommission — docs corrected the recollection).** The - active ana-side restic file+DB target (ana-docker/ana-ml2/esh-docker-vm/ - vm-esh-nas) is crash-looping: `/etc/fstab` has `10.250.50.50:/mnt/backup` - as an NFS mount, but `mnt-backup.mount` is in a `failed` state → /mnt/backup - is the empty local ghost dir → `.htpasswd permission denied` (the - documented disaster-recovery.md failure). ana-nas is pingable; the mount - just isn't attached (bare-`defaults` fstab, no auto-retry). So file+DB - restic for 4 hosts has been failing for ~months (PBS VM-image backups are - the only cover). **DEFERRED to clean context:** the documented recovery - (clear failed mount → rm ghost → `mount /mnt/backup` → restart rest-server) - needs root → infra-ops; plus prevention (fstab `_netdev`/retry + a - backup-freshness alert — the gap went unnoticed for months). +- **Worldtree demo → v0.37.7 GREEN (2026-06-20) — two-half fix.** worldtree-dev's + v0.37.5 config (3fbf338) carried a dangling `granite-4.1-8b` catalog ref → + demo crash-blocked at startup validation. Fix needed BOTH halves: (a) gitea + was silently not creating Actions runs so no deploy could fire (the earlier + disk-full wedged its notifier — `docker restart gitea` cleared it; see + Tried-and-abandoned); (b) the v0.37.7 IMAGE alone wouldn't fix demo because + demo+personal read BIND-MOUNTED config — I re-staged `config/providers.yaml` + + `config/model_roles.yaml` from gitea at **439bebf** byte-identical into BOTH + `/opt/worldtree{,-personal}/config` (sha256-verified across source + both + instances; `granite-4.1-8b`→`summarizer`, `granite-classifier/summarizer`→ + `classifier/summarizer`; old configs saved `.bak-pre-439bebf`; NO container + recreate — avoided the stale-`:latest` footgun). worldtree-dev self-served the + CI re-run via the claude-bot Actions token → **run 1290 GREEN, demo on + v0.37.7**, capability routing live. claude-bot token migration DONE for + worldtree-dev (off vh's personal PAT). **QUEUED (NON-URGENT):** the ratatoskr + readonly-admin tier lives in `config/policies.yaml` (006d233, in 439bebf) — + re-stage that file to personal's bind-mount + reload personal Heimdall, then + mint a `ratatoskr-readonly` key (tier=readonly-admin, distinct from their + send-only c990f0be) + deliver. + +- **🟠 BACKUP DIAGNOSIS (2026-06-20, full probe) — REVISED from "all ana + backups down": PBS + nh3-restic are HEALTHY & CURRENT; only the ANA-side + restic file+DB layer is dead, and the one host with NO other cover is + ana-ml2.** Three layers: + - ✅ **PBS-ANA (VM images)** — healthy, snapshots TODAY. All PFI guests + backed up under PBS namespaces `ana-pve` / `esh-pve` / `nh3-pve` (+ tenant + `sfsrv-pve`); PBS-ANA's own NFS datastore mounts ana-nas fine (1.2T/20T). + Gaps to note: `ana-pve/vm/100` has no snapshots; **there is NO `esh-pve-nas` + PBS namespace** → vm-esh-nas + esh-vm-db may be PBS-UNCOVERED (VERIFY). + - ✅ **restic file+DB nh3-side** (`rest-server-nh3` @ nh3-nas:8000, UP→401) — + irv-ml1 + nh3-docker, last snapshot TODAY 01:00. Repos at + `nh3-nas:/volume1/Backup/restic/{irv-ml1,nh3-docker}`. + - 🔴 **restic file+DB ANA-side** (`rest-server-ana` @ ana-docker:8000) — DOWN, + crash-looping. Last successful snapshot **2026-05-06 (~6.5 weeks)** across + ALL 5 clients (ana-docker, ana-ml2, esh-docker-vm, esh-vm-db, vm-esh-nas). + - **Scheduler:** NOT Backrest (it has **0 plans** — it's just a repo viewer). + Backups run via **`resticprofile` systemd timers** (`resticprofile-backup@ + profile-default.timer`, daily 01:00; `…-check@…` Sun 05:00). The ana timer + fires daily and **FAILS SILENTLY** (no alerting → why the gap went unnoticed + 6.5 wks). + **Root cause:** ana-nas NFS fstab bare-`defaults` (no `_netdev`/retry) → + `mnt-backup.mount` stuck `failed` → rest-server-ana's bind source + `/mnt/backup/restic/repo/ana` resolves to an empty LOCAL ghost dir (no + `.htpasswd`) → crash-loop (`cannot load /data/.htpasswd: permission denied`). + The REAL repos + `.htpasswd` are intact on ana-nas at + `/mnt/backup/restic/repo/ana/{/,.htpasswd}`. ana-nas itself is FINE. + **Real exposure, ranked:** (1) 🔴 **ana-ml2** — BARE METAL, no PBS coverage; + ana-side restic is its ONLY layer → ~6.5 wks fully unprotected (bespoke + at-risk: in-house NVFP4 quants / configs / scripts; bulk `/tank` models are + re-downloadable). (2) 🟠 **esh-vm-db** (a DB VM: postgres+mongo) + vm-esh-nas + IF PBS-uncovered (verify the esh-pve-nas gap). (3) 🟡 **ana-docker / + esh-docker-vm** — VMs, PBS image-CURRENT today, only the granular restic + layer is stale (full-VM restore available; no recent file/DB-granular point). + **RECOVERY (needs root on ana-docker; the session there = `lkraven`, NO + NOPASSWD sudo → operator `!`-sudo or bootstrap infra-ops there):** `sudo + mount -a` → `sudo docker restart rest-server` (recreate if the bind doesn't + re-resolve onto the NFS; if the ghost persists: stop container → umount → rm + the ghost `repo/ana` → remount → start) → verify → next 01:00 succeeds. + **PREVENTION (the systemic fixes):** fstab `_netdev`+retry so it self-heals + on reboot; a **backup-freshness alert** (the real miss — 6.5 wks unnoticed). + Recovery steps mirror `docs/runbooks/disaster-recovery.md` Tier-0. + **⚠️ Disclosed creds:** diagnosing dumped the Backrest config, exposing the 5 + per-repo rest-server basic-auth passwords into the session transcript → + rotate after recovery (add to the disclosed-keys rotation queue). - **ana-docker disk incident RESOLVED — root cause was a 94 GB unrotated `langfuse-clickhouse` log, NOT (just) image bloat.** ClickHouse logs @@ -268,6 +319,33 @@ _101 older entries archived to archival-memory.md._ ## Tried and abandoned +- `[2026-06-20]` **A full disk wedges gitea's Actions notifier SILENTLY — and it + STAYS wedged after disk recovers; only a `docker restart gitea` clears it.** After + the 94 GB clickhouse disk-full event, gitea stopped CREATING workflow runs on push + (push succeeds, ref lands, but NO run spawns — newest run stale, queue empty), while + the act_runner stayed registered. NOT run-pickup, NOT disk-now (52% free), NO gitea + errors/panics — a silent notifier stall. Hit BOTH vh/arbo (comfy-dev) and vh/Worldtree + (worldtree-dev's deploys) → it's global, not repo-specific. Fix: `docker restart gitea` + on ana-docker (it had been up 2 weeks, never bounced). A restart does NOT retroactively + fire pushes made during the outage — re-push/re-trigger after. Reinforces the deferred + docker-daemon-default log-cap fix (the disk-full that started it can't be allowed to recur). + +- `[2026-06-20]` **Mistral-Medium-3.5-128B NVFP4 (RecViking) on ana-ml2 GPU 0 — + ABANDONED, too slow (operator confirmed).** Stood up as a temporary speed-check + displacing mistral-small-4; measured base **~19.2 tok/s** decode on one Blackwell + (vs `gen`/Qwopus3.5-122B ~92.7 tok/s warm) → confirmed the operator's "too slow" + suspicion. KEEP these serving findings: (1) vLLM **nightly** loads the HF-format + NVFP4 weights DIRECTLY — no Mistral native-convert (unlike Small 4, same `mistral3` + arch) — via the FlashInfer Cutlass NVFP4 kernel + `turboquant_4bit_nc` KV; TP=1 + fits the ~70 GB NVFP4 on one 96 GB Blackwell. (2) **EAGLE was a dead end**: the + official native FP8 head (`mistralai/Mistral-Medium-3.5-128B-EAGLE`) crashes in + CUDA-graph replay with graphs on, and nets NO speedup with `--enforce-eager` + (~18.5 vs ~19.2 base) — likely low draft acceptance against the NVFP4 + (non-official-FP8) base. Untracked workspace files (`stacks/mistral-medium-3.5/` + + `playbooks/serve-mistral-medium-3.5.yaml`) removed. ⚠️ Weights still staged on + ana-ml2 `/tank/aimodels/mistral-medium-3.5-{nvfp4,eagle}` (~70 GB+) — pending + operator OK to free. + - `[2026-06-20]` **A 94 GB unrotated `langfuse-clickhouse` json.log was the REAL disk-filler (not image bloat) — there was NO log rotation anywhere.** No per-stack `logging:` limit and no docker-daemon default → ClickHouse's