diff --git a/persistent-memory.md b/persistent-memory.md index dc63302..ebb1800 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -122,7 +122,7 @@ _As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections ca - **Owed by me (no input needed):** - Destroy `zroot/ROOT/debian@pre-patch-20261008` on fv-ml1 ~10-15 if nothing regressed. - nh3-svos housekeeping: releases `6c6abbe` and `a737a06` are no longer referenced by the front door and can be deleted (`ac42ace` holds the same static as the current `b588510`). -- **Watching: Worldtree memory gate.** 10-09 FAILED on filing/user (9, 11, 8 of 12; median 0.75 < 0.80), the first fail that is not #416. Bootstrap from 27 prior runs puts P(median ≤ 9) ≈ 0.06 per batch, and the misses are the habitual fixtures, so it is not called a regression yet. But per-run means drifted 10.6 → 10.0 → 9.7 → 9.3 since the 10-08 fv-ml1 patch (writer + judge = flash-next on GPU 2). If the 10-10 batch also fails filing, infra-hermes frames it as a possible regression after that patch and sends it to me; then investigate flash-next. ⚠ worldtree-dev has not read mail in 6+ days, so the reports dead-end there (Prime's call, below). +- **Worldtree memory gate watch CLOSED 10-10:** batch 20261010T143011Z PASS (user median 0.83 over runs 0.83/0.75/0.92), so the 10-09 filing fail did not repeat and flash-next is not investigated. ⚠ worldtree-dev still is not reading mail (Prime's call, below). - **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens. jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it, as gen-large did (preemptions; decode down to 2–16 tok/s). jekyll's context grows ~1k/call toward the 262k ceiling. It runs on the shared all-agents-local key; a key of its own would make its traffic attributable. - **Waiting on Prime:** - albok `.browse/` share: vh holds every albok read group since 10-09 2244 (Prime: "vh gets albok-everything"; four Mac keys authorized for vh). The macOS run of the withdrawal test is staged on nh3-albok and waits on Prime running the one-liner in `scripts/oneoff/albok-browse-withdrawal/README.md`; then read `~vh/albok-browse-test/results/`, report to albok-dev, remove the replica. @@ -132,7 +132,7 @@ _As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections ca - The cleanup bundle offered 10-05: close stale parks 49/46/40; refresh the fv-ml1 README stack tables plus the serial-keyed NVMe table and `zpool replace` line (the 2084-media-error drive is now `/dev/nvme6`, not nvme7); wire ZED pool alerts into althing-alert-bridge; prune the exited one-shot containers on fv-ml1 (keep the `parakeet` rollback). - Push Booth commit `ce095c1`. A trusted HTTPS name for ESH Homepage. nh3-dev reboot (kernel 6.1.0-53 installed). - ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope. -- **Scheduled / unchecked:** esh-nas-pve `nvme/ROOT@pre-pve9` destroy ~10-10 (ours). 10-11 0540: first weekly sudo-io archiver run (check it). 10-11 0900: infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet. Next scheduled tank scrub on fv-ml1: Sun 10-11 0024 (the September slot was missed); check it ran. +- **Scheduled / unchecked:** 10-11 0540: first weekly sudo-io archiver run (check it). 10-11 0900: infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet. Next scheduled tank scrub on fv-ml1: Sun 10-11 0024 (the September slot was missed); check it ran. - **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`. The LiteLLM compose.yaml on ana-docker differs from `stacks/litellm/compose.yaml` (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%). - **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in. - **Older open items, one line each:** esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (`stacks/blender/README.md`). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran `51168a2e` on 09-27). nh3-dev's root disk is at ~66% since heid's runaway sweep was killed 10-08 (it flapped on the Beszel 85% line before). ESH has a single outside route (esh-scale on esh-pve), noted and untracked. @@ -186,6 +186,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache- ## Recent decisions +- `[2026-10-10]` **esh-nas-pve `nvme/ROOT@pre-pve9` destroyed 0750 (607M)** after a week clean on PVE 9.2.21 / 7.0.14-20 (up 6 d, no err-level journal in 7 d, pools healthy, all guests up; only `openipmi.service` failed, the known item). Left: `nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z` (now 1.43G, a pre-PVE-9 rollback point that is useless as one) and `nvme/subvol-105-disk-0@pre-vaapi-20260911`. - `[2026-10-09]` **vh gets every albok read group (Prime 2244: "vh gets albok-everything")**: albok-read, albok-personal and albok-feedback (the restricted wing's default-deny ends for vh only), plus his four Macs' keys on vh (`playbooks/nh3-albok-vh-viewer.yaml`). No net new root path: those Macs already reach root there via lkraven@nh3-dev and the infra-ops key. Later restricted groups (1513+) are NOT included. - `[2026-10-09]` **nh3-dev: orphaned svos pytest killed 2242** (14.4 GB RSS, +2.3 GB/h, swap 100%, Beszel 90% alert; SIGTERM ignored, SIGKILL; used 25.4 → 11.3 GB). highseat-dev's retry-loop test with an instant fake sleep; they added pytest-timeout 300 s and a 50-attempt bound. - `[2026-10-09]` **Post-clear checks closed (2015–2025 PT):** albok `.browse/` sshfs pathname-withdrawal test PASSED (replica on nh3-albok, Linux sshfs 3.7.3 client in a container; fresh open ENOENT 5/5 default and 5/5 hour-long caches; null + positive controls ok; failed rename = path served; reported to albok-dev) → `scripts/oneoff/albok-browse-withdrawal/README.md`. fv-ml1 plateau (Beszel 120m, 42 h after the 0034 10-08 restart): GPU 0 flat at 5,606 MiB free (~3.9 GB at parakeet's cap), GPU 1 median 9,459 free (~3.3 GB at intern-decision's 32k peak), no creep; GPU 2 643 MiB free = flash-next util 0.96, single tenant, by design. nh3-nas `/volume1` 71% (79% on 10-07). The "sfsrv-ana pre-pbs4" item was a mislabel: the pre-pbs4 snapshots were pfi-pve VM 100 and nh3-pve VM 105, dropped by infra-hermes 10-05 and read gone 10-09.