From df7161bbe06fd5150040beb0f82e04efa7d9f732 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 11 Oct 2026 09:01:14 -0700 Subject: [PATCH] memory: U13 E2 through step 6; fv-ml1 evidence capture staged; scheduled 10-11 checks closed --- persistent-memory.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/persistent-memory.md b/persistent-memory.md index ac20f33..3735837 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -132,7 +132,7 @@ _As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections ca - The cleanup bundle offered 10-05: close stale parks 49/46/40; refresh the fv-ml1 README stack tables plus the serial-keyed NVMe table and `zpool replace` line (the 2084-media-error drive is now `/dev/nvme6`, not nvme7); wire ZED pool alerts into althing-alert-bridge; prune the exited one-shot containers on fv-ml1 (keep the `parakeet` rollback). - Push Booth commit `ce095c1`. A trusted HTTPS name for ESH Homepage. nh3-dev reboot (kernel 6.1.0-53 installed). - ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope. -- **Scheduled / unchecked:** 10-11 0540: first weekly sudo-io archiver run (check it). 10-11 0900: infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet. Next scheduled tank scrub on fv-ml1: Sun 10-11 0024 (the September slot was missed); check it ran. +- **Scheduled / unchecked (10-11 all checked):** the first weekly sudo-io archiver ran 0540 clean (success; per-host lines in ops-log, though they read `unattributed(lkraven@nh3-dev)`); infra-hermes deleted both PBS4 one-off VM vzdumps at 0900; fv-ml1's tank scrub ran 0024–0218 with 0 errors. Never `zpool upgrade` any pool yet. - **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`. The LiteLLM compose.yaml on ana-docker differs from `stacks/litellm/compose.yaml` (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%). - **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in. - **Older open items, one line each:** esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (`stacks/blender/README.md`). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran `51168a2e` on 09-27). nh3-dev's root disk is at ~66% since heid's runaway sweep was killed 10-08 (it flapped on the Beszel 85% line before). ESH has a single outside route (esh-scale on esh-pve), noted and untracked. @@ -186,6 +186,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache- ## Recent decisions +- `[2026-10-11]` **Prime 0721: stage fv-ml1 evidence capture; run U13; Mac keys on vh stay full access; delete stale snapshots.** Done: fv-ml1 ZFSBootMenu cmdline `quiet console=ttyS1,115200n8 console=tty0` (SOL = ttyS1, marker-tested; next boot) + nh3-dev user unit `fv-ml1-sol-logger` holding SOL into the journal (marker verified); ERST pstore was already registered, so the 10-11 hang was not a kernel panic. esh-nas-pve `pre-upgrade-20260818` and `subvol-105@pre-vaapi-20260911` destroyed (2.5 GB). U13 E1 (highseat-dev 0733) and E2 steps 1–6 done 0734–0816: reload A `a439431771dd013e`, kernel env (Matrix token + relay credential, kernel restarted), terminal on nh3-dev (Prime approved 0.0.3 at 0813 after the first card's 15-min TTL lapsed), reload B `44be73a252d85fbb` (13 seats adopted), miranda + agent up, Hermes swapped to `svos` (gateway restart 0814:58), bridge on nh3-dev (Prime approved 0.0.2 ~0816; miranda handle push). Open: Prime raises + approves the bridge's 3 runtime grants (owner-only, svos-dev's console snippet), then acceptance (bus test; URGENT page test on his word). ⚠ A local-transcript slip printed the High Seat's Bifrost dispatch key (leaving use with U13). - `[2026-10-11]` **fv-ml1 hard hang ~0005, self-reset ~0008, back by 0015 (all seats).** Journal ends mid-normal-operation (no shutdown), pstore empty, SOL shows nothing (no serial kernel console), BMC restart cause `unknown`, SEL only shows the 00:08 boot events (⚠ the BMC clock is 7 h behind, so ipmitool prints them as 17:08 the day before). Idle at the time (Beszel 0004: CPU 0.15%, no Scriberr job). The sp5100 TCO watchdog exists (60 s). infra-ops sent an ACPI soft-off at 00:12:14 believing it still hung; it landed in early boot before logind watched the button, so no effect. Open (Prime): capture evidence next time (serial console on the kernel cmdline for SOL, and/or kdump); the pre-patch snapshot is HELD. - `[2026-10-10]` **Prime's U13 rulings (~1705): seat roles `dev` + `ops`; artifact host label KEPT `nh3-svos` (old inbox links keep working; infra-ops copies the WHOLE `/var/lib/svos/apps/terminal` (E2 step 2) and `/var/lib/svos/apps/postoffice` (step 6) to nh3-dev at the same paths, owner lkraven, modes kept, each after its nh3-svos unit stops and before the nh3-dev one starts: the artifacts.sqlite index and the bridge's journal must travel with the files); Matrix relay persona `matrix-relay`.** A two-label hybrid is impossible: `svos_connector.artifacts.relative()` refuses any host but the app's one `artifact_host`. Relay secret `nh3-svos/matrix-relay-enrollment-secret` minted. svos-dev's payload: svoperatingsystem `docs/operations/u13-edit-payload.md` @ `7d7ed1f` (enrollment sha256 placeholders: fill at E2 from `secret get | sha256` — never commit them). - `[2026-10-10]` **U13 phase B built (Prime 1639: "implement as asked")**: B1 kernel 0.0.9 + schema 12 (vzdump `vm/111/2026-10-10T23:45:41Z` + offline `pre-afab123` copy first), B2 `svos-kernel.phasefinal.com` → `--app-listen 127.0.0.1:8410` (allowlist nh3-dev; `/auth/kernel-key` ADDED to svos-dev's path list because every connector client fetches it), B3 seat-host origin + bridge env on nh3-dev, B4 app-high-seat origin; then svos-dev's dc241f8 amendment: full method-pinned path list, Miranda's app + agent env and `svos-miranda` origin on nh3-dev, three secrets minted to the vault, app-high-seat declared. Found + fixed: nh3-dev's fleet Caddy shared admin :2019 with ttyd-caddy (SO_REUSEPORT), now :2021. Kernel answers 401 `no principal` through the origin; other sources 403.