From 5202847abb9cc348c24cd9c6f076c691a36fbb23 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 11 Oct 2026 13:54:24 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=202026-10-11=20U?= =?UTF-8?q?13=20cutover=20executed=20+=20accepted=20(one=20card=20+=20svos?= =?UTF-8?q?-dev=20answers=20open),=20fv-ml1=20hang=20+=20evidence=20captur?= =?UTF-8?q?e,=20stale=20snapshots=20gone;=206=20entries=20archived?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- archival-memory.md | 22 ++++++++ ...-09-26-coder-seat-stays-on-fv-ml1-prime.md | 3 -- ...h-as-parity-for-a-small-generative-seat.md | 3 -- ...post-clear-checks-albok-withdrawal-pass.md | 3 ++ .../2026-10-09-u13-ingress-approved.md | 3 ++ .../2026-10-09-vh-albok-everything.md | 3 ++ ...26-10-10-esh-nas-pve-pre-pve9-destroyed.md | 5 ++ .../2026-10-10-u13-phase-b-built.md | 3 ++ .../2026-10-10-u13-rulings-p1-p3.md | 3 ++ .../2026-10-11-fv-ml1-hard-hang.md | 3 ++ .../2026-10-11-u13-cutover-executed.md | 23 ++++++++ persistent-memory.md | 54 +++++++------------ 12 files changed, 88 insertions(+), 40 deletions(-) delete mode 100644 persistent-memory.d/2026-09-26-coder-seat-stays-on-fv-ml1-prime.md delete mode 100644 persistent-memory.d/2026-09-26-greedy-exact-match-as-parity-for-a-small-generative-seat.md create mode 100644 persistent-memory.d/2026-10-09-post-clear-checks-albok-withdrawal-pass.md create mode 100644 persistent-memory.d/2026-10-09-u13-ingress-approved.md create mode 100644 persistent-memory.d/2026-10-09-vh-albok-everything.md create mode 100644 persistent-memory.d/2026-10-10-esh-nas-pve-pre-pve9-destroyed.md create mode 100644 persistent-memory.d/2026-10-10-u13-phase-b-built.md create mode 100644 persistent-memory.d/2026-10-10-u13-rulings-p1-p3.md create mode 100644 persistent-memory.d/2026-10-11-fv-ml1-hard-hang.md create mode 100644 persistent-memory.d/2026-10-11-u13-cutover-executed.md diff --git a/archival-memory.md b/archival-memory.md index 4c86251..b26984a 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -15292,3 +15292,25 @@ latency-sensitive. - `[2026-09-25]` **elway against an unpinned host-key name** — every `when:` hit ssh rc 255, so all 9 steps showed SKIPPED, not failed. Only the verify phase went red. Pin the name (the IP key matched) or use the IP. Untracked elway defect: rc 255 in `when:` should fail. _Archived 2026-10-09._ + +## Recent decisions (archived 2026-10-11 batch) + +- `[2026-09-26]` **GPU-LXC Temperature alerts watch the GPU, not the host CPU** (`SENSORS=-coretemp_*,acpitz`); hypervisor CPU alerts at >95 °C on nh3-pve/esh-pve. The first nh3-ml1 alert was nh3-pve's CPU during vzdump. `6d901c0`. + _Archived 2026-10-11._ + +- `[2026-09-26]` **Abliterated LFM2.5-VL-3B parallel seat (:8032) passed brokkr's 6-image eval and stays** (direct only; stock seat kept for SFW A/B). `9cc3824`, `944bb36`. + _Archived 2026-10-11._ + +- `[2026-09-26]` **VibeVoice ASR → Q8_0 (Prime).** WER on the 4 bundled LibriSpeech clips 3/69 → 2/69 (only the I'm/I am artifact left), ×3 identical; RTF 0.09–0.17; +1.1 GB VRAM (nh3-ml1 ~11.3/16 GB). Q4_K file removed. + _Archived 2026-10-11._ + +- **Coder seat STAYS on fv-ml1 (Prime).** The nh3-ml1 copy gave the same quality (teacher-forced true-code logprob diff +0.008 ± 0.019) but ran ~5× slower (64-tok FIM ~1.0 s vs 0.2 s; 63 vs 338 tok/s), and freeing 6.3 GB on fv-ml1 GPU 1 (20 GB spare) bought little. Copy removed; recipe kept in `stacks/coder-seat/`. The RTX 2000E suits embed/rerank/classify, not latency-sensitive generation. Only coder was a candidate: parakeet stays (earlier ruling), voices is a generation seat. + _Archived 2026-10-11._ + +## Tried and abandoned (archived 2026-10-11 batch) + +- **Greedy exact-match as parity for a small generative seat (coder)** — same-host repeats matched only 30–38% (run B hit the prefix cache, so a different numeric path; near-tie tokens cascade). Use teacher-forced logprobs of real text with `cache_salt`: each host bit-exact with itself. `stacks/coder-seat/README.md`. + _Archived 2026-10-11._ + +- `[2026-09-26]` **zsh traps in ad-hoc test loops:** `set -- $a` does NOT word-split, so alert POSTs went out with empty fields; and `local path=…` inside a function CLOBBERS `$PATH` (zsh's tied array), so the failover test ran nothing while TEI sat stopped. Use other names and `${=var}`. + _Archived 2026-10-11._ diff --git a/persistent-memory.d/2026-09-26-coder-seat-stays-on-fv-ml1-prime.md b/persistent-memory.d/2026-09-26-coder-seat-stays-on-fv-ml1-prime.md deleted file mode 100644 index e38a1ec..0000000 --- a/persistent-memory.d/2026-09-26-coder-seat-stays-on-fv-ml1-prime.md +++ /dev/null @@ -1,3 +0,0 @@ -# `[2026-09-26]` Coder seat STAYS on fv-ml1 (Prime). - -**Coder seat STAYS on fv-ml1 (Prime).** The nh3-ml1 copy gave the same quality (teacher-forced true-code logprob diff +0.008 ± 0.019) but ran ~5× slower (64-tok FIM ~1.0 s vs 0.2 s; 63 vs 338 tok/s), and freeing 6.3 GB on fv-ml1 GPU 1 (20 GB spare) bought little. Copy removed; recipe kept in `stacks/coder-seat/`. The RTX 2000E suits embed/rerank/classify, not latency-sensitive generation. Only coder was a candidate: parakeet stays (earlier ruling), voices is a generation seat. diff --git a/persistent-memory.d/2026-09-26-greedy-exact-match-as-parity-for-a-small-generative-seat.md b/persistent-memory.d/2026-09-26-greedy-exact-match-as-parity-for-a-small-generative-seat.md deleted file mode 100644 index 0e4e5b2..0000000 --- a/persistent-memory.d/2026-09-26-greedy-exact-match-as-parity-for-a-small-generative-seat.md +++ /dev/null @@ -1,3 +0,0 @@ -# `[2026-09-26]` Greedy exact-match as parity for a small generative seat (coder) - -**Greedy exact-match as parity for a small generative seat (coder)** — same-host repeats matched only 30–38% (run B hit the prefix cache, so a different numeric path; near-tie tokens cascade). Use teacher-forced logprobs of real text with `cache_salt`: each host bit-exact with itself. `stacks/coder-seat/README.md`. diff --git a/persistent-memory.d/2026-10-09-post-clear-checks-albok-withdrawal-pass.md b/persistent-memory.d/2026-10-09-post-clear-checks-albok-withdrawal-pass.md new file mode 100644 index 0000000..f1a0a64 --- /dev/null +++ b/persistent-memory.d/2026-10-09-post-clear-checks-albok-withdrawal-pass.md @@ -0,0 +1,3 @@ +# 2026-10-09-post-clear-checks-albok-withdrawal-pass + +`[2026-10-09]` **Post-clear checks closed (2015–2025 PT):** albok `.browse/` sshfs pathname-withdrawal test PASSED (replica on nh3-albok, Linux sshfs 3.7.3 client in a container; fresh open ENOENT 5/5 default and 5/5 hour-long caches; null + positive controls ok; failed rename = path served; reported to albok-dev) → `scripts/oneoff/albok-browse-withdrawal/README.md`. fv-ml1 plateau (Beszel 120m, 42 h after the 0034 10-08 restart): GPU 0 flat at 5,606 MiB free (~3.9 GB at parakeet's cap), GPU 1 median 9,459 free (~3.3 GB at intern-decision's 32k peak), no creep; GPU 2 643 MiB free = flash-next util 0.96, single tenant, by design. nh3-nas `/volume1` 71% (79% on 10-07). The "sfsrv-ana pre-pbs4" item was a mislabel: the pre-pbs4 snapshots were pfi-pve VM 100 and nh3-pve VM 105, dropped by infra-hermes 10-05 and read gone 10-09. diff --git a/persistent-memory.d/2026-10-09-u13-ingress-approved.md b/persistent-memory.d/2026-10-09-u13-ingress-approved.md new file mode 100644 index 0000000..1775bde --- /dev/null +++ b/persistent-memory.d/2026-10-09-u13-ingress-approved.md @@ -0,0 +1,3 @@ +# 2026-10-09-u13-ingress-approved + +`[2026-10-09]` **APPROVED by Prime (10-09), BUILD DEFERRED until after his snapshot/clear: SVOS app-traffic ingress for U13** (apps on nh3-dev reach the nh3-svos kernel). Shape agreed with svos-dev: (1) kernel `--app-listen 127.0.0.1:8410`, which never trusts loopback (svos-dev builds it in U9a); (2) a Caddy origin `svos-kernel.phasefinal.com` on 10.100.50.43, no Basic auth, bearer passed, app paths only, allowlisted to nh3-dev; (3) kernel→bridge over plain LAN HTTP; kernel→seat host (terminal) via nh3-dev's wildcard Caddy `svos-seathost.nh3.phasefinal.com` (TLS, because the leg carries secrets). nh3-dev is NOT on the mesh; the contracts now say "fleet network". `--app-listen` is ON MAIN (`68c9933`, 10-09). It ships in kernel **0.0.9** with schema **11 → 12** (migration 0012 rebuilds dispatches + summons, so the 'schema 11' acceptance line changes and a pre-refresh backup is mandatory). At the cutover the bridge gets three owner-approved runtime grants (app-postoffice → `app:app-miranda` write, `app:app-postoffice` write, `ns:pages` write, all autonomous) and installs as `svos-postoffice[push]` with `althing-core` from `~/development/althing` in apps-env. **Release is cut: svoperatingsystem `afab123` (tag v0.0.9)**: kernel 0.0.9, svos-postoffice 0.0.2, svos-terminal 0.0.3, svos-connector 0.0.7, svos-agent 0.0.3, svos-miranda 0.0.2 (U9a + U5b). The SVOS side of the cutover is written up in `docs/operations/u13-svos-edit.md` (app listener on nh3-svos, terminal app moved to nh3-dev with `[seat_host]`, bridge on nh3-dev with the converse mapping). Gotchas: the terminal app serves WITHOUT its seat host if the socket or launch dir fails (one stderr line); `serve` refuses `--app-listen` on the primary port. Build only when the edit is scheduled (highseat-dev owns the schedule) and NOT before the snapshot; the runbook refresh is ours at deploy. Tracking: svos-dev thread `01M4GPXSBKQHCQ30FVPH4R3N7G`. diff --git a/persistent-memory.d/2026-10-09-vh-albok-everything.md b/persistent-memory.d/2026-10-09-vh-albok-everything.md new file mode 100644 index 0000000..e333ae1 --- /dev/null +++ b/persistent-memory.d/2026-10-09-vh-albok-everything.md @@ -0,0 +1,3 @@ +# 2026-10-09-vh-albok-everything + +`[2026-10-09]` **vh gets every albok read group (Prime 2244: "vh gets albok-everything")**: albok-read, albok-personal and albok-feedback (the restricted wing's default-deny ends for vh only), plus his four Macs' keys on vh (`playbooks/nh3-albok-vh-viewer.yaml`). No net new root path: those Macs already reach root there via lkraven@nh3-dev and the infra-ops key. Later restricted groups (1513+) are NOT included. diff --git a/persistent-memory.d/2026-10-10-esh-nas-pve-pre-pve9-destroyed.md b/persistent-memory.d/2026-10-10-esh-nas-pve-pre-pve9-destroyed.md new file mode 100644 index 0000000..163b32f --- /dev/null +++ b/persistent-memory.d/2026-10-10-esh-nas-pve-pre-pve9-destroyed.md @@ -0,0 +1,5 @@ +# 2026-10-10-esh-nas-pve-pre-pve9-destroyed + +`[2026-10-10]` **esh-nas-pve `nvme/ROOT@pre-pve9` destroyed 0750 (607M)** after a week clean on PVE 9.2.21 / 7.0.14-20 (up 6 d, no err-level journal in 7 d, pools healthy, all guests up; only `openipmi.service` failed, the known item). Left: `nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z` (now 1.43G, a pre-PVE-9 rollback point that is useless as one) and `nvme/subvol-105-disk-0@pre-vaapi-20260911`. + +`[2026-10-11]` On Prime's "delete stale snapshots": `nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z` (1.43G, a pre-PVE-9 rollback point that was useless as one) and `nvme/subvol-105-disk-0@pre-vaapi-20260911` (1.04G, a raw zfs snapshot, not in `pct listsnapshot 105`) destroyed via elway after a dry run; `nvme` now has no snapshots. diff --git a/persistent-memory.d/2026-10-10-u13-phase-b-built.md b/persistent-memory.d/2026-10-10-u13-phase-b-built.md new file mode 100644 index 0000000..693932d --- /dev/null +++ b/persistent-memory.d/2026-10-10-u13-phase-b-built.md @@ -0,0 +1,3 @@ +# 2026-10-10-u13-phase-b-built + +`[2026-10-10]` **U13 phase B built (Prime 1639: "implement as asked")**: B1 kernel 0.0.9 + schema 12 (vzdump `vm/111/2026-10-10T23:45:41Z` + offline `pre-afab123` copy first), B2 `svos-kernel.phasefinal.com` → `--app-listen 127.0.0.1:8410` (allowlist nh3-dev; `/auth/kernel-key` ADDED to svos-dev's path list because every connector client fetches it), B3 seat-host origin + bridge env on nh3-dev, B4 app-high-seat origin; then svos-dev's dc241f8 amendment: full method-pinned path list, Miranda's app + agent env and `svos-miranda` origin on nh3-dev, three secrets minted to the vault, app-high-seat declared. Found + fixed: nh3-dev's fleet Caddy shared admin :2019 with ttyd-caddy (SO_REUSEPORT), now :2021. Kernel answers 401 `no principal` through the origin; other sources 403. diff --git a/persistent-memory.d/2026-10-10-u13-rulings-p1-p3.md b/persistent-memory.d/2026-10-10-u13-rulings-p1-p3.md new file mode 100644 index 0000000..809b54d --- /dev/null +++ b/persistent-memory.d/2026-10-10-u13-rulings-p1-p3.md @@ -0,0 +1,3 @@ +# 2026-10-10-u13-rulings-p1-p3 + +`[2026-10-10]` **Prime's U13 rulings (~1705): seat roles `dev` + `ops`; artifact host label KEPT `nh3-svos` (old inbox links keep working; infra-ops copies the WHOLE `/var/lib/svos/apps/terminal` (E2 step 2) and `/var/lib/svos/apps/postoffice` (step 6) to nh3-dev at the same paths, owner lkraven, modes kept, each after its nh3-svos unit stops and before the nh3-dev one starts: the artifacts.sqlite index and the bridge's journal must travel with the files); Matrix relay persona `matrix-relay`.** A two-label hybrid is impossible: `svos_connector.artifacts.relative()` refuses any host but the app's one `artifact_host`. Relay secret `nh3-svos/matrix-relay-enrollment-secret` minted. svos-dev's payload: svoperatingsystem `docs/operations/u13-edit-payload.md` @ `7d7ed1f` (enrollment sha256 placeholders: fill at E2 from `secret get | sha256` — never commit them). diff --git a/persistent-memory.d/2026-10-11-fv-ml1-hard-hang.md b/persistent-memory.d/2026-10-11-fv-ml1-hard-hang.md new file mode 100644 index 0000000..49db1a8 --- /dev/null +++ b/persistent-memory.d/2026-10-11-fv-ml1-hard-hang.md @@ -0,0 +1,3 @@ +# 2026-10-11-fv-ml1-hard-hang + +`[2026-10-11]` **fv-ml1 hard hang ~0005, self-reset ~0008, back by 0015 (all seats).** Journal ends mid-normal-operation (no shutdown), pstore empty, SOL shows nothing (no serial kernel console), BMC restart cause `unknown`, SEL only shows the 00:08 boot events (⚠ the BMC clock is 7 h behind, so ipmitool prints them as 17:08 the day before). Idle at the time (Beszel 0004: CPU 0.15%, no Scriberr job). The sp5100 TCO watchdog exists (60 s). infra-ops sent an ACPI soft-off at 00:12:14 believing it still hung; it landed in early boot before logind watched the button, so no effect. Open (Prime): capture evidence next time (serial console on the kernel cmdline for SOL, and/or kdump); the pre-patch snapshot is HELD. diff --git a/persistent-memory.d/2026-10-11-u13-cutover-executed.md b/persistent-memory.d/2026-10-11-u13-cutover-executed.md new file mode 100644 index 0000000..e19d512 --- /dev/null +++ b/persistent-memory.d/2026-10-11-u13-cutover-executed.md @@ -0,0 +1,23 @@ +# 2026-10-11-u13-cutover-executed + +`[2026-10-11]` **Prime 0721: stage fv-ml1 evidence capture; run U13; Mac keys on vh stay full access; delete stale snapshots.** Done: fv-ml1 ZFSBootMenu cmdline `quiet console=ttyS1,115200n8 console=tty0` (SOL = ttyS1, marker-tested; next boot) + nh3-dev user unit `fv-ml1-sol-logger` holding SOL into the journal (marker verified); ERST pstore was already registered, so the 10-11 hang was not a kernel panic. esh-nas-pve `pre-upgrade-20260818` and `subvol-105@pre-vaapi-20260911` destroyed (2.5 GB). U13 E1 (highseat-dev 0733) and E2 steps 1–6 done 0734–0816: reload A `a439431771dd013e`, kernel env (Matrix token + relay credential, kernel restarted), terminal on nh3-dev (Prime approved 0.0.3 at 0813 after the first card's 15-min TTL lapsed), reload B `44be73a252d85fbb` (13 seats adopted), miranda + agent up, Hermes swapped to `svos` (gateway restart 0814:58), bridge on nh3-dev (Prime approved 0.0.2 ~0816; miranda handle push). Open: Prime raises + approves the bridge's 3 runtime grants (owner-only, svos-dev's console snippet), then acceptance (bus test; URGENT page test on his word). ⚠ A local-transcript slip printed the High Seat's Bifrost dispatch key (leaving use with U13). + +## The edit, step by step (2026-10-11, PT) + +- **0721 go** (Prime, "run u13"). His charter review (B8) is treated as covered by the go (highseat-dev agreed: nothing in the edit depends on it). +- **E0/E1 (highseat-dev):** zero pending cards; svos `987746d` deletes the High Seat's `miranda:` AND `hermes:` blocks (the config parses miranda whenever hermes is on); restart 0733:00, board back 0734:43; journal `matrix inbound DISABLED` 0734:41, `app-high-seat REGISTERED rev 5c466c9f1f0c` (pinned directly, no card); `miranda` handle → pull. +- **E2.1 reload A** (`scratchpad u13-reload-A.yaml` shape: back up `instance.toml.pre-u13-A`, two exact `base_url` swaps asserted once each, append the payload's additions with the three enrollment sha256s filled from the vault, `check-config`, restore the backup on refusal, `systemctl reload`): `1bd948c754e571da → a439431771dd013e`. +- **Enrollments over `svos-kernel.phasefinal.com`** (`POST /auth/enroll`, `Authorization: Bearer sve_…` → `{"credential":"svc_…"}`): app-miranda → vault `nh3-svos/svos-miranda-credential` → `/etc/svos/env/miranda.env`; persona `matrix-relay` → vault `nh3-svos/matrix-relay-credential`. ⚠ The kernel reads `SVOS_MATRIX_TOKEN` / `SVOS_RELAY_CREDENTIAL` only at START: `/etc/svos/kernel.env` (root 0600; there was none before) needed a kernel RESTART (~0738), which the payload did not place. svos-dev confirmed the Matrix transport live from 0734:48 (it logs only failures). +- **E2.2 terminal:** nh3-svos unit stopped + disabled, hold drop-in removed; `/var/lib/svos/apps/terminal` copied by `sudo tar | tar -xpf` (tar is exempt from sudo-io `log_output`), byte-identical. svos-terminal 0.0.3 as an lkraven USER unit. ⚠ **Manifest cards expire after 15 min**: the first lapsed at 0753 unapproved and the kernel discarded that revision; svos-dev re-raised by restarting the user unit (0811); Prime approved rev 3 at 0813. The browser origin `svos-terminal.phasefinal.com` now upstreams to `https://svos-seathost.nh3.phasefinal.com` with `header_up Host {upstream_hostport}` (the payload omitted this; without it the pane breaks). +- **E2.3 reload B:** `a439431771dd013e → 44be73a252d85fbb`; `/seats` complete, profiles configured, 13 seats adopted live (dev-* ×12 + ops-infra). +- **E2.4:** svos-miranda + svos-agent user units; the agent enrolled on first start from `/etc/svos/env/agent-enroll.env` (then deleted; it refuses to start with both) and wrote `/etc/svos/miranda/agent.credential`. app-miranda registered + pinned directly (0.0.3, up). Spent enrollment secrets removed from the vault. +- **E2.5 Hermes swap** (`scratchpad u13-hermes-swap.py`: two `svos_miranda` list items → `svos`, an `entries.svos.settings` block, top-level `tools.tool_search.enabled: "off"`, YAML asserted, backup `config.yaml.pre-u13`); `systemctl --user restart hermes-gateway` 08:14:58; svos-agent logged `handshake session=hermes-nh3-dev tools=23`. +- **E2.6 bridge:** nh3-svos postoffice stopped/disabled, hold drop-in removed; `/var/lib/svos/apps/postoffice` (journal + wal/shm, mail, provider) copied byte-identical; svos-postoffice on nh3-dev `10.100.10.50:8406`, `push route miranda: live`, `postbox --handle miranda status` → push/reachable. Prime approved 0.0.2 (~0816). The three bridge runtime grants are **owner-only** (`kernel/grant` is a human principal over the front door): Prime raised them with svos-dev's browser-console snippet and approved (three `/verbs/kernel/grant` 202 + three card decisions, before 1350). +- **Acceptance (1350):** URGENT mail from infra-ops → bridge ingested and dispatched (200) → `svos` reply on the thread at 1350:54 saying the page fired (text, then speech, per the connector receipt). Miranda's explicit reply to infra-ops is held on card `05cf119e…`, because the payload grants `miranda → app:app-postoffice write` WITHOUT `autonomous`. Two observations to svos-dev (thread `01M4NNGCM3X75DANGXQEQ1EQGX`): the text on the thread reads as written to Prime; is every bus reply meant to need Prime's approval? Health 1416: up 6, degraded 1 (the terminal; reason not surfaced), down 0. + +## Gotchas paid for during the edit + +- nh3-dev's fleet Caddy shared admin `localhost:2019` with ttyd-caddy through SO_REUSEPORT, so `reload` landed on either process; it now uses `admin localhost:2021` (see `servers/nh3-dev/README.md`). +- A `caddy validate` of a file under a 0700 staging dir fails "permission denied" as user caddy; a `| tail` hid it once. Validate from `/etc/caddy/Caddyfile.check` and check the exit code (positive-controlled with a bogus directive → rc 1). +- `secret` (Vaultwarden via `bw`) calls are serialized and can take minutes each when another session is reading the vault; run them in the background. +- Printing `~/.hermes/config.yaml` sections exposed the High Seat's Bifrost dispatch key in the local transcript; Prime ruled no rotation (10-11) since the key leaves use with U13. diff --git a/persistent-memory.md b/persistent-memory.md index 3735837..9e519d7 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-10-09 ~1955 PT (nh3-svos: SVOS served on the LAN too, voice live, refreshed to ac42ace then kernel 0.0.8 — thaw race and the SQLite-connection fd outage both fixed; SVOS app-traffic ingress for U13 APPROVED, build deferred; fv-ml1 off the mesh by design; albok library building + token swap; Worldtree gate 10-09 filing FAIL under watch; Homepage descriptions.)_ +_Last updated: 2026-10-11 ~1400 PT (U13 cutover executed + acceptance; fv-ml1 hard hang and evidence capture staged; esh-nas-pve stale snapshots destroyed; vh albok-everything; eshpfi pushed.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -115,24 +115,16 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched._ -### 2026-10-09: live now (as of 2026-10-09 ~1955 PT) +### 2026-10-11: live now (as of 2026-10-11 ~1400 PT) -- **Nothing is mid-action.** No claims held (`scripts/ops-log claims`). eshpfi is pushed through this snapshot's commit. ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- ` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose (auto-regenerated). -- **U13 phase B (infra) DONE 2026-10-10 1646–1650 on Prime's go:** nh3-svos on v0.0.9 (`afab123`, schema 12) with app-terminal + app-postoffice HELD on b588510 by drop-ins (remove at the edit; keep `releases/b588510` until then); kernel app origin `svos-kernel.phasefinal.com` (LAN, nh3-dev only); `svos-seathost` + `svos-highseat` + `svos-miranda` origins on nh3-dev (nh3-svos only); apps env `/opt/svos/current/apps-env` (dc241f8: bridge, terminal, miranda 0.0.3, agent) on nh3-dev; three enrollment secrets vaulted, app-high-seat declared (B5). Next is the atomic edit (highseat-dev's runbook `svos/docs/u13-cutover-runbook.md`, Prime's go; E2 is S→I). → `servers/nh3-svos/README.md`, `servers/nh3-dev/README.md` -- **Owed by me (no input needed):** - - ⚠ **HOLD** the destroy of `zroot/ROOT/debian@pre-patch-20261008` on fv-ml1 (was ~10-15): fv-ml1 hard-hung 2026-10-11 ~0005, the first crash since that patch (NVIDIA 580.178.04, kernel 6.12.111). Keep the rollback point until the cause is known or a clean week passes after the last crash. - - nh3-svos housekeeping: releases `6c6abbe` and `a737a06` are no longer referenced by the front door and can be deleted (`ac42ace` holds the same static as the current `b588510`). -- **Worldtree memory gate watch CLOSED 10-10:** batch 20261010T143011Z PASS (user median 0.83 over runs 0.83/0.75/0.92), so the 10-09 filing fail did not repeat and flash-next is not investigated. ⚠ worldtree-dev still is not reading mail (Prime's call, below). -- **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens. jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it, as gen-large did (preemptions; decode down to 2–16 tok/s). jekyll's context grows ~1k/call toward the 262k ceiling. It runs on the shared all-agents-local key; a key of its own would make its traffic attributable. -- **Waiting on Prime:** - - albok `.browse/` share: vh holds every albok read group since 10-09 2244 (Prime: "vh gets albok-everything"; four Mac keys authorized for vh). The macOS run of the withdrawal test is staged on nh3-albok and waits on Prime running the one-liner in `scripts/oneoff/albok-browse-withdrawal/README.md`; then read `~vh/albok-browse-test/results/`, report to albok-dev, remove the replica. - - Who sees Worldtree gate failures while worldtree-dev is not reading mail (Prime, Miranda, or keep waiting). - - Rotate the fv-ml1 BMC password? It was printed into a local session transcript on 10-08 (never left nh3-dev); infra-ops leans no. - - albok's 200-object load check, 2.01× against a 2× bar (albok-dev: the ruling is the operator's). - - The cleanup bundle offered 10-05: close stale parks 49/46/40; refresh the fv-ml1 README stack tables plus the serial-keyed NVMe table and `zpool replace` line (the 2084-media-error drive is now `/dev/nvme6`, not nvme7); wire ZED pool alerts into althing-alert-bridge; prune the exited one-shot containers on fv-ml1 (keep the `parakeet` rollback). - - Push Booth commit `ce095c1`. A trusted HTTPS name for ESH Homepage. nh3-dev reboot (kernel 6.1.0-53 installed). - - ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope. -- **Scheduled / unchecked (10-11 all checked):** the first weekly sudo-io archiver ran 0540 clean (success; per-host lines in ops-log, though they read `unattributed(lkraven@nh3-dev)`); infra-hermes deleted both PBS4 one-off VM vzdumps at 0900; fv-ml1's tank scrub ran 0024–0218 with 0 errors. Never `zpool upgrade` any pool yet. +- **Nothing is mid-action.** No claims held. eshpfi is pushed through `df7161b`; this snapshot's commit follows. ⚠ The working tree and index are shared with infra-hermes and subagents: commit with `git commit -- `. `graphify-out/GRAPH_REPORT.md` stays modified on purpose. +- **U13 cutover: executed 10-11 0721–0816, acceptance 1350.** Miranda now runs in SVOS: app-miranda + svos-agent on nh3-dev, the Hermes gateway on plugin `svos`, and the bridge app-postoffice on nh3-dev owning the `miranda` push route. The URGENT bus test (thread `01M4PBNX5PP0PGHDTY5BG7HG95`) ingested, paged per the connector receipt, and a reply came back on the thread from `svos`. → `persistent-memory.d/2026-10-11-u13-cutover-executed.md` + - **Open:** Prime approves card `05cf119e…` (Miranda's own bus reply to infra-ops, held because `miranda → app:app-postoffice write` is not autonomous). svos-dev owes answers on that grant, on summary-vs-reply on the thread, and on whether the speech leg played (10-11 summons speech was `synthesis_failed` at 15.0 s; if it recurs, `request_timeout_seconds = 30` in `[notifications.matrix]` at the next reload). Phase R (24 h soak, doctrine + broadcast texts, High Seat removal) is highseat-dev's (svos `docs/u13-cutover-runbook.md`). + - **Rollback material, keep until the soak ends:** nh3-svos `releases/b588510`, `instance.toml.pre-u13-{A,B}`, `/etc/svos/kernel.env.pre-u13` (absent: there was none); nh3-dev `~/.hermes/config.yaml.pre-u13` and `~/.hermes/plugins/svos_miranda`. After the soak: nh3-svos releases `6c6abbe`, `a737a06`, `b588510` and nh3-dev `/opt/svos/releases/afab123` (unreferenced) can go. +- **fv-ml1: hard hang 10-11 ~0005, self-reset, no record** (pstore/ERST registered and empty, so not a kernel panic). Evidence capture staged: kernel console on ttyS1 from its next boot + nh3-dev user unit `fv-ml1-sol-logger` (`journalctl --user -u fv-ml1-sol-logger`). ⚠ **HOLD** `zroot/ROOT/debian@pre-patch-20261008` (do not destroy) until the cause is known or a clean week passes after the last crash. +- **albok `.browse/` share:** the Linux-client withdrawal test passed (10-09); vh holds every albok read group; Prime's Mac keys on vh stay full access (Prime 10-11). Owed: the macOS run, staged as vh in `~vh/albok-browse-test/` on nh3-albok, waits on Prime pasting the one-liner (`scripts/oneoff/albok-browse-withdrawal/README.md`); afterwards read `results/`, report to albok-dev, remove the replica. +- **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens; jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it. It runs on the shared all-agents-local key. +- **Waiting on Prime:** who sees Worldtree gate failures while worldtree-dev is not reading mail; rotate the fv-ml1 BMC password (printed into a local transcript 10-08; infra-ops leans no); albok's 2.01× load check against a 2× bar; the 10-05 cleanup bundle (stale parks 49/46/40, fv-ml1 README stack + NVMe tables, ZED alerts into althing-alert-bridge, prune exited one-shot containers on fv-ml1); push Booth `ce095c1`; ESH Homepage HTTPS name; nh3-dev reboot (kernel 6.1.0-53). ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope. - **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`. The LiteLLM compose.yaml on ana-docker differs from `stacks/litellm/compose.yaml` (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%). - **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in. - **Older open items, one line each:** esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (`stacks/blender/README.md`). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran `51168a2e` on 09-27). nh3-dev's root disk is at ~66% since heid's runaway sweep was killed 10-08 (it flapped on the Beszel 85% line before). ESH has a single outside route (esh-scale on esh-pve), noted and untracked. @@ -186,17 +178,17 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache- ## Recent decisions -- `[2026-10-11]` **Prime 0721: stage fv-ml1 evidence capture; run U13; Mac keys on vh stay full access; delete stale snapshots.** Done: fv-ml1 ZFSBootMenu cmdline `quiet console=ttyS1,115200n8 console=tty0` (SOL = ttyS1, marker-tested; next boot) + nh3-dev user unit `fv-ml1-sol-logger` holding SOL into the journal (marker verified); ERST pstore was already registered, so the 10-11 hang was not a kernel panic. esh-nas-pve `pre-upgrade-20260818` and `subvol-105@pre-vaapi-20260911` destroyed (2.5 GB). U13 E1 (highseat-dev 0733) and E2 steps 1–6 done 0734–0816: reload A `a439431771dd013e`, kernel env (Matrix token + relay credential, kernel restarted), terminal on nh3-dev (Prime approved 0.0.3 at 0813 after the first card's 15-min TTL lapsed), reload B `44be73a252d85fbb` (13 seats adopted), miranda + agent up, Hermes swapped to `svos` (gateway restart 0814:58), bridge on nh3-dev (Prime approved 0.0.2 ~0816; miranda handle push). Open: Prime raises + approves the bridge's 3 runtime grants (owner-only, svos-dev's console snippet), then acceptance (bus test; URGENT page test on his word). ⚠ A local-transcript slip printed the High Seat's Bifrost dispatch key (leaving use with U13). -- `[2026-10-11]` **fv-ml1 hard hang ~0005, self-reset ~0008, back by 0015 (all seats).** Journal ends mid-normal-operation (no shutdown), pstore empty, SOL shows nothing (no serial kernel console), BMC restart cause `unknown`, SEL only shows the 00:08 boot events (⚠ the BMC clock is 7 h behind, so ipmitool prints them as 17:08 the day before). Idle at the time (Beszel 0004: CPU 0.15%, no Scriberr job). The sp5100 TCO watchdog exists (60 s). infra-ops sent an ACPI soft-off at 00:12:14 believing it still hung; it landed in early boot before logind watched the button, so no effect. Open (Prime): capture evidence next time (serial console on the kernel cmdline for SOL, and/or kdump); the pre-patch snapshot is HELD. -- `[2026-10-10]` **Prime's U13 rulings (~1705): seat roles `dev` + `ops`; artifact host label KEPT `nh3-svos` (old inbox links keep working; infra-ops copies the WHOLE `/var/lib/svos/apps/terminal` (E2 step 2) and `/var/lib/svos/apps/postoffice` (step 6) to nh3-dev at the same paths, owner lkraven, modes kept, each after its nh3-svos unit stops and before the nh3-dev one starts: the artifacts.sqlite index and the bridge's journal must travel with the files); Matrix relay persona `matrix-relay`.** A two-label hybrid is impossible: `svos_connector.artifacts.relative()` refuses any host but the app's one `artifact_host`. Relay secret `nh3-svos/matrix-relay-enrollment-secret` minted. svos-dev's payload: svoperatingsystem `docs/operations/u13-edit-payload.md` @ `7d7ed1f` (enrollment sha256 placeholders: fill at E2 from `secret get | sha256` — never commit them). -- `[2026-10-10]` **U13 phase B built (Prime 1639: "implement as asked")**: B1 kernel 0.0.9 + schema 12 (vzdump `vm/111/2026-10-10T23:45:41Z` + offline `pre-afab123` copy first), B2 `svos-kernel.phasefinal.com` → `--app-listen 127.0.0.1:8410` (allowlist nh3-dev; `/auth/kernel-key` ADDED to svos-dev's path list because every connector client fetches it), B3 seat-host origin + bridge env on nh3-dev, B4 app-high-seat origin; then svos-dev's dc241f8 amendment: full method-pinned path list, Miranda's app + agent env and `svos-miranda` origin on nh3-dev, three secrets minted to the vault, app-high-seat declared. Found + fixed: nh3-dev's fleet Caddy shared admin :2019 with ttyd-caddy (SO_REUSEPORT), now :2021. Kernel answers 401 `no principal` through the origin; other sources 403. -- `[2026-10-10]` **esh-nas-pve `nvme/ROOT@pre-pve9` destroyed 0750 (607M)** after a week clean on PVE 9.2.21 / 7.0.14-20 (up 6 d, no err-level journal in 7 d, pools healthy, all guests up; only `openipmi.service` failed, the known item). Left: `nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z` (now 1.43G, a pre-PVE-9 rollback point that is useless as one) and `nvme/subvol-105-disk-0@pre-vaapi-20260911`. -- `[2026-10-09]` **vh gets every albok read group (Prime 2244: "vh gets albok-everything")**: albok-read, albok-personal and albok-feedback (the restricted wing's default-deny ends for vh only), plus his four Macs' keys on vh (`playbooks/nh3-albok-vh-viewer.yaml`). No net new root path: those Macs already reach root there via lkraven@nh3-dev and the infra-ops key. Later restricted groups (1513+) are NOT included. +- `[2026-10-11]` **U13 cutover executed on Prime's go (0721) + his four rulings** — E1 0733, E2 steps 1–6 0734–0816, URGENT acceptance test 1350; fv-ml1 evidence capture, stale snapshots, Mac keys, no key rotation. → `persistent-memory.d/2026-10-11-u13-cutover-executed.md` +- `[2026-10-11]` **fv-ml1 hard hang ~0005, self-reset, no panic record; pre-patch snapshot HELD** — and infra-ops' soft-off that landed mid-boot. → `persistent-memory.d/2026-10-11-fv-ml1-hard-hang.md` +- `[2026-10-10]` **Prime's U13 rulings: roles dev+ops, artifact label KEPT nh3-svos (whole app dirs copied at E2), relay `matrix-relay`.** → `persistent-memory.d/2026-10-10-u13-rulings-p1-p3.md` +- `[2026-10-10]` **U13 phase B built (B1–B6)** — kernel 0.0.9/schema 12, `svos-kernel.phasefinal.com` (nh3-dev only, method-pinned paths), nh3-dev origins + apps env, enrollment secrets. → `persistent-memory.d/2026-10-10-u13-phase-b-built.md` +- `[2026-10-10]` **esh-nas-pve `nvme/ROOT@pre-pve9` destroyed after a clean week** (the two older snapshots followed on 10-11, Prime). → `persistent-memory.d/2026-10-10-esh-nas-pve-pre-pve9-destroyed.md` +- `[2026-10-09]` **vh gets every albok read group + four Mac keys (Prime: "albok-everything")** — no net new root path. → `persistent-memory.d/2026-10-09-vh-albok-everything.md` - `[2026-10-09]` **nh3-dev: orphaned svos pytest killed 2242** (14.4 GB RSS, +2.3 GB/h, swap 100%, Beszel 90% alert; SIGTERM ignored, SIGKILL; used 25.4 → 11.3 GB). highseat-dev's retry-loop test with an instant fake sleep; they added pytest-timeout 300 s and a 50-attempt bound. -- `[2026-10-09]` **Post-clear checks closed (2015–2025 PT):** albok `.browse/` sshfs pathname-withdrawal test PASSED (replica on nh3-albok, Linux sshfs 3.7.3 client in a container; fresh open ENOENT 5/5 default and 5/5 hour-long caches; null + positive controls ok; failed rename = path served; reported to albok-dev) → `scripts/oneoff/albok-browse-withdrawal/README.md`. fv-ml1 plateau (Beszel 120m, 42 h after the 0034 10-08 restart): GPU 0 flat at 5,606 MiB free (~3.9 GB at parakeet's cap), GPU 1 median 9,459 free (~3.3 GB at intern-decision's 32k peak), no creep; GPU 2 643 MiB free = flash-next util 0.96, single tenant, by design. nh3-nas `/volume1` 71% (79% on 10-07). The "sfsrv-ana pre-pbs4" item was a mislabel: the pre-pbs4 snapshots were pfi-pve VM 100 and nh3-pve VM 105, dropped by infra-hermes 10-05 and read gone 10-09. +- `[2026-10-09]` **albok `.browse/` sshfs withdrawal test PASSED (Linux) + post-clear checks closed** (fv-ml1 plateau, nh3-nas 71%, pre-pbs4 mislabel). → `persistent-memory.d/2026-10-09-post-clear-checks-albok-withdrawal-pass.md` - `[2026-10-09]` **albok `library` building (fiction, nonfiction) live for unit 4.5; Nemi + albok-dev re-minted with library grants (token grants are immutable), old tokens revoked 1555 by albok-dev.** → `stacks/albok-service/README.md` - `[2026-10-09]` **Homepage: The High Seat's description no longer says "SVOS" (Prime: author it yourself); SVOS's now says "Internal network + mesh".** -- `[2026-10-09]` **APPROVED by Prime (10-09), BUILD DEFERRED until after his snapshot/clear: SVOS app-traffic ingress for U13** (apps on nh3-dev reach the nh3-svos kernel). Shape agreed with svos-dev: (1) kernel `--app-listen 127.0.0.1:8410`, which never trusts loopback (svos-dev builds it in U9a); (2) a Caddy origin `svos-kernel.phasefinal.com` on 10.100.50.43, no Basic auth, bearer passed, app paths only, allowlisted to nh3-dev; (3) kernel→bridge over plain LAN HTTP; kernel→seat host (terminal) via nh3-dev's wildcard Caddy `svos-seathost.nh3.phasefinal.com` (TLS, because the leg carries secrets). nh3-dev is NOT on the mesh; the contracts now say "fleet network". `--app-listen` is ON MAIN (`68c9933`, 10-09). It ships in kernel **0.0.9** with schema **11 → 12** (migration 0012 rebuilds dispatches + summons, so the 'schema 11' acceptance line changes and a pre-refresh backup is mandatory). At the cutover the bridge gets three owner-approved runtime grants (app-postoffice → `app:app-miranda` write, `app:app-postoffice` write, `ns:pages` write, all autonomous) and installs as `svos-postoffice[push]` with `althing-core` from `~/development/althing` in apps-env. **Release is cut: svoperatingsystem `afab123` (tag v0.0.9)**: kernel 0.0.9, svos-postoffice 0.0.2, svos-terminal 0.0.3, svos-connector 0.0.7, svos-agent 0.0.3, svos-miranda 0.0.2 (U9a + U5b). The SVOS side of the cutover is written up in `docs/operations/u13-svos-edit.md` (app listener on nh3-svos, terminal app moved to nh3-dev with `[seat_host]`, bridge on nh3-dev with the converse mapping). Gotchas: the terminal app serves WITHOUT its seat host if the socket or launch dir fails (one stderr line); `serve` refuses `--app-listen` on the primary port. Build only when the edit is scheduled (highseat-dev owns the schedule) and NOT before the snapshot; the runbook refresh is ours at deploy. Tracking: svos-dev thread `01M4GPXSBKQHCQ30FVPH4R3N7G`. +- `[2026-10-09]` **SVOS app-traffic ingress for U13 approved (shape agreed with svos-dev)** — EXECUTED 10-10 (phase B) and 10-11 (the edit). → `persistent-memory.d/2026-10-09-u13-ingress-approved.md` - `[2026-10-09]` **nh3-svos → kernel 0.0.8 (`b588510`, a bounded SQLite reader pool): the 10-08 22:55 EMFILE outage is fixed at the source and the LimitNOFILE stopgap is removed.** A 4-min browser soak held `svos.db` handles flat at 9–12 (0.0.7 grew +22 in 2 min). Root cause: one cached connection per short-lived anyio worker thread. → `servers/nh3-svos/README.md` - `[2026-10-08]` **nh3-svos refreshed to ac42ace (interface UI 0.0.7 polish + kernel 0.0.7 with the `Type=notify` unit as a pair); thaw race FIXED**: the pre-refresh drill (old kernel) made two apps fail at thaw, the post-refresh drill made 0. Previous release is now a737a06; 6c6abbe is deletable. → `servers/nh3-svos/README.md` - `[2026-10-08]` **SVOS voice ON (Prime, via svos-dev): app-voice on 127.0.0.1:8402, apps up 4 + terminal degraded; Connect → "Voice connected", Enable lists the gateway's 30 voices (Chromium over the LAN door).** `/voice/*` is routed BEFORE Basic auth, because the browser sends `Authorization: Bearer ` there (`web/src/lib/voice.ts`) and Basic auth would reject it. All four app credentials are now vaulted as `nh3-svos/svos--credential`. → `servers/nh3-svos/README.md` @@ -286,14 +278,10 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache- - `[2026-09-26]` **augaman: infra-ops owes the gallery backup + first deploy — DEFERRED until augaman-dev reaches deploy.** → `persistent-memory.d/2026-09-26-augaman-infra-ops-owes-the-gallery-backup-first-deploy.md` — `[2026-09-27]` **DONE:** deployed (v0.1.3 on esh-ml1), backup wired, identity-level restore verified (`d8f59a1`). - `[2026-09-26]` **`zellij-fleet@.service` installed, NOT enabled** (for svos-dev's seat_up; owns the fleet zellij server in its own cgroup). Enabling `@Claude` at boot is Prime's call when seat_up ships. Tracked: `0ad7799`, `services/zellij-fleet/README.md`. - `[2026-09-26]` **btusb blacklist on nh3-pve + esh-pve — proposed, awaiting Prime (low priority).** Every 6.8.12-4x boot oopses in btmtk (benign). Also noted for the next NH3 visit: nh3-pve runs ~20 °C warmer than esh-pve (workload-confounded; check airflow). Tracked: `servers/nh3-pve/README.md`. -- `[2026-09-26]` **GPU-LXC Temperature alerts watch the GPU, not the host CPU** (`SENSORS=-coretemp_*,acpitz`); hypervisor CPU alerts at >95 °C on nh3-pve/esh-pve. The first nh3-ml1 alert was nh3-pve's CPU during vzdump. `6d901c0`. -- `[2026-09-26]` **Abliterated LFM2.5-VL-3B parallel seat (:8032) passed brokkr's 6-image eval and stays** (direct only; stock seat kept for SFW A/B). `9cc3824`, `944bb36`. - `[2026-09-25]` **AMT follow-ups PARKED until the two MS-03s arrive and esh-pve's MS-01 is on AMT (Prime 2326).** — Parked: MeshCommander test; dummy HDMI plug, then NanoKVM → gx10; an OOB path not through nh3-scale… → `persistent-memory.d/2026-09-25-amt-follow-ups-parked-until-the-two-ms-03s-arrive-and-esh.md` -- `[2026-09-26]` **VibeVoice ASR → Q8_0 (Prime).** WER on the 4 bundled LibriSpeech clips 3/69 → 2/69 (only the I'm/I am artifact left), ×3 identical; RTF 0.09–0.17; +1.1 GB VRAM (nh3-ml1 ~11.3/16 GB). Q4_K file removed. - `[2026-09-26]` **esh-matter LIVE: a Matter server (matter.js 1.4.0) on CT 111 @ 10.0.90.20, VLAN 90 only** — , for ha-dev (operator-approved, relayed). → `persistent-memory.d/2026-09-26-esh-matter-live-a-matter-server-matter-js-1-4-0-on-ct-111.md` - `[2026-09-26]` **Embed/rerank LOAD-SHARED across esh-ml1 + nh3-ml1 (Prime).** — Second deployments were added for qwen3-embedding and reranker (config) and for reranker-a3-bge-v2-m3 (DB… → `persistent-memory.d/2026-09-26-embed-rerank-load-shared-across-esh-ml1-nh3-ml1-prime.md` - `[2026-09-26]` **Two dataset-foundry utility seats LIVE on nh3-ml1 (brokkr; operator approval relayed):** — LFM2.5-VL-3B on llama.cpp `:8030` (gateway `lfm25-vl-3b`, LiteLLM restarted 36 s at 0039) and… → `persistent-memory.d/2026-09-26-two-dataset-foundry-utility-seats-live-on-nh3-ml1-brokkr.md` -- `[2026-09-26]` **Coder seat STAYS on fv-ml1 (Prime).** — The nh3-ml1 copy gave the same quality (teacher-forced true-code logprob diff +0.008 ± 0.019) but ran ~5×… → `persistent-memory.d/2026-09-26-coder-seat-stays-on-fv-ml1-prime.md` - `[2026-09-25]` **nh3-pve OOB: HOLD blind until the next NH3 visit (Prime via Miranda, 1333).** Target: NanoKVM → pfi-gx10, and the MS-01 on its own vPro. The IGFX BIOS fix rides the same visit, since AMT KVM captures only the iGPU. Tracked: `servers/nh3-pve/README.md` (`2b49be4`, `cdd7605`). - `[2026-09-25]` **Direct ESH→esh-ml1 consumer path — PARKED** (no ESH-side callers in 7 days; consumers go through the gateway). Tracked at `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md`. - `[2026-09-24]` **Worldtree `admin.memory.forget` stays OFF on demo/personal** until an instance needs it — a destructive erase; enabling it is a per-change operator yes via deploy-wt-config. @@ -310,7 +298,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache- - `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md` -_236 older entries archived to archival-memory.md._ +_240 older entries archived to archival-memory.md._ ## Tried and abandoned @@ -339,7 +327,5 @@ _236 older entries archived to archival-memory.md._ - `[2026-09-27]` **Translating a CUDA OOM by raising inside `except` (or `from exc`)** — the chained exception's traceback pins the failed call's frames, and their GPU tensors (11.9 GiB after the 503). Raise after the block, unchained, `gc.collect()` before `empty_cache()`. - `[2026-09-27]` **`deploy-stack.sh --yes` / `dns-sync.py` fed a blind `y`** — the harness refuses a blind apply. Review with `echo n |` first, then apply with `echo y |`. - `[2026-09-27]` **Relying on the build cache surviving on esh-ml1** → `persistent-memory.d/2026-09-27-relying-on-the-build-cache-surviving-on-esh-ml1.md` -- `[2026-09-26]` **Greedy exact-match as parity for a small generative seat (coder)** — same-host repeats matched only 30–38% (run B hit the prefix cache, so a different numeric path; near-tie… → `persistent-memory.d/2026-09-26-greedy-exact-match-as-parity-for-a-small-generative-seat.md` -- `[2026-09-26]` **zsh traps in ad-hoc test loops:** `set -- $a` does NOT word-split, so alert POSTs went out with empty fields; and `local path=…` inside a function CLOBBERS `$PATH` (zsh's tied array), so the failover test ran nothing while TEI sat stopped. Use other names and `${=var}`. -_126 older entries archived to archival-memory.md._ +_128 older entries archived to archival-memory.md._