vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: snapshot — 2026-10-09 SVOS LAN/voice/ac42ace/kernel 0.0.8 (fd outage fixed), U13 ingress approved+deferred, albok library, gate watch; 11 entries archived

Browse Source
This commit is contained in:
vh
2026-10-09 19:56:00 -07:00
parent d0358b562d
commit d89864b5c4
8 changed files with 299 additions and 275 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+14 -21
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-10-08 ~0740 PT (patch night: fv-ml1 kernel 6.12.111 + NVIDIA 580.178.04 + ZFS 2.3.9, Docker 29.8.2 on nh3-docker + esh-docker-vm; nh3-svos on SVOS a737a06 with knowledge/terminal/post office live; albok-service 0.1.13 → 0.1.14; gateway glm-5.3 + gen-frontier on 5.3; heid's runaway 76 GB sweep killed; Hermes launchers repaired.)_
_Last updated: 2026-10-09 ~1610 PT (nh3-svos: SVOS served on the LAN too, voice live, refreshed to ac42ace then kernel 0.0.8 — thaw race and the SQLite-connection fd outage both fixed; SVOS app-traffic ingress for U13 APPROVED, build deferred; fv-ml1 off the mesh by design; albok library building + token swap; Worldtree gate 10-09 filing FAIL under watch; Homepage descriptions.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -113,27 +113,29 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight
_As of 2026-10-08 ~0740 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched._
_As of 2026-10-09 ~1610 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched._
### 2026-10-08: live now (as of 2026-10-08 ~0740 PT)
### 2026-10-09: live now (as of 2026-10-09 ~1610 PT)
- **Nothing is mid-action.** No claims held (`scripts/ops-log claims`). eshpfi is pushed through this snapshot's commit. ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- <paths>` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose (auto-regenerated).
- **Approved, NOT built (do not start before Prime says so after this snapshot):** SVOS app-traffic ingress for U13. Details, release (`afab123`, v0.0.9, schema 12) and gotchas are in the 2026-10-09 Recent-decisions entry; svos-dev's edit doc is `docs/operations/u13-svos-edit.md`; highseat-dev owns the schedule.
- **Owed by me (no input needed):**
- albok `.browse/` exposure test before ANY share: pick a surface (sshfs `follow_symlinks` is the lead), rename `.browse/` on nh3-albok, and a fresh open through the client's cached listing must FAIL; also record the failed-rename case. Report on albok-dev's thread; the operator's user needs the restricted wings' groups on the share host. → `stacks/albok-service/README.md`
- fv-ml1 GPU 0/1 plateau read (Beszel `stats.g.<n>.mu` × 1.024), now ~10-10, since every seat restarted at 0034 10-08.
- fv-ml1 GPU 0/1 plateau read (Beszel `stats.g.<n>.mu` × 1.024) ~10-10; every seat restarted at 0034 10-08.
- Destroy `zroot/ROOT/debian@pre-patch-20261008` on fv-ml1 ~10-15 if nothing regressed.
- sfsrv-ana `pre-pbs4` snapshot delete is unverified (infra-ops has no key there; ask infra-hermes).
- SVOS thaw fix needs repeats (one clean drill is evidence, not proof): after the nightly 21:30 nh3-pve-2 vzdumps, grep nh3-svos `journalctl -u 'svos-*' --since 21:29` for `startup or service failed`; if one reappears, send svos-dev one line.
- Re-read nh3-nas usage after the 10-08 GC (79% at 10-07 2350; the mirror prune ran OK).
- **Verified 2026-10-07/08 (closed):** the 10-07 0800 backup check read all fresh with no DESTROYED/DEGRADED lines; the pbs-nh3 mirror prune ran OK at 00:00 on 10-07 and 10-08; `pre-pve9` snapshots are gone on pfi-pve and esh-pve.
- Re-read nh3-nas usage after the 10-08/09 GCs (79% at 10-07 2350).
- nh3-svos housekeeping: releases `6c6abbe` and `a737a06` are no longer referenced by the front door and can be deleted (`ac42ace` holds the same static as the current `b588510`).
- **Watching: Worldtree memory gate.** 10-09 FAILED on filing/user (9, 11, 8 of 12; median 0.75 < 0.80), the first fail that is not #416. Bootstrap from 27 prior runs puts P(median ≤ 9) ≈ 0.06 per batch, and the misses are the habitual fixtures, so it is not called a regression yet. But per-run means drifted 10.6 → 10.0 → 9.7 → 9.3 since the 10-08 fv-ml1 patch (writer + judge = flash-next on GPU 2). If the 10-10 batch also fails filing, infra-hermes frames it as a possible regression after that patch and sends it to me; then investigate flash-next. ⚠ worldtree-dev has not read mail in 6+ days, so the reports dead-end there (Prime's call, below).
- **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens. jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it, as gen-large did (preemptions; decode down to 2–16 tok/s). jekyll's context grows ~1k/call toward the 262k ceiling. It runs on the shared all-agents-local key; a key of its own would make its traffic attributable.
- **Waiting on Prime:**
- Who sees Worldtree gate failures while worldtree-dev is not reading mail (Prime, Miranda, or keep waiting).
- Rotate the fv-ml1 BMC password? It was printed into a local session transcript on 10-08 (never left nh3-dev); infra-ops leans no.
- albok's 200-object load check, 2.01× against a 2× bar (albok-dev: the ruling is the operator's).
- The cleanup bundle offered 10-05: close stale parks 49/46/40; refresh the fv-ml1 README stack tables plus the serial-keyed NVMe table and `zpool replace` line (the 2084-media-error drive is now `/dev/nvme6`, not nvme7); wire ZED pool alerts into althing-alert-bridge; prune the exited one-shot containers on fv-ml1 (keep the `parakeet` rollback).
- Push Booth commit `ce095c1`. A trusted HTTPS name for ESH Homepage. nh3-dev reboot (kernel 6.1.0-53 installed).
- ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope.
- **Scheduled / unchecked:** esh-nas-pve `nvme/ROOT@pre-pve9` destroy ~10-10 (ours). 10-11 0540: first weekly sudo-io archiver run (check it). 10-11 0900: infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet. Next scheduled tank scrub on fv-ml1: Sun 10-11 0024 (the September slot was missed); check it ran.
- **Worldtree memory gate:** every FAIL since 09-30 (0930T0809, 1004, 1007) is ONE fixture, `u-negative-ephemeral` FILED in one run — the #416 ephemeral residual, ~3 of 42 fixture-runs since bbcc1788 (~1 batch in 5 FAILs on it), NOT a recall regression; recall (filing/user median 0.83–0.92, opener/cross_session/inline_core 100%) has held throughout. Negative is a structural class, so one run's leak fails the aggregate (`_run_structural_failed`); filing alone is judged by median. 10-07: infra-hermes told to classify FAILs by failing row (msg to its skill's daily-duty section); worldtree-dev told #416 is not closed (pull-only, unread 4 days) — reopen is theirs/Prime's.
- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`. The LiteLLM compose.yaml on ana-docker differs from `stacks/litellm/compose.yaml` (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%).
- **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in.
- **Older open items, one line each:** esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (`stacks/blender/README.md`). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran `51168a2e` on 09-27). nh3-dev's root disk is at ~66% since heid's runaway sweep was killed 10-08 (it flapped on the Beszel 85% line before). ESH has a single outside route (esh-scale on esh-pve), noted and untracked.
@@ -187,6 +189,8 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
## Recent decisions
- `[2026-10-09]` **albok `library` building (fiction, nonfiction) live for unit 4.5; Nemi + albok-dev re-minted with library grants (token grants are immutable), old tokens revoked 1555 by albok-dev.** → `stacks/albok-service/README.md`
- `[2026-10-09]` **Homepage: The High Seat's description no longer says "SVOS" (Prime: author it yourself); SVOS's now says "Internal network + mesh".**
- `[2026-10-09]` **APPROVED by Prime (10-09), BUILD DEFERRED until after his snapshot/clear: SVOS app-traffic ingress for U13** (apps on nh3-dev reach the nh3-svos kernel). Shape agreed with svos-dev: (1) kernel `--app-listen 127.0.0.1:8410`, which never trusts loopback (svos-dev builds it in U9a); (2) a Caddy origin `svos-kernel.phasefinal.com` on 10.100.50.43, no Basic auth, bearer passed, app paths only, allowlisted to nh3-dev; (3) kernel→bridge over plain LAN HTTP; kernel→seat host (terminal) via nh3-dev's wildcard Caddy `svos-seathost.nh3.phasefinal.com` (TLS, because the leg carries secrets). nh3-dev is NOT on the mesh; the contracts now say "fleet network". `--app-listen` is ON MAIN (`68c9933`, 10-09). It ships in kernel **0.0.9** with schema **11 → 12** (migration 0012 rebuilds dispatches + summons, so the 'schema 11' acceptance line changes and a pre-refresh backup is mandatory). At the cutover the bridge gets three owner-approved runtime grants (app-postoffice → `app:app-miranda` write, `app:app-postoffice` write, `ns:pages` write, all autonomous) and installs as `svos-postoffice[push]` with `althing-core` from `~/development/althing` in apps-env. **Release is cut: svoperatingsystem `afab123` (tag v0.0.9)**: kernel 0.0.9, svos-postoffice 0.0.2, svos-terminal 0.0.3, svos-connector 0.0.7, svos-agent 0.0.3, svos-miranda 0.0.2 (U9a + U5b). The SVOS side of the cutover is written up in `docs/operations/u13-svos-edit.md` (app listener on nh3-svos, terminal app moved to nh3-dev with `[seat_host]`, bridge on nh3-dev with the converse mapping). Gotchas: the terminal app serves WITHOUT its seat host if the socket or launch dir fails (one stderr line); `serve` refuses `--app-listen` on the primary port. Build only when the edit is scheduled (highseat-dev owns the schedule) and NOT before the snapshot; the runbook refresh is ours at deploy. Tracking: svos-dev thread `01M4GPXSBKQHCQ30FVPH4R3N7G`.
- `[2026-10-09]` **nh3-svos → kernel 0.0.8 (`b588510`, a bounded SQLite reader pool): the 10-08 22:55 EMFILE outage is fixed at the source and the LimitNOFILE stopgap is removed.** A 4-min browser soak held `svos.db` handles flat at 9–12 (0.0.7 grew +22 in 2 min). Root cause: one cached connection per short-lived anyio worker thread. → `servers/nh3-svos/README.md`
- `[2026-10-08]` **nh3-svos refreshed to ac42ace (interface UI 0.0.7 polish + kernel 0.0.7 with the `Type=notify` unit as a pair); thaw race FIXED**: the pre-refresh drill (old kernel) made two apps fail at thaw, the post-refresh drill made 0. Previous release is now a737a06; 6c6abbe is deletable. → `servers/nh3-svos/README.md`
@@ -280,21 +284,13 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
- `[2026-09-26]` **GPU-LXC Temperature alerts watch the GPU, not the host CPU** (`SENSORS=-coretemp_*,acpitz`); hypervisor CPU alerts at >95 °C on nh3-pve/esh-pve. The first nh3-ml1 alert was nh3-pve's CPU during vzdump. `6d901c0`.
- `[2026-09-26]` **Abliterated LFM2.5-VL-3B parallel seat (:8032) passed brokkr's 6-image eval and stays** (direct only; stock seat kept for SFW A/B). `9cc3824`, `944bb36`.
- `[2026-09-25]` **AMT follow-ups PARKED until the two MS-03s arrive and esh-pve's MS-01 is on AMT (Prime 2326).** — Parked: MeshCommander test; dummy HDMI plug, then NanoKVM → gx10; an OOB path not through nh3-scale… → `persistent-memory.d/2026-09-25-amt-follow-ups-parked-until-the-two-ms-03s-arrive-and-esh.md`
- `[2026-09-25]` **nh3-pve AMT LIVE: static `10.100.250.61` on nh3-mgmt (UDM port 6), KVM on, Opt-in None** — (Prime). → `persistent-memory.d/2026-09-25-nh3-pve-amt-live-static-10-100-250-61-on-nh3-mgmt-udm-port.md`
- `[2026-09-25]` **pfi-gx10 AC-restore VALIDATED by Prime's AC pull; Homepage stays on esh-docker-vm (Prime) after the mmap_lock wedge reboot.** `servers/pfi-gx10/README.md`, `servers/esh-docker-vm/README.md`.
- `[2026-09-26]` **VibeVoice ASR → Q8_0 (Prime).** WER on the 4 bundled LibriSpeech clips 3/69 → 2/69 (only the I'm/I am artifact left), ×3 identical; RTF 0.09–0.17; +1.1 GB VRAM (nh3-ml1 ~11.3/16 GB). Q4_K file removed.
- `[2026-09-26]` **esh-matter LIVE: a Matter server (matter.js 1.4.0) on CT 111 @ 10.0.90.20, VLAN 90 only** — , for ha-dev (operator-approved, relayed). → `persistent-memory.d/2026-09-26-esh-matter-live-a-matter-server-matter-js-1-4-0-on-ct-111.md`
- `[2026-09-26]` **Embed/rerank LOAD-SHARED across esh-ml1 + nh3-ml1 (Prime).** — Second deployments were added for qwen3-embedding and reranker (config) and for reranker-a3-bge-v2-m3 (DB… → `persistent-memory.d/2026-09-26-embed-rerank-load-shared-across-esh-ml1-nh3-ml1-prime.md`
- `[2026-09-26]` **Two dataset-foundry utility seats LIVE on nh3-ml1 (brokkr; operator approval relayed):** — LFM2.5-VL-3B on llama.cpp `:8030` (gateway `lfm25-vl-3b`, LiteLLM restarted 36 s at 0039) and… → `persistent-memory.d/2026-09-26-two-dataset-foundry-utility-seats-live-on-nh3-ml1-brokkr.md`
- `[2026-09-26]` **Coder seat STAYS on fv-ml1 (Prime).** — The nh3-ml1 copy gave the same quality (teacher-forced true-code logprob diff +0.008 ± 0.019) but ran ~5×… → `persistent-memory.d/2026-09-26-coder-seat-stays-on-fv-ml1-prime.md`
- `[2026-09-25]` **nh3-ml1 LIVE after the NH3 visit (SB off, IGFX restored): driver + CT 109 + TEI, parity vs esh-ml1 indistinguishable (cos min 0.999993 = own floor; overlap@10 1.000 vs MRL-256 control 0.684), same speed; monitoring/DNS wired. Autonomous: lxc-pve 6.0.0-2 upgrade on nh3-pve (Docker-in-LXC fix #7006). Gateway routing + AMT follow-up are Prime's.** → `persistent-memory.d/2026-09-25-nh3-ml1-live.md`
- `[2026-09-25]` **nh3-ml1 = CT 109 @ 10.100.50.80, second TEI embed/rerank backend on nh3-pve's RTX 2000E; GPU playbooks made host-generic — BUILD DEFERRED: blocked on nh3-pve Secure Boot until the NH3 visit; SB handling and gateway routing are Prime's calls.** Tracked: `7ddd116`, Current state. → `persistent-memory.d/2026-09-25-nh3-ml1-standup.md`
- `[2026-09-25]` **nh3-pve OOB: HOLD blind until the next NH3 visit (Prime via Miranda, 1333).** Target: NanoKVM → pfi-gx10, and the MS-01 on its own vPro. The IGFX BIOS fix rides the same visit, since AMT KVM captures only the iGPU. Tracked: `servers/nh3-pve/README.md` (`2b49be4`, `cdd7605`).
- `[2026-09-25]` **MS-01 foot-gun: a GPU in the PCIe slot renames every NIC** — (the slot's root port takes bus 01, so the X710 goes `enp2s0f0np0`→`enp3s0f0np0`). → `persistent-memory.d/2026-09-25-ms-01-foot-gun-a-gpu-in-the-pcie-slot-renames-every-nic.md`
- `[2026-09-25]` **Direct ESH→esh-ml1 consumer path — PARKED** (no ESH-side callers in 7 days; consumers go through the gateway). Tracked at `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md`.
- `[2026-09-25]` Created empty private repo `corviduo/norn` (git@gitea.phasefinal.com:corviduo/norn.git) as claude-bot, at brokkr-smithy-dev's relay of the operator's Norn ruling; the `norn-dev` handle is the operator's to declare.
- `[2026-09-25]` **Reward seat audited (Skywork-Reward-V2-Llama-3.1-8B still #1 of 188 on RewardBench 2; our AWQ ≈ bf16 within noise; double BOS costs ~2.7 pts) and moved fv-ml1 → esh-ml1; esh-ml1 monitoring wired (Beszel+GPU, Kuma, Homepage, Dozzle); Dozzle hub's stale agent IPs fixed; nh3-dev Beszel agent revived.** → `persistent-memory.d/2026-09-25-reward-move-and-esh-ml1-monitoring.md`
- `[2026-09-25]` **TEI adopted as the fleet embed/rerank engine (Prime); esh-ml1 is the sole gateway backend for `qwen3-embedding` + `reranker`; fv-ml1's vLLM embed/rerank seats retired (~6.1 GB freed); parakeet stays on fv-ml1.** → `persistent-memory.d/2026-09-25-tei-fleet-embed-rerank.md`
- `[2026-09-24]` **Worldtree `admin.memory.forget` stays OFF on demo/personal** until an instance needs it — a destructive erase; enabling it is a per-change operator yes via deploy-wt-config.
@@ -309,7 +305,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md`
_228 older entries archived to archival-memory.md._
_236 older entries archived to archival-memory.md._
## Tried and abandoned
@@ -340,8 +336,5 @@ _228 older entries archived to archival-memory.md._
- `[2026-09-27]` **Relying on the build cache surviving on esh-ml1** → `persistent-memory.d/2026-09-27-relying-on-the-build-cache-surviving-on-esh-ml1.md`
- `[2026-09-26]` **Greedy exact-match as parity for a small generative seat (coder)** — same-host repeats matched only 30–38% (run B hit the prefix cache, so a different numeric path; near-tie… → `persistent-memory.d/2026-09-26-greedy-exact-match-as-parity-for-a-small-generative-seat.md`
- `[2026-09-26]` **zsh traps in ad-hoc test loops:** `set -- $a` does NOT word-split, so alert POSTs went out with empty fields; and `local path=…` inside a function CLOBBERS `$PATH` (zsh's tied array), so the failover test ran nothing while TEI sat stopped. Use other names and `${=var}`.
- `[2026-09-25]` **Concluding "not on the UDM" from a port table read 15 s after link-up** — UniFi polls (~60 s); AMT was on port 6 as Prime said. Wait a poll, and prefer a which-port-lists-the-MAC check. auto-memory `feedback_polled_stats_lag_the_event`.
- `[2026-09-25]` **The esh-pve NVIDIA DKMS recipe on nh3-pve** — it built, then the kernel refused to load it: nh3-pve has SECURE BOOT ON (esh-pve, the same MS-01, has it off). The installer rolled back. The playbook now pre-flights `mokutil --sb-state` (`7ddd116`).
- `[2026-09-25]` **elway against an unpinned host-key name** — every `when:` hit ssh rc 255, so all 9 steps showed SKIPPED, not failed. Only the verify phase went red. Pin the name (the IP key matched) or use the IP. Untracked elway defect: rc 255 in `when:` should fail.
_123 older entries archived to archival-memory.md._
_126 older entries archived to archival-memory.md._