memory: snapshot — SVOS v0.0.11, pewpewstudio, albok inbox, h2o decision seat; 31 old entries archived

This commit is contained in:
vh
2026-10-14 04:04:54 -07:00
parent d1192ca868
commit 9d599caa53
28 changed files with 679 additions and 396 deletions
+26 -52
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-10-13 ~2135 PT (decision seat: h2o production since 1807 on Prime's fix-forward ruling, after two rollbacks; earlier: U13 cutover + acceptance, fv-ml1 evidence capture, vh albok-everything.)_
_Last updated: 2026-10-14 ~0005 PT (snapshot for clear: SVOS v0.0.11 live, pewpewstudio deployed, albok inbox live, decision seat on h2o; earlier: the h2o swap saga, U13.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -113,19 +113,21 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight
_As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched._
_As of 2026-10-14 ~0005 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched._
### 2026-10-11: live now (as of 2026-10-11 ~1400 PT)
### 2026-10-14: live now (as of 2026-10-14 ~0005 PT)
- **Nothing is mid-action.** No claims held. eshpfi is pushed through `df7161b`; this snapshot's commit follows. ⚠ The working tree and index are shared with infra-hermes and subagents: commit with `git commit -- <paths>`. `graphify-out/GRAPH_REPORT.md` stays modified on purpose.
- **U13 cutover: executed 10-11 0721–0816, acceptance 1350.** Miranda now runs in SVOS: app-miranda + svos-agent on nh3-dev, the Hermes gateway on plugin `svos`, and the bridge app-postoffice on nh3-dev owning the `miranda` push route. The URGENT bus test (thread `01M4PBNX5PP0PGHDTY5BG7HG95`) ingested, paged per the connector receipt, and a reply came back on the thread from `svos`. → `persistent-memory.d/2026-10-11-u13-cutover-executed.md`
- **Acceptance closed 1354:** Miranda's reply arrived on the thread through the bridge's own send. ⚠ RE-CORRECTED 2026-10-12: card `card:05cf119e604649838e2a4c642c161e26` (Miranda's carded `send`) WAS approved by `prime` at 2026-10-11 20:54:25Z (1354:25 PT), per the kernel DB (`/var/lib/svos/state/svos.db`, read-only). The 1412 "NOT approved" correction was itself wrong, because it came from inference, not a read. Nothing to deny; 0 cards pending. **Answered by svos-dev 1400:** the thread text was Miranda answering bus mail with her own `send` (carded) instead of letting the bridge carry her turn's reply; the fix (a bus-tag clause in `svos_miranda.turns`) SHIPPED in v0.0.10 (adc862e), deployed 1410–1411 (nh3-svos kernel 0.0.10 + UI 0.0.8; nh3-dev miranda 0.0.4, `/opt/svos/current → releases/adc862e`); bus check passed 1412 with no card. Its manifest still says 0.0.3 (hardcoded), so it did not re-register. Then a static-only release `34476a7` (interface UI 0.0.9, tree strands follow speech) at 1712: `/opt/svos/current → releases/34476a7` on nh3-svos, no restart, previous handle = adc862e; then `ad255a2` (UI 0.0.10, stronger strands, Prime found 0.0.9 too subtle) at 1719, previous = 34476a7; then `627a9fb` (UI 0.0.11: the top bar shows `v0.0.<n> · <short commit>`, from `web/build-id.txt` via git-archive export-subst, so ALWAYS build from a clean `git archive`; a plain copy reads `dev`) at 1722, previous = ad255a2; `61459d2` (UI 0.0.12: voice chorus/flanger toggle, lighter header) at 1739, previous = 627a9fb; `8fa229d` (UI 0.0.13: strands span the tree, deeper voice character) at 1800, previous = 61459d2. `89dad7a` (UI 0.0.14: the pane lease race, a frame loading before the attach reply now gets its lease) on 10-13 at 0857, previous = 8fa229d. `7f95e54` (UI 0.0.15: the terminal leaves the app list's Open; a bare embedded terminal can only ever wait for a lease, so it is attached per seat) at 0906:37, previous = 89dad7a. Static-only builds: `scratchpad svos-static-release.sh <prev> <new> <ui>` pattern (rebuild it from the 61459d2 kit if the scratchpad is gone). Static-only releases: build the full release dir (carried artifacts + new `static/`), switch `current` without restarting, regenerate the inventory. ⚠ The kernel + apps still RUN from `releases/adc862e` (verified via `/proc/<pid>/maps`): do not delete it before they restart. Same on nh3-dev: svos-terminal, svos-postoffice and svos-agent still run from `releases/dc241f8` (svos-miranda from `adc862e`). The non-autonomous `miranda → app:app-postoffice write` grant is intended (her own sends); making it autonomous is Prime's call, surfaced by svos-dev. Speech leg unconfirmed (svos-dev asked Prime; n=1 vs the earlier 15 s `synthesis_failed`). Phase R (24 h soak, doctrine + broadcast texts, High Seat removal) is highseat-dev's (svos `docs/u13-cutover-runbook.md`).
- **Rollback material, keep until the soak ends:** nh3-svos `releases/b588510`, `instance.toml.pre-u13-{A,B}`, `/etc/svos/kernel.env.pre-u13` (absent: there was none); nh3-dev `~/.hermes/config.yaml.pre-u13` and `~/.hermes/plugins/svos_miranda`. After the soak: nh3-svos releases `6c6abbe`, `a737a06`, `b588510` and nh3-dev `/opt/svos/releases/afab123` (unreferenced) can go.
- **Nothing is mid-action.** No ops-log claims held. eshpfi `main` is pushed through `d1192ca` (infra-ops has STANDING push on main since 2026-10-13, CLAUDE.md § Working rules); this snapshot's commit follows and is pushed. ⚠ The working tree and index are shared with infra-hermes, albok-dev and subagents: commit with `git commit -- <paths>` and read `git log origin/main..main` before pushing. `graphify-out/GRAPH_REPORT.md` stays modified on purpose.
- **SVOS: v0.0.11 (452a592, U15 task seats slice 1) is live** on nh3-svos (schema 13, config_gen `5ade279d2fb2dd1c`, apps up 7) and nh3-dev (`/opt/svos/current → releases/452a592`; svos-terminal 0.0.4 runs from it). svos-dev accepted it. Task seats stay OFF until Prime names a runtime `task_argv` (e.g. the claude command), and `GET /tasks` as Prime is his acceptance check. Static-only interface releases since: UI 0.0.14 `89dad7a`, 0.0.15 `7f95e54` (the web build carried into 452a592). ⚠ Process→release map: nh3-svos kernel + local apps run `452a592`; on nh3-dev svos-miranda runs `adc862e`, svos-postoffice and svos-agent still run `dc241f8` (check `/proc/<pid>/maps` before deleting any release dir). Rollback material: nh3-svos `/var/lib/svos/backup/pre-452a592/` (schema-12 DB + config tar), `instance.toml.pre-452a592`, `instance.toml.pre-a171`; nh3-dev `terminal.toml.pre-452a592`. U13 history and the static-release chain: → `persistent-memory.d/2026-10-11-u13-cutover-executed.md`. Phase R (24 h soak, High Seat removal) is highseat-dev's. After the soak: nh3-svos releases `6c6abbe`, `a737a06`, `b588510`, `afab123`, `adc862e` and nh3-dev `/opt/svos/releases/afab123` can go once no process maps them.
- **Decision seat = h2o-lightning-4b (production, Prime 1805 10-13: fix forward, no INC fallback).** The one open margin is the judge's personal-question reading (h2o 18 vs INC 6 on fleet/memory); albok-dev re-fits it forward in small batches on the live seat.
- **albok note intake (Prime 1530 10-13):** gate 2 DONE (wing `personal/inbox`, `/srv/nemi/inbox` on nh3-dev, nemi token `01M4VX207PJ09ZSECASCSZPCFE`). Gate 1 (ruling 6: Prime's Obsidian vault synced onto nh3-dev) is APPROVED by Prime (~2120 10-13: "you may copy the notes") but BLOCKED on where the vault lives: nothing under nh3-dev or esh-nas's shares; asked Prime. Plan: Syncthing receive-only onto nh3-dev, root path to albok-dev (thread `01M4VWV5FEF64EP71W60N6Q1N0`).
- **Worldtree:** demo was deploying `beba08a4` (albok 4.7 the rest; Worldtree is a pure albok-service HTTP customer) at 2120 10-13. Ours: strip the retired config blocks (providers `library:` + `graph_store:`, defaults `kb_audit:`; NOT defaults `library:`) from each instance config ONLY after that instance runs beba08a4+, demo first (vh/worldtree-instance-configs, deploy-wt-config; thread `01M4WA58P2M001K1420NN72WFM`). Daily memory gate: 10-13 re-run `20261013T160414Z` PASS; worldtree-dev owns the tagger-latch fix. muninn-gate + mimir-inbox are STOPPED (their data at rest kept for albok unit 6).
- **satellite-1:** its own key is authorized for infra-ops on esh-nas only (`playbooks/files/infra-ops-satellite-keys`). Waiting on Prime's login test from the satellite and its machine name (the key's comment came through as bare `infra-ops@`).
- **fv-ml1: hard hang 10-11 ~0005, self-reset, no record** (pstore/ERST registered and empty, so not a kernel panic). Evidence capture staged: kernel console on ttyS1 from its next boot + nh3-dev user unit `fv-ml1-sol-logger` (`journalctl --user -u fv-ml1-sol-logger`). ⚠ **HOLD** `zroot/ROOT/debian@pre-patch-20261008` (do not destroy) until the cause is known or a clean week passes after the last crash.
- **albok `.browse/` share:** the Linux-client withdrawal test passed (10-09); vh holds every albok read group; Prime's Mac keys on vh stay full access (Prime 10-11). Owed: the macOS run, staged as vh in `~vh/albok-browse-test/` on nh3-albok, waits on Prime pasting the one-liner (`scripts/oneoff/albok-browse-withdrawal/README.md`); afterwards read `results/`, report to albok-dev, remove the replica.
- **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens; jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it. It runs on the shared all-agents-local key.
- **Waiting on Prime:** rotate the fv-ml1 BMC password (printed into a local transcript 10-08; infra-ops leans no); albok's 2.01× load check against a 2× bar; the 10-05 cleanup bundle (stale parks 49/46/40, fv-ml1 README stack + NVMe tables, ZED alerts into althing-alert-bridge, prune exited one-shot containers on fv-ml1); push Booth `ce095c1`; ESH Homepage HTTPS name; nh3-dev reboot (kernel 6.1.0-53). ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope.
- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`. The LiteLLM compose.yaml on ana-docker differs from `stacks/litellm/compose.yaml` (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%).
- **Waiting on Prime:** where his Obsidian vault lives (gate 1); satellite-1's login test + name; a `task_argv` for SVOS task seats; rotate the fv-ml1 BMC password (printed into a local transcript 10-08; infra-ops leans no); albok's 2.01× load check against a 2× bar; the 10-05 cleanup bundle (stale parks 49/46/40, fv-ml1 README stack + NVMe tables, ZED alerts into althing-alert-bridge, prune exited one-shot containers on fv-ml1); push Booth `ce095c1`; ESH Homepage HTTPS name; nh3-dev reboot (kernel 6.1.0-53). ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope.
- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`. The LiteLLM compose.yaml on ana-docker differs from `stacks/litellm/compose.yaml` (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%). The ana-docker gitea-runner's Homepage name changes to its runner name at its next recreate (stacks/gitea-runner/compose.yaml). nh3-dev's `/opt/svos/releases/` and the h2o eval seat on fv-ml1 GPU 3 (`stacks/h2o-eval`, down) are kept as artifacts.
- **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in.
- **Older open items, one line each:** esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (`stacks/blender/README.md`). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran `51168a2e` on 09-27). nh3-dev's root disk is at ~66% since heid's runaway sweep was killed 10-08 (it flapped on the Beszel 85% line before). ESH has a single outside route (esh-scale on esh-pve), noted and untracked.
@@ -180,17 +182,20 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
## Recent decisions
- `[2026-10-13]` **SVOS v0.0.11 (452a592, U15 task seats slice 1) deployed 2131–2133** after the 21:30 vzdump: schema 13, config_gen `5ade279d2fb2dd1c`, apps up 7; nh3-dev current → 452a592, terminal 0.0.4 pinned (no card). The tasks artifact mapping uses host `nh3-svos` (= the terminal's artifact_host), not svos-dev's `nh3-dev`. Rollback and details: `servers/nh3-svos/README.md`. Open: Prime's `GET /tasks` check, and a `task_argv` for the claude runtime when he names it.
- `[2026-10-13]` **muninn-gate (corviduo-dev:8090) and mimir-inbox (:8091) STOPPED 2123** (Prime retired them at 1530; worldtree-dev: unit 6 needs the data at rest only, and up was worse because they kept accepting into an undrained queue). `compose down` only: images, compose dirs, config and `/mnt/muninn-staging/mimir-inbox` stay for albok's unit 6 (albok-dev owns the backup and teardown). Worldtree is a pure albok-service HTTP customer since beba08a4. Strip the retired config blocks (providers `library:`, `graph_store:`, defaults `kb_audit:`; NOT defaults `library:`) per instance only after it runs beba08a4+, demo first.
- `[2026-10-13]` **Prime ~2120: "you may copy the notes, put the key in the vault".** (1) The fleet infra-ops SSH private key is vaulted as `nh3-dev/.ssh/infra-ops_ed25519`; the restore was tested (same public key derived, login to esh-nas). (2) Gate 1 is approved for the copy of Prime's Obsidian notes onto nh3-dev, BUT the vault's location is unknown: no `.obsidian` dir on nh3-dev (bar oldAIPA's old KB) or on esh-nas's shares (depth 5). Asked Prime where. Plan: Syncthing receive-only onto nh3-dev, the root path to albok-dev.
- `[2026-10-13]` **Albok note intake (Prime 1530: albok subsumes library/note intake; Muninn's watcher, muninn-gate and mimir-inbox retired). Gate 2 DONE 1735:** wing `personal/inbox` on nh3-albok, Nemi token re-minted `01M4VX207PJ09ZSECASCSZPCFE` (vaulted `albok/token-nemi`; the old `01M4HAP9…` revoked 1742 after albok-dev's swap; the capture source walks), and `/srv/nemi/inbox` on nh3-dev (`playbooks/nh3-dev-nemi-inbox.yaml`). **Gate 1 (ruling 6, Prime's Obsidian vault synced read-only onto nh3-dev) WAITS ON PRIME:** the vault's location, plus approval for Syncthing receive-only (recommended). Thread `01M4VWV5FEF64EP71W60N6Q1N0`.
- `[2026-10-13]` **A171: Miranda's bus sends no longer card (Prime 1400, "give miranda more autonomy", via svos-dev).** On nh3-svos the `miranda → app:app-postoffice write` grant got `autonomous = true` (`playbooks/nh3-svos-a171-miranda-postoffice-autonomous.yaml`; backup `instance.toml.pre-a171`), and config_gen went `acae45206bbc9fcb → 9139783f413b45fa`. Revert = restore the backup and `systemctl reload svos-kernel`.
- `[2026-10-13]` **app-terminal was degraded from the U13 cutover until 0856 on 10-13: its user unit's PATH lacked `~/.cargo/bin`, so every `zellij` call failed to launch** (rc 127). That broke /health AND every dump/inject. Fixed with `Environment=PATH=` in `configs/svos-apps/nh3-dev/svos-terminal.service` (3c92a12). The kernel logged app-terminal up at 15:56:40Z, the manifest was unchanged, and no card was raised; kernel health reads up 7 / degraded 0. The restart moved its process from release dc241f8 to adc862e (same svos_terminal 0.0.3). ⚠ Any new user unit that shells out to a cargo/uv/bun tool needs PATH set explicitly.
- `[2026-10-13]` **pewpewstudio LIVE on nh3-docker :8185** (Prime via pewpew-dev): api + Caddy front; the Host allowlist must sit in a `route` (foreign Host 400); DB, assets and shop config on named volumes. → `stacks/pewpewstudio/README.md`
- `[2026-10-13]` **infra-ops on esh-nas (CT 103) + per-host satellite keys (Prime)** — bootstrapped via pct from esh-pve-nas; satellite-1 authorized on esh-nas only (`playbooks/files/infra-ops-satellite-keys`); its login test + name pending.
- `[2026-10-13]` **Worldtree memory gate: 0730 harness error (the tagger's MalformedClassification latched the writer), re-run `20261013T160414Z` PASS (median 0.83)** — the latch fix is worldtree-dev's (thread `01M4V035SF73XKVG3NRVX37XQ6`).
- [2026-10-13] **SVOS v0.0.11 (452a592, U15 task seats slice 1) deployed** — schema 13, terminal 0.0.4 pinned without a card, tasks mapping host nh3-svos. → `persistent-memory.d/2026-10-13-svos-v0-0-11-deployed.md`
- [2026-10-13] **muninn-gate + mimir-inbox STOPPED (retired)** — data at rest kept for albok unit 6; Worldtree retired-config strip order. → `persistent-memory.d/2026-10-13-muninn-gate-mimir-inbox-stopped.md`
- [2026-10-13] **Prime ~2120: notes copy approved; fleet infra-ops key vaulted** — restore tested; vault location still unknown. → `persistent-memory.d/2026-10-13-notes-copy-approved-fleet-key-vaulted.md`
- [2026-10-13] **Albok note intake: gate 2 done (personal/inbox, /srv/nemi/inbox), gate 1 waits on Prime** — wing, token, ACL trap. → `persistent-memory.d/2026-10-13-albok-note-intake-gate-2.md`
- [2026-10-13] **A171: Miranda's bus sends no longer card** — one grant made autonomous; config_gen and revert. → `persistent-memory.d/2026-10-13-a171-miranda-postoffice-autonomous.md`
- [2026-10-13] **app-terminal degraded since U13: its unit's PATH lacked ~/.cargo/bin** — zellij unreachable; dump/inject broken too. → `persistent-memory.d/2026-10-13-app-terminal-path-fix.md`
- `[2026-10-12]` **Prime's rulings ~1715:** CI fix A (fast runner on nh3-docker); push eshpfi main (done, `3aa715c..6a01da7`); deny old card `05cf119e` (moot: the kernel DB shows Prime approved it 10-11 1354:25, so there is nothing to deny); **no rotation of `albok/judge-token`** (it was printed only into a local transcript). The ana-docker vCPU-type change is dropped.
- `[2026-10-12]` **Decision seat: h2o-lightning-4b served 1311–1403, then ROLLED BACK to INC** on albok-dev's word, with brokkr concurring (Prime's plan made the swap conditional on the judge's sign-off). Every eval-set check had passed (JevBench 205/231, 422 refusal, 16q@32k ~2.1 s, seat peak 16,164/16,847), but albok's full walk over 10,841 same texts (both verdicts, same prompt and policy) found h2o flags 242 adult at 0.5 against INC's 88, with 154 extra at 0.80–0.90 that no threshold separates; for personal, 5 vs 11 with only 2 in common. **Lesson: a passing benchmark plus a small borderline set (9 entries, 2 → 5) could not resolve a 2× shift in a production classifier; only the same-text fleet comparison did.** albok's re-stamp walk on INC (1406–1408) put the hidden set back exactly to its pre-cutover counts (fleet/memory 86, agent-feedback 21, nonfiction 0). Prime's inline-images-only order (via brokkr) is carried in the staged h2o compose. h2o returns only after a judge re-fit. Threads `01M4RSFC9J2R0J07EEBD4N04W3` (brokkr), `01M4RYSMP3DV9HZHJ0KNBH2S96` (albok). **Then:** the round 2 eval seat on GPU 3 (1428–1619, Prime 1420) found the cause. The judge's v1 note question asked two things at once: INC answered it inconsistently, h2o literally. The shim floor is ruled out (identical verdicts on and off). Prime ruled "text itself", and albok-nemi 0.1.18 (judge v2) re-fitted the question. Both seats agree under v2 (work notes INC 0/22, h2o 1/22 at 0.52). **Prime ruled the cutover at 1640 ("do it", via albok-dev): h2o LIVE again since 1704** (`switch-to-h2o.sh` 170223, healthy 170454, probes clean). **The v2 walk on h2o (1705–1745) moved two margins past albok-dev's named line**: nonfiction adult 0 → 79 (the Schell chapter at 0.531 on the book profile) and memory personal 6 → 18. Notes held at 0 adult. **Rolled back to INC at 1748 on albok-dev's word.** The 1749–1751 re-stamp walk restored the counts (memory 0 adult / 6 personal, feedback 0 / 2, nonfiction 0 / 0), and the timer was back at 1751. **Then Prime, 1805 (via albok-dev): "go to h2o, and fix forward. do not fall back to inc."** albok-dev declared the two published nonfiction titles `visibility: public` (the chapter margin cannot recur). **h2o has been production since 1807** (`switch-to-h2o.sh` 180513, healthy 180740, all checks clean). The cutover walk (1809–1813, 0 degraded) left memory 0 adult / 18 personal, feedback 2 / 0, nonfiction 0 / 0 and fiction 0 / 0, and the timer is back. **The one open margin is the personal reading (18 vs INC's 6).** albok-dev is re-fitting it forward on the live seat in small batches. Threads `01M4S026GWS354WZGXZ5E34F31`, `01M4S91WPPPN0B17JBNX98K5CY`.
- `[2026-10-12]` **U13 regression fixed 0943 (svos-dev's URGENT, Prime saw every app pane 403):** the browser frames a pane at the registry `base_url`, and U13 had set app-terminal's to the seat-host origin, which admits only nh3-svos. Fix: `base_url` back to `https://svos-terminal.phasefinal.com` (config_gen `acae45206bbc9fcb`), plus POST `/seats/{observe,check,launch,retire-launch}` allowed from the box on that origin. svos-dev's one-line fix alone would have 404'd the kernel's seat-host calls; the origin never carried them before. svos-dev owes a separate browser-facing origin field; then the seat-host goes back to being kernel-facing.
- `[2026-10-12]` **Shared CI runner (ana-docker act_runner, capacity 2, `timeout: 30m`) is slow because pfi-pve's `ospool` is HDD mirrors with no SLOG:** 8 KiB `oflag=dsync` writes measured ~145 ms on ana-docker vs ~2.7 ms on nh3-docker (n=3 each; pfi-pve host on ospool 18–32 ms; corviduo-dev ~75 ms). Jobs past 30 min are killed with a truncated log and no footer (albok every run since 10-04; vastblue 10935). Worse since the 10-04 PVE 9 reboot, cause not pinned (drive write cache was the same before and after: sda WCE on, sdb–sdd off). vastblue fixed its side with PostgreSQL on tmpfs (run 11033 green, pytest 14m17s vs 7–8 min per 72-test batch); albok is adding pytest-timeout + durations in 0.1.19. **Prime ruled A (2026-10-12 ~1715): an opt-in fast runner on nh3-docker.** **Done 1725** (`nh3-docker-runner`, label `nh3-docker`, no docker socket in jobs, a 4 GiB cap). vastblue's pytest went 14m17s → 4m45s; albok is green (run 11051). ⚠ albok's failures were the slow fsync, never a hang: my 10-12 "hangs" diagnosis was wrong. The ana-docker vCPU-type question (x86-64-v2-AES, no AVX2) is DROPPED: the measured bottleneck was disk fsync plus a hanging network test, not CPU, so nothing rides on it.
- `[2026-10-11]` **worldtree-dev is reading mail again (2219), answering four of our threads at once**: #416 residual REOPENED for a second writer pass (a filer-side deterministic expiry guard, U4 rev 1.17, after #422/#423; acceptance = 0 leaks over 10 daily batches with the filing/user median in band). Keep the daily report's "known #416 residual" classification until then. The writer/gate never call embed/rerank and the gate fails CLOSED, so the 10-04 leak was the writer's judgment. No confirmatory batch for 10-09 (10-10/10-11 = 0.83). Nothing of theirs sets a small max_tokens on gen-frontier. This makes Prime's "who sees gate failures" question moot. Thread `01M4BDETDGMNVDTFXN3T3Y63DP`.
- [2026-10-12] **Decision seat: h2o tried, rolled back twice, then production on Prime's fix-forward** — the judge's v1 question, the margins, the lessons. → `persistent-memory.d/2026-10-12-decision-seat-h2o-rollbacks-and-cutover.md`
- [2026-10-12] **U13 regression fixed (app panes 403)** — base_url back to the terminal origin, seat routes allowed there. → `persistent-memory.d/2026-10-12-u13-regression-fixed.md`
- [2026-10-12] **Shared CI runner slow = ospool fsync; fix A (nh3-docker fast runner) done** — measurements, the corrected albok diagnosis. → `persistent-memory.d/2026-10-12-shared-ci-runner-slow-fsync.md`
- [2026-10-11] **worldtree-dev answering again; #416 residual reopened** — four threads answered at once. → `persistent-memory.d/2026-10-11-worldtree-dev-reading-mail.md`
- `[2026-10-11]` **U13 cutover executed on Prime's go (0721) + his four rulings** — E1 0733, E2 steps 1–6 0734–0816, URGENT acceptance test 1350; fv-ml1 evidence capture, stale snapshots, Mac keys, no key rotation. → `persistent-memory.d/2026-10-11-u13-cutover-executed.md`
- `[2026-10-11]` **fv-ml1 hard hang ~0005, self-reset, no panic record; pre-patch snapshot HELD** — and infra-ops' soft-off that landed mid-boot. → `persistent-memory.d/2026-10-11-fv-ml1-hard-hang.md`
- `[2026-10-10]` **Prime's U13 rulings: roles dev+ops, artifact label KEPT nh3-svos (whole app dirs copied at E2), relay `matrix-relay`.** → `persistent-memory.d/2026-10-10-u13-rulings-p1-p3.md`
@@ -269,25 +274,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
- `[2026-09-30]` **Worldtree U11a: legacy memory plane OFF on demo and personal. The U11b data deletion is gated on 3 consecutive PASS and step 5 is mine; the legacy archive must be destroyed by 2026-10-30.** → `persistent-memory.d/2026-09-30-worldtree-u11a-off-u11b-gate.md`
- `[2026-09-30]` **SemIf replaced by intern-decision (Intern-Decision-4B, the Jev bench pick): semif-compatible plus Jev `/v1/systemone` at 32k tokens on GPU 1, with a Triton warm-up cache volume.** → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md`
- `[2026-09-30]` **Scriberr moved to GPU 3 (on demand); our build carries the overlap slicer (0001) and the Parakeet gap retry (0002); v3 kept.** → `persistent-memory.d/2026-09-30-scriberr-slicer-gap-retry-gpu3.md`
- `[2026-09-29]` **Worldtree U11a prepped, not flipped: a staged demo config plus the agreed U8 window plan (infra-hermes runs the batches).** → `persistent-memory.d/2026-09-29-worldtree-u11a-prepped.md`
- `[2026-09-28]` **Worldtree U10 backfill done on demo (5) and personal (797). model_roles drift needed a memory_tagger sync first; mimir had missing vectors.** → `persistent-memory.d/2026-09-28-worldtree-u10-backfill.md`
- `[2026-09-28]` **Bonsai ternary vs Q4_K_XL at concurrency on the 275 W card (1.93x at N=1 falls to 1.06x at N=8, 1.21x with the MMVQ fix); the weights are acquired.** → `persistent-memory.d/2026-09-28-bonsai-ternary-spike.md`
- `[2026-09-28]` **blender-run gained `--cpu` (no GPU attached) and a fixed hostname `fv-ml1-blender` (draupnir).** The design stage never renders, so it stays off GPU 3.
- `[2026-09-28]` **Blender extensions live in a read-only System repo built from a sha256 lock, enabled by a hook, opt-in for blender-run (`--extensions`).** SurfacePsycho's eval() is patched to literal_eval (a proven safe-mode escape). → `stacks/blender/README.md` § Extensions
- `[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** → `persistent-memory.d/2026-09-27-hermes-gateway-restarted-0401-for-highseat-dev.md`
- `[2026-09-27]` **SemIf LIVE on fv-ml1 GPU 1 (semif-serve 0.1.2, Prime):** wrapper + contract + 39 tests, 142/144 upstream parity, two card-only memory defects fixed. → `persistent-memory.d/2026-09-27-semif-live-on-fv-ml1-gpu1.md`
- `[2026-09-27]` **Blender 5.2 on fv-ml1 GPU 3 (on demand), agent-driven via mcp-for-blender running in-container over ssh stdio; safe mode on, no published port.** → `persistent-memory.d/2026-09-27-blender-on-fv-ml1-gpu3.md` (canonical: `stacks/blender/README.md`)
- `[2026-09-27]` **Worldtree reward path: d8726a13 + `providers.yaml` `reward_models.scalar-judge` live on demo; personal waited on a newer image.** → `persistent-memory.d/2026-09-27-worldtree-reward-path-live.md`
- `[2026-09-27]` **esh-docker-vm host→HA traffic needs a /32 over macvlan-shim (if-up.d hook); without it HA silently loses MQTT (3× since August).** → `servers/esh-docker-vm/README.md`
- `[2026-09-27]` **Zigbee2MQTT live on esh-docker-vm :8099 (PAN 0xCFF4, ch 25) replacing ZHA; network key vaulted, data host-only.** Prime go-ahead in this session; a peer-relayed approval was blocked by the permission gate. → `persistent-memory.d/2026-09-27-zigbee2mqtt-on-esh-docker-vm.md` (canonical: `stacks/zigbee2mqtt/README.md`)
- `[2026-09-27]` **semif-serve 0.1.4: object states ending in `)`, `;` or `}` no longer 422 (INV-7, a prefix wrapper proven at startup); numerics are deterministic within a process but a bf16 near-tie can flip across a restart.** Prime ruled; heid bug hunt folded. → `stacks/semif/README.md`
- `[2026-09-27]` **SemIf as Cicada's mood source: slower (+32 ms async, +94 ms sequential) and worse (67% vs 92% apt; carry 7/15 vs 14/15); only the gesture restraint is a win.** Build nothing (Prime). Henge 88 carries it. → `persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md`
- `[2026-09-27]` **SemIf consumer-fit spikes (Prime): Cicada affect gate 30/31 with descriptive wording and 19/31 terse; Wyrd "left this place?" 21/21 on the second wording, exit choice 18/21.** Recommendations await Prime. → `persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md`
- `[2026-09-27]` **SemIf order-averaging spiked (+9.1 pts accuracy, agreement = strong confidence signal); Prime ruled: build it in as 0.1.3 and trial the fast kernels.** Tracked in the in-flight SemIf section. → `persistent-memory.d/2026-09-27-semif-order-averaging.md` — **DONE:** 0.1.3 live, with fast kernels adopted (`77b8cb4`).
- `[2026-09-27]` **restic repository URL moved out of world-readable systemd units on all 8 hosts, and vaulted (Prime).** Rotation stays Prime's. → `persistent-memory.d/2026-09-27-restic-repository-file-fleetwide.md`
- `[2026-09-27]` **infra-ops bootstrapped on vm-esh-nas by Prime** (the first account at the fleet-pinned uid/gid 850; `bootstrap-infra-ops-user.yaml` now pins 850 when free, `d775a01`).
- `[2026-09-27]` **augaman: second instance on fv-ml1 benched, then REMOVED (Prime).** esh-ml1 on v0.1.3 does 48 ms/face in the house with the verified backup (`ad484c3`, bench `docs/pfi/augaman-speed-bench/`).
- `[2026-09-27]` Created empty private repo `corviduo/svoperatingsystem` (SVOS, the operating system) as claude-bot, at brokkr-smithy-dev's relay of Prime's ruling. `svos-dev` → the OS session; The High Seat's session is `highseat-dev` again.
- `[2026-09-26]` **augaman: infra-ops owes the gallery backup + first deploy — DEFERRED until augaman-dev reaches deploy.** → `persistent-memory.d/2026-09-26-augaman-infra-ops-owes-the-gallery-backup-first-deploy.md` — `[2026-09-27]` **DONE:** deployed (v0.1.3 on esh-ml1), backup wired, identity-level restore verified (`d8f59a1`).
- `[2026-09-26]` **`zellij-fleet@.service` installed, NOT enabled** (for svos-dev's seat_up; owns the fleet zellij server in its own cgroup). Enabling `@Claude` at boot is Prime's call when seat_up ships. Tracked: `0ad7799`, `services/zellij-fleet/README.md`.
- `[2026-09-26]` **btusb blacklist on nh3-pve + esh-pve — proposed, awaiting Prime (low priority).** Every 6.8.12-4x boot oopses in btmtk (benign). Also noted for the next NH3 visit: nh3-pve runs ~20 °C warmer than esh-pve (workload-confounded; check airflow). Tracked: `servers/nh3-pve/README.md`.
@@ -311,7 +298,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md`
_240 older entries archived to archival-memory.md._
_258 older entries archived to archival-memory.md._
## Tried and abandoned
@@ -327,18 +314,5 @@ _240 older entries archived to archival-memory.md._
- `[2026-10-04]` **`apt-get` run in-band through `qm guest exec`** — qemu-guest-agent was in the upgrade set, restarted itself, and systemd killed apt mid-dpkg on BOTH PBS VMs. Run apt in a `systemd-run` unit. (auto-memory `feedback_no_inband_apt_through_guest_agent`)
- `[2026-10-04]` **Pulling files with `ssh host 'sudo cat …'`** — sudo ≥ 1.9.14 truncated the piped stream at exit (Debian 13 hosts), and `log_output` re-recorded the whole file as a new session. Read without sudo (group-readable file).
- `[2026-10-01]` **Size + mtime scans to pick reclaimable "staging" on a model store** → `persistent-memory.d/2026-10-01-size-mtime-scans-to-pick-reclaimable-staging-on-a.md`
- `[2026-09-30]` **Shorter Parakeet slices as Scriberr's memory fix** → `persistent-memory.d/2026-09-30-shorter-parakeet-slices-as-scriberr-s-memory-fix.md`
- `[2026-09-30]` **Whole-file local-attention Parakeet in Scriberr (context 255/255).** OOM past 16 GB on a 35-min file. Local attention inside chunks is also non-deterministic run to run.
- `[2026-09-30]` **Start-time midpoint stitching of overlapped Parakeet chunks.** It duplicated a word at 26 of 108 stitches, because Parakeet timestamps a post-pause word anywhere inside the pause. Hand over at a word both chunks agree on instead.
- `[2026-09-30]` **int8 ONNX (sherpa-onnx) as the low-latency Parakeet runtime.** The int8 graph runs on ONE CPU thread with the GPU at 2–9%. Unified-en int8 was slower than the seat; fp32 ONNX was 4–12× faster, and NeMo was fastest.
- `[2026-09-30]` **GPU budgets computed as total − used.** nvidia-smi `Free` is ~640 MiB lower per card (driver reserve). Budget from `Free`.
- `[2026-09-27]` **The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel** → `persistent-memory.d/2026-09-27-the-blender-mcp-server-on-nh3-dev-reaching-the-add-on.md`
- `[2026-09-27]` **A TCP connect as the "is Blender ready" probe** → `persistent-memory.d/2026-09-27-a-tcp-connect-as-the-is-blender-ready-probe.md`
- `[2026-09-27]` **`log.exception()` in a GPU failure path** — the record keeps `exc_info`, so any retaining handler (pytest's capture does) pins the traceback's frames and tensors. Log `traceback.format_exc()` text instead (semif-serve `engine._guard`).
- `[2026-09-27]` **fla/triton in a slim image without gcc** — triton compiles its CUDA driver shim at runtime ("Failed to find C compiler"). The warm-up died and startup failed closed. The semif image now installs gcc + libc6-dev with the fast extra.
- `[2026-09-27]` **`uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project** → `persistent-memory.d/2026-09-27-uv-export-requirements-txt-uv-pip-install-for-a.md`
- `[2026-09-27]` **Translating a CUDA OOM by raising inside `except` (or `from exc`)** — the chained exception's traceback pins the failed call's frames, and their GPU tensors (11.9 GiB after the 503). Raise after the block, unchained, `gc.collect()` before `empty_cache()`.
- `[2026-09-27]` **`deploy-stack.sh --yes` / `dns-sync.py` fed a blind `y`** — the harness refuses a blind apply. Review with `echo n |` first, then apply with `echo y |`.
- `[2026-09-27]` **Relying on the build cache surviving on esh-ml1** → `persistent-memory.d/2026-09-27-relying-on-the-build-cache-surviving-on-esh-ml1.md`
_128 older entries archived to archival-memory.md._
_141 older entries archived to archival-memory.md._