vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: fv-ml1 hard hang 2026-10-11 ~0005 (self-reset); hold the pre-patch snapshot destroy

Browse Source
This commit is contained in:
vh
2026-10-11 00:19:46 -07:00
parent 754e0fd6dc
commit 3c05ae8ba2
1 changed files with 2 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+2 -1
View File
@@ -120,7 +120,7 @@ _As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections ca
- **Nothing is mid-action.** No claims held (`scripts/ops-log claims`). eshpfi is pushed through this snapshot's commit. ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- <paths>` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose (auto-regenerated). - **Nothing is mid-action.** No claims held (`scripts/ops-log claims`). eshpfi is pushed through this snapshot's commit. ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- <paths>` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose (auto-regenerated).
- **U13 phase B (infra) DONE 2026-10-10 1646–1650 on Prime's go:** nh3-svos on v0.0.9 (`afab123`, schema 12) with app-terminal + app-postoffice HELD on b588510 by drop-ins (remove at the edit; keep `releases/b588510` until then); kernel app origin `svos-kernel.phasefinal.com` (LAN, nh3-dev only); `svos-seathost` + `svos-highseat` + `svos-miranda` origins on nh3-dev (nh3-svos only); apps env `/opt/svos/current/apps-env` (dc241f8: bridge, terminal, miranda 0.0.3, agent) on nh3-dev; three enrollment secrets vaulted, app-high-seat declared (B5). Next is the atomic edit (highseat-dev's runbook `svos/docs/u13-cutover-runbook.md`, Prime's go; E2 is S→I). → `servers/nh3-svos/README.md`, `servers/nh3-dev/README.md` - **U13 phase B (infra) DONE 2026-10-10 1646–1650 on Prime's go:** nh3-svos on v0.0.9 (`afab123`, schema 12) with app-terminal + app-postoffice HELD on b588510 by drop-ins (remove at the edit; keep `releases/b588510` until then); kernel app origin `svos-kernel.phasefinal.com` (LAN, nh3-dev only); `svos-seathost` + `svos-highseat` + `svos-miranda` origins on nh3-dev (nh3-svos only); apps env `/opt/svos/current/apps-env` (dc241f8: bridge, terminal, miranda 0.0.3, agent) on nh3-dev; three enrollment secrets vaulted, app-high-seat declared (B5). Next is the atomic edit (highseat-dev's runbook `svos/docs/u13-cutover-runbook.md`, Prime's go; E2 is S→I). → `servers/nh3-svos/README.md`, `servers/nh3-dev/README.md`
- **Owed by me (no input needed):** - **Owed by me (no input needed):**
- Destroy `zroot/ROOT/debian@pre-patch-20261008` on fv-ml1 ~10-15 if nothing regressed. - ⚠ **HOLD** the destroy of `zroot/ROOT/debian@pre-patch-20261008` on fv-ml1 (was ~10-15): fv-ml1 hard-hung 2026-10-11 ~0005, the first crash since that patch (NVIDIA 580.178.04, kernel 6.12.111). Keep the rollback point until the cause is known or a clean week passes after the last crash.
- nh3-svos housekeeping: releases `6c6abbe` and `a737a06` are no longer referenced by the front door and can be deleted (`ac42ace` holds the same static as the current `b588510`). - nh3-svos housekeeping: releases `6c6abbe` and `a737a06` are no longer referenced by the front door and can be deleted (`ac42ace` holds the same static as the current `b588510`).
- **Worldtree memory gate watch CLOSED 10-10:** batch 20261010T143011Z PASS (user median 0.83 over runs 0.83/0.75/0.92), so the 10-09 filing fail did not repeat and flash-next is not investigated. ⚠ worldtree-dev still is not reading mail (Prime's call, below). - **Worldtree memory gate watch CLOSED 10-10:** batch 20261010T143011Z PASS (user median 0.83 over runs 0.83/0.75/0.92), so the 10-09 filing fail did not repeat and flash-next is not investigated. ⚠ worldtree-dev still is not reading mail (Prime's call, below).
- **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens. jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it, as gen-large did (preemptions; decode down to 2–16 tok/s). jekyll's context grows ~1k/call toward the 262k ceiling. It runs on the shared all-agents-local key; a key of its own would make its traffic attributable. - **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens. jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it, as gen-large did (preemptions; decode down to 2–16 tok/s). jekyll's context grows ~1k/call toward the 262k ceiling. It runs on the shared all-agents-local key; a key of its own would make its traffic attributable.
@@ -186,6 +186,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
## Recent decisions ## Recent decisions
- `[2026-10-11]` **fv-ml1 hard hang ~0005, self-reset ~0008, back by 0015 (all seats).** Journal ends mid-normal-operation (no shutdown), pstore empty, SOL shows nothing (no serial kernel console), BMC restart cause `unknown`, SEL only shows the 00:08 boot events (⚠ the BMC clock is 7 h behind, so ipmitool prints them as 17:08 the day before). Idle at the time (Beszel 0004: CPU 0.15%, no Scriberr job). The sp5100 TCO watchdog exists (60 s). infra-ops sent an ACPI soft-off at 00:12:14 believing it still hung; it landed in early boot before logind watched the button, so no effect. Open (Prime): capture evidence next time (serial console on the kernel cmdline for SOL, and/or kdump); the pre-patch snapshot is HELD.
- `[2026-10-10]` **Prime's U13 rulings (~1705): seat roles `dev` + `ops`; artifact host label KEPT `nh3-svos` (old inbox links keep working; infra-ops copies the WHOLE `/var/lib/svos/apps/terminal` (E2 step 2) and `/var/lib/svos/apps/postoffice` (step 6) to nh3-dev at the same paths, owner lkraven, modes kept, each after its nh3-svos unit stops and before the nh3-dev one starts: the artifacts.sqlite index and the bridge's journal must travel with the files); Matrix relay persona `matrix-relay`.** A two-label hybrid is impossible: `svos_connector.artifacts.relative()` refuses any host but the app's one `artifact_host`. Relay secret `nh3-svos/matrix-relay-enrollment-secret` minted. svos-dev's payload: svoperatingsystem `docs/operations/u13-edit-payload.md` @ `7d7ed1f` (enrollment sha256 placeholders: fill at E2 from `secret get <item> | sha256` — never commit them). - `[2026-10-10]` **Prime's U13 rulings (~1705): seat roles `dev` + `ops`; artifact host label KEPT `nh3-svos` (old inbox links keep working; infra-ops copies the WHOLE `/var/lib/svos/apps/terminal` (E2 step 2) and `/var/lib/svos/apps/postoffice` (step 6) to nh3-dev at the same paths, owner lkraven, modes kept, each after its nh3-svos unit stops and before the nh3-dev one starts: the artifacts.sqlite index and the bridge's journal must travel with the files); Matrix relay persona `matrix-relay`.** A two-label hybrid is impossible: `svos_connector.artifacts.relative()` refuses any host but the app's one `artifact_host`. Relay secret `nh3-svos/matrix-relay-enrollment-secret` minted. svos-dev's payload: svoperatingsystem `docs/operations/u13-edit-payload.md` @ `7d7ed1f` (enrollment sha256 placeholders: fill at E2 from `secret get <item> | sha256` — never commit them).
- `[2026-10-10]` **U13 phase B built (Prime 1639: "implement as asked")**: B1 kernel 0.0.9 + schema 12 (vzdump `vm/111/2026-10-10T23:45:41Z` + offline `pre-afab123` copy first), B2 `svos-kernel.phasefinal.com` → `--app-listen 127.0.0.1:8410` (allowlist nh3-dev; `/auth/kernel-key` ADDED to svos-dev's path list because every connector client fetches it), B3 seat-host origin + bridge env on nh3-dev, B4 app-high-seat origin; then svos-dev's dc241f8 amendment: full method-pinned path list, Miranda's app + agent env and `svos-miranda` origin on nh3-dev, three secrets minted to the vault, app-high-seat declared. Found + fixed: nh3-dev's fleet Caddy shared admin :2019 with ttyd-caddy (SO_REUSEPORT), now :2021. Kernel answers 401 `no principal` through the origin; other sources 403. - `[2026-10-10]` **U13 phase B built (Prime 1639: "implement as asked")**: B1 kernel 0.0.9 + schema 12 (vzdump `vm/111/2026-10-10T23:45:41Z` + offline `pre-afab123` copy first), B2 `svos-kernel.phasefinal.com` → `--app-listen 127.0.0.1:8410` (allowlist nh3-dev; `/auth/kernel-key` ADDED to svos-dev's path list because every connector client fetches it), B3 seat-host origin + bridge env on nh3-dev, B4 app-high-seat origin; then svos-dev's dc241f8 amendment: full method-pinned path list, Miranda's app + agent env and `svos-miranda` origin on nh3-dev, three secrets minted to the vault, app-high-seat declared. Found + fixed: nh3-dev's fleet Caddy shared admin :2019 with ttyd-caddy (SO_REUSEPORT), now :2021. Kernel answers 401 `no principal` through the origin; other sources 403.
- `[2026-10-10]` **esh-nas-pve `nvme/ROOT@pre-pve9` destroyed 0750 (607M)** after a week clean on PVE 9.2.21 / 7.0.14-20 (up 6 d, no err-level journal in 7 d, pools healthy, all guests up; only `openipmi.service` failed, the known item). Left: `nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z` (now 1.43G, a pre-PVE-9 rollback point that is useless as one) and `nvme/subvol-105-disk-0@pre-vaapi-20260911`. - `[2026-10-10]` **esh-nas-pve `nvme/ROOT@pre-pve9` destroyed 0750 (607M)** after a week clean on PVE 9.2.21 / 7.0.14-20 (up 6 d, no err-level journal in 7 d, pools healthy, all guests up; only `openipmi.service` failed, the known item). Left: `nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z` (now 1.43G, a pre-PVE-9 rollback point that is useless as one) and `nvme/subvol-105-disk-0@pre-vaapi-20260911`.