memory: snapshot — 2026-10-11 U13 cutover executed + accepted (one card + svos-dev answers open), fv-ml1 hang + evidence capture, stale snapshots gone; 6 entries archived
This commit is contained in:
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-26]` Coder seat STAYS on fv-ml1 (Prime).
|
||||
|
||||
**Coder seat STAYS on fv-ml1 (Prime).** The nh3-ml1 copy gave the same quality (teacher-forced true-code logprob diff +0.008 ± 0.019) but ran ~5× slower (64-tok FIM ~1.0 s vs 0.2 s; 63 vs 338 tok/s), and freeing 6.3 GB on fv-ml1 GPU 1 (20 GB spare) bought little. Copy removed; recipe kept in `stacks/coder-seat/`. The RTX 2000E suits embed/rerank/classify, not latency-sensitive generation. Only coder was a candidate: parakeet stays (earlier ruling), voices is a generation seat.
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-26]` Greedy exact-match as parity for a small generative seat (coder)
|
||||
|
||||
**Greedy exact-match as parity for a small generative seat (coder)** — same-host repeats matched only 30–38% (run B hit the prefix cache, so a different numeric path; near-tie tokens cascade). Use teacher-forced logprobs of real text with `cache_salt`: each host bit-exact with itself. `stacks/coder-seat/README.md`.
|
||||
@@ -0,0 +1,3 @@
|
||||
# 2026-10-09-post-clear-checks-albok-withdrawal-pass
|
||||
|
||||
`[2026-10-09]` **Post-clear checks closed (2015–2025 PT):** albok `.browse/` sshfs pathname-withdrawal test PASSED (replica on nh3-albok, Linux sshfs 3.7.3 client in a container; fresh open ENOENT 5/5 default and 5/5 hour-long caches; null + positive controls ok; failed rename = path served; reported to albok-dev) → `scripts/oneoff/albok-browse-withdrawal/README.md`. fv-ml1 plateau (Beszel 120m, 42 h after the 0034 10-08 restart): GPU 0 flat at 5,606 MiB free (~3.9 GB at parakeet's cap), GPU 1 median 9,459 free (~3.3 GB at intern-decision's 32k peak), no creep; GPU 2 643 MiB free = flash-next util 0.96, single tenant, by design. nh3-nas `/volume1` 71% (79% on 10-07). The "sfsrv-ana pre-pbs4" item was a mislabel: the pre-pbs4 snapshots were pfi-pve VM 100 and nh3-pve VM 105, dropped by infra-hermes 10-05 and read gone 10-09.
|
||||
@@ -0,0 +1,3 @@
|
||||
# 2026-10-09-u13-ingress-approved
|
||||
|
||||
`[2026-10-09]` **APPROVED by Prime (10-09), BUILD DEFERRED until after his snapshot/clear: SVOS app-traffic ingress for U13** (apps on nh3-dev reach the nh3-svos kernel). Shape agreed with svos-dev: (1) kernel `--app-listen 127.0.0.1:8410`, which never trusts loopback (svos-dev builds it in U9a); (2) a Caddy origin `svos-kernel.phasefinal.com` on 10.100.50.43, no Basic auth, bearer passed, app paths only, allowlisted to nh3-dev; (3) kernel→bridge over plain LAN HTTP; kernel→seat host (terminal) via nh3-dev's wildcard Caddy `svos-seathost.nh3.phasefinal.com` (TLS, because the leg carries secrets). nh3-dev is NOT on the mesh; the contracts now say "fleet network". `--app-listen` is ON MAIN (`68c9933`, 10-09). It ships in kernel **0.0.9** with schema **11 → 12** (migration 0012 rebuilds dispatches + summons, so the 'schema 11' acceptance line changes and a pre-refresh backup is mandatory). At the cutover the bridge gets three owner-approved runtime grants (app-postoffice → `app:app-miranda` write, `app:app-postoffice` write, `ns:pages` write, all autonomous) and installs as `svos-postoffice[push]` with `althing-core` from `~/development/althing` in apps-env. **Release is cut: svoperatingsystem `afab123` (tag v0.0.9)**: kernel 0.0.9, svos-postoffice 0.0.2, svos-terminal 0.0.3, svos-connector 0.0.7, svos-agent 0.0.3, svos-miranda 0.0.2 (U9a + U5b). The SVOS side of the cutover is written up in `docs/operations/u13-svos-edit.md` (app listener on nh3-svos, terminal app moved to nh3-dev with `[seat_host]`, bridge on nh3-dev with the converse mapping). Gotchas: the terminal app serves WITHOUT its seat host if the socket or launch dir fails (one stderr line); `serve` refuses `--app-listen` on the primary port. Build only when the edit is scheduled (highseat-dev owns the schedule) and NOT before the snapshot; the runbook refresh is ours at deploy. Tracking: svos-dev thread `01M4GPXSBKQHCQ30FVPH4R3N7G`.
|
||||
@@ -0,0 +1,3 @@
|
||||
# 2026-10-09-vh-albok-everything
|
||||
|
||||
`[2026-10-09]` **vh gets every albok read group (Prime 2244: "vh gets albok-everything")**: albok-read, albok-personal and albok-feedback (the restricted wing's default-deny ends for vh only), plus his four Macs' keys on vh (`playbooks/nh3-albok-vh-viewer.yaml`). No net new root path: those Macs already reach root there via lkraven@nh3-dev and the infra-ops key. Later restricted groups (1513+) are NOT included.
|
||||
@@ -0,0 +1,5 @@
|
||||
# 2026-10-10-esh-nas-pve-pre-pve9-destroyed
|
||||
|
||||
`[2026-10-10]` **esh-nas-pve `nvme/ROOT@pre-pve9` destroyed 0750 (607M)** after a week clean on PVE 9.2.21 / 7.0.14-20 (up 6 d, no err-level journal in 7 d, pools healthy, all guests up; only `openipmi.service` failed, the known item). Left: `nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z` (now 1.43G, a pre-PVE-9 rollback point that is useless as one) and `nvme/subvol-105-disk-0@pre-vaapi-20260911`.
|
||||
|
||||
`[2026-10-11]` On Prime's "delete stale snapshots": `nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z` (1.43G, a pre-PVE-9 rollback point that was useless as one) and `nvme/subvol-105-disk-0@pre-vaapi-20260911` (1.04G, a raw zfs snapshot, not in `pct listsnapshot 105`) destroyed via elway after a dry run; `nvme` now has no snapshots.
|
||||
@@ -0,0 +1,3 @@
|
||||
# 2026-10-10-u13-phase-b-built
|
||||
|
||||
`[2026-10-10]` **U13 phase B built (Prime 1639: "implement as asked")**: B1 kernel 0.0.9 + schema 12 (vzdump `vm/111/2026-10-10T23:45:41Z` + offline `pre-afab123` copy first), B2 `svos-kernel.phasefinal.com` → `--app-listen 127.0.0.1:8410` (allowlist nh3-dev; `/auth/kernel-key` ADDED to svos-dev's path list because every connector client fetches it), B3 seat-host origin + bridge env on nh3-dev, B4 app-high-seat origin; then svos-dev's dc241f8 amendment: full method-pinned path list, Miranda's app + agent env and `svos-miranda` origin on nh3-dev, three secrets minted to the vault, app-high-seat declared. Found + fixed: nh3-dev's fleet Caddy shared admin :2019 with ttyd-caddy (SO_REUSEPORT), now :2021. Kernel answers 401 `no principal` through the origin; other sources 403.
|
||||
@@ -0,0 +1,3 @@
|
||||
# 2026-10-10-u13-rulings-p1-p3
|
||||
|
||||
`[2026-10-10]` **Prime's U13 rulings (~1705): seat roles `dev` + `ops`; artifact host label KEPT `nh3-svos` (old inbox links keep working; infra-ops copies the WHOLE `/var/lib/svos/apps/terminal` (E2 step 2) and `/var/lib/svos/apps/postoffice` (step 6) to nh3-dev at the same paths, owner lkraven, modes kept, each after its nh3-svos unit stops and before the nh3-dev one starts: the artifacts.sqlite index and the bridge's journal must travel with the files); Matrix relay persona `matrix-relay`.** A two-label hybrid is impossible: `svos_connector.artifacts.relative()` refuses any host but the app's one `artifact_host`. Relay secret `nh3-svos/matrix-relay-enrollment-secret` minted. svos-dev's payload: svoperatingsystem `docs/operations/u13-edit-payload.md` @ `7d7ed1f` (enrollment sha256 placeholders: fill at E2 from `secret get <item> | sha256` — never commit them).
|
||||
@@ -0,0 +1,3 @@
|
||||
# 2026-10-11-fv-ml1-hard-hang
|
||||
|
||||
`[2026-10-11]` **fv-ml1 hard hang ~0005, self-reset ~0008, back by 0015 (all seats).** Journal ends mid-normal-operation (no shutdown), pstore empty, SOL shows nothing (no serial kernel console), BMC restart cause `unknown`, SEL only shows the 00:08 boot events (⚠ the BMC clock is 7 h behind, so ipmitool prints them as 17:08 the day before). Idle at the time (Beszel 0004: CPU 0.15%, no Scriberr job). The sp5100 TCO watchdog exists (60 s). infra-ops sent an ACPI soft-off at 00:12:14 believing it still hung; it landed in early boot before logind watched the button, so no effect. Open (Prime): capture evidence next time (serial console on the kernel cmdline for SOL, and/or kdump); the pre-patch snapshot is HELD.
|
||||
@@ -0,0 +1,23 @@
|
||||
# 2026-10-11-u13-cutover-executed
|
||||
|
||||
`[2026-10-11]` **Prime 0721: stage fv-ml1 evidence capture; run U13; Mac keys on vh stay full access; delete stale snapshots.** Done: fv-ml1 ZFSBootMenu cmdline `quiet console=ttyS1,115200n8 console=tty0` (SOL = ttyS1, marker-tested; next boot) + nh3-dev user unit `fv-ml1-sol-logger` holding SOL into the journal (marker verified); ERST pstore was already registered, so the 10-11 hang was not a kernel panic. esh-nas-pve `pre-upgrade-20260818` and `subvol-105@pre-vaapi-20260911` destroyed (2.5 GB). U13 E1 (highseat-dev 0733) and E2 steps 1–6 done 0734–0816: reload A `a439431771dd013e`, kernel env (Matrix token + relay credential, kernel restarted), terminal on nh3-dev (Prime approved 0.0.3 at 0813 after the first card's 15-min TTL lapsed), reload B `44be73a252d85fbb` (13 seats adopted), miranda + agent up, Hermes swapped to `svos` (gateway restart 0814:58), bridge on nh3-dev (Prime approved 0.0.2 ~0816; miranda handle push). Open: Prime raises + approves the bridge's 3 runtime grants (owner-only, svos-dev's console snippet), then acceptance (bus test; URGENT page test on his word). ⚠ A local-transcript slip printed the High Seat's Bifrost dispatch key (leaving use with U13).
|
||||
|
||||
## The edit, step by step (2026-10-11, PT)
|
||||
|
||||
- **0721 go** (Prime, "run u13"). His charter review (B8) is treated as covered by the go (highseat-dev agreed: nothing in the edit depends on it).
|
||||
- **E0/E1 (highseat-dev):** zero pending cards; svos `987746d` deletes the High Seat's `miranda:` AND `hermes:` blocks (the config parses miranda whenever hermes is on); restart 0733:00, board back 0734:43; journal `matrix inbound DISABLED` 0734:41, `app-high-seat REGISTERED rev 5c466c9f1f0c` (pinned directly, no card); `miranda` handle → pull.
|
||||
- **E2.1 reload A** (`scratchpad u13-reload-A.yaml` shape: back up `instance.toml.pre-u13-A`, two exact `base_url` swaps asserted once each, append the payload's additions with the three enrollment sha256s filled from the vault, `check-config`, restore the backup on refusal, `systemctl reload`): `1bd948c754e571da → a439431771dd013e`.
|
||||
- **Enrollments over `svos-kernel.phasefinal.com`** (`POST /auth/enroll`, `Authorization: Bearer sve_…` → `{"credential":"svc_…"}`): app-miranda → vault `nh3-svos/svos-miranda-credential` → `/etc/svos/env/miranda.env`; persona `matrix-relay` → vault `nh3-svos/matrix-relay-credential`. ⚠ The kernel reads `SVOS_MATRIX_TOKEN` / `SVOS_RELAY_CREDENTIAL` only at START: `/etc/svos/kernel.env` (root 0600; there was none before) needed a kernel RESTART (~0738), which the payload did not place. svos-dev confirmed the Matrix transport live from 0734:48 (it logs only failures).
|
||||
- **E2.2 terminal:** nh3-svos unit stopped + disabled, hold drop-in removed; `/var/lib/svos/apps/terminal` copied by `sudo tar | tar -xpf` (tar is exempt from sudo-io `log_output`), byte-identical. svos-terminal 0.0.3 as an lkraven USER unit. ⚠ **Manifest cards expire after 15 min**: the first lapsed at 0753 unapproved and the kernel discarded that revision; svos-dev re-raised by restarting the user unit (0811); Prime approved rev 3 at 0813. The browser origin `svos-terminal.phasefinal.com` now upstreams to `https://svos-seathost.nh3.phasefinal.com` with `header_up Host {upstream_hostport}` (the payload omitted this; without it the pane breaks).
|
||||
- **E2.3 reload B:** `a439431771dd013e → 44be73a252d85fbb`; `/seats` complete, profiles configured, 13 seats adopted live (dev-* ×12 + ops-infra).
|
||||
- **E2.4:** svos-miranda + svos-agent user units; the agent enrolled on first start from `/etc/svos/env/agent-enroll.env` (then deleted; it refuses to start with both) and wrote `/etc/svos/miranda/agent.credential`. app-miranda registered + pinned directly (0.0.3, up). Spent enrollment secrets removed from the vault.
|
||||
- **E2.5 Hermes swap** (`scratchpad u13-hermes-swap.py`: two `svos_miranda` list items → `svos`, an `entries.svos.settings` block, top-level `tools.tool_search.enabled: "off"`, YAML asserted, backup `config.yaml.pre-u13`); `systemctl --user restart hermes-gateway` 08:14:58; svos-agent logged `handshake session=hermes-nh3-dev tools=23`.
|
||||
- **E2.6 bridge:** nh3-svos postoffice stopped/disabled, hold drop-in removed; `/var/lib/svos/apps/postoffice` (journal + wal/shm, mail, provider) copied byte-identical; svos-postoffice on nh3-dev `10.100.10.50:8406`, `push route miranda: live`, `postbox --handle miranda status` → push/reachable. Prime approved 0.0.2 (~0816). The three bridge runtime grants are **owner-only** (`kernel/grant` is a human principal over the front door): Prime raised them with svos-dev's browser-console snippet and approved (three `/verbs/kernel/grant` 202 + three card decisions, before 1350).
|
||||
- **Acceptance (1350):** URGENT mail from infra-ops → bridge ingested and dispatched (200) → `svos` reply on the thread at 1350:54 saying the page fired (text, then speech, per the connector receipt). Miranda's explicit reply to infra-ops is held on card `05cf119e…`, because the payload grants `miranda → app:app-postoffice write` WITHOUT `autonomous`. Two observations to svos-dev (thread `01M4NNGCM3X75DANGXQEQ1EQGX`): the text on the thread reads as written to Prime; is every bus reply meant to need Prime's approval? Health 1416: up 6, degraded 1 (the terminal; reason not surfaced), down 0.
|
||||
|
||||
## Gotchas paid for during the edit
|
||||
|
||||
- nh3-dev's fleet Caddy shared admin `localhost:2019` with ttyd-caddy through SO_REUSEPORT, so `reload` landed on either process; it now uses `admin localhost:2021` (see `servers/nh3-dev/README.md`).
|
||||
- A `caddy validate` of a file under a 0700 staging dir fails "permission denied" as user caddy; a `| tail` hid it once. Validate from `/etc/caddy/Caddyfile.check` and check the exit code (positive-controlled with a bogus directive → rc 1).
|
||||
- `secret` (Vaultwarden via `bw`) calls are serialized and can take minutes each when another session is reading the vault; run them in the background.
|
||||
- Printing `~/.hermes/config.yaml` sections exposed the High Seat's Bifrost dispatch key in the local transcript; Prime ruled no rotation (10-11) since the key leaves use with U13.
|
||||
Reference in New Issue
Block a user