esh-pfi-infrastructure/persistent-memory.d/2026-10-11-u13-cutover-executed.md

2026-10-11-u13-cutover-executed

[2026-10-11] Prime 0721: stage fv-ml1 evidence capture; run U13; Mac keys on vh stay full access; delete stale snapshots. Done: fv-ml1 ZFSBootMenu cmdline quiet console=ttyS1,115200n8 console=tty0 (SOL = ttyS1, marker-tested; next boot) + nh3-dev user unit fv-ml1-sol-logger holding SOL into the journal (marker verified); ERST pstore was already registered, so the 10-11 hang was not a kernel panic. esh-nas-pve pre-upgrade-20260818 and subvol-105@pre-vaapi-20260911 destroyed (2.5 GB). U13 E1 (highseat-dev 0733) and E2 steps 1–6 done 0734–0816: reload A a439431771dd013e, kernel env (Matrix token + relay credential, kernel restarted), terminal on nh3-dev (Prime approved 0.0.3 at 0813 after the first card's 15-min TTL lapsed), reload B 44be73a252d85fbb (13 seats adopted), miranda + agent up, Hermes swapped to svos (gateway restart 0814:58), bridge on nh3-dev (Prime approved 0.0.2 ~0816; miranda handle push). Open: Prime raises + approves the bridge's 3 runtime grants (owner-only, svos-dev's console snippet), then acceptance (bus test; URGENT page test on his word). ⚠ A local-transcript slip printed the High Seat's Bifrost dispatch key (leaving use with U13).

The edit, step by step (2026-10-11, PT)

• 0721 go (Prime, "run u13"). His charter review (B8) is treated as covered by the go (highseat-dev agreed: nothing in the edit depends on it).
• E0/E1 (highseat-dev): zero pending cards; svos 987746d deletes the High Seat's miranda: AND hermes: blocks (the config parses miranda whenever hermes is on); restart 0733:00, board back 0734:43; journal matrix inbound DISABLED 0734:41, app-high-seat REGISTERED rev 5c466c9f1f0c (pinned directly, no card); miranda handle → pull.
• E2.1 reload A (scratchpad u13-reload-A.yaml shape: back up instance.toml.pre-u13-A, two exact base_url swaps asserted once each, append the payload's additions with the three enrollment sha256s filled from the vault, check-config, restore the backup on refusal, systemctl reload): 1bd948c754e571da → a439431771dd013e.
• Enrollments over svos-kernel.phasefinal.com (POST /auth/enroll, Authorization: Bearer sve_… → {"credential":"svc_…"}): app-miranda → vault nh3-svos/svos-miranda-credential → /etc/svos/env/miranda.env; persona matrix-relay → vault nh3-svos/matrix-relay-credential. ⚠ The kernel reads SVOS_MATRIX_TOKEN / SVOS_RELAY_CREDENTIAL only at START: /etc/svos/kernel.env (root 0600; there was none before) needed a kernel RESTART (~0738), which the payload did not place. svos-dev confirmed the Matrix transport live from 0734:48 (it logs only failures).
• E2.2 terminal: nh3-svos unit stopped + disabled, hold drop-in removed; /var/lib/svos/apps/terminal copied by sudo tar | tar -xpf (tar is exempt from sudo-io log_output), byte-identical. svos-terminal 0.0.3 as an lkraven USER unit. ⚠ Manifest cards expire after 15 min: the first lapsed at 0753 unapproved and the kernel discarded that revision; svos-dev re-raised by restarting the user unit (0811); Prime approved rev 3 at 0813. The browser origin svos-terminal.phasefinal.com now upstreams to https://svos-seathost.nh3.phasefinal.com with header_up Host {upstream_hostport} (the payload omitted this; without it the pane breaks).
• E2.3 reload B: a439431771dd013e → 44be73a252d85fbb; /seats complete, profiles configured, 13 seats adopted live (dev-* ×12 + ops-infra).
• E2.4: svos-miranda + svos-agent user units; the agent enrolled on first start from /etc/svos/env/agent-enroll.env (then deleted; it refuses to start with both) and wrote /etc/svos/miranda/agent.credential. app-miranda registered + pinned directly (0.0.3, up). Spent enrollment secrets removed from the vault.
• E2.5 Hermes swap (scratchpad u13-hermes-swap.py: two svos_miranda list items → svos, an entries.svos.settings block, top-level tools.tool_search.enabled: "off", YAML asserted, backup config.yaml.pre-u13); systemctl --user restart hermes-gateway 08:14:58; svos-agent logged handshake session=hermes-nh3-dev tools=23.
• E2.6 bridge: nh3-svos postoffice stopped/disabled, hold drop-in removed; /var/lib/svos/apps/postoffice (journal + wal/shm, mail, provider) copied byte-identical; svos-postoffice on nh3-dev 10.100.10.50:8406, push route miranda: live, postbox --handle miranda status → push/reachable. Prime approved 0.0.2 (~0816). The three bridge runtime grants are owner-only (kernel/grant is a human principal over the front door): Prime raised them with svos-dev's browser-console snippet and approved (three /verbs/kernel/grant 202 + three card decisions, before 1350).
• Acceptance (1350): URGENT mail from infra-ops → bridge ingested and dispatched (200) → svos reply on the thread at 1350:54 saying the page fired (text, then speech, per the connector receipt). Miranda's explicit reply to infra-ops is held on card 05cf119e…, because the payload grants miranda → app:app-postoffice write WITHOUT autonomous. (Read back 2026-10-12 from the kernel DB: Prime APPROVED that card at 20:54:25Z, 1354:25 PT, so her reply went out.) Two observations to svos-dev (thread 01M4NNGCM3X75DANGXQEQ1EQGX): the text on the thread reads as written to Prime; is every bus reply meant to need Prime's approval? Health 1416: up 6, degraded 1 (the terminal; reason not surfaced), down 0.

Gotchas paid for during the edit

• nh3-dev's fleet Caddy shared admin localhost:2019 with ttyd-caddy through SO_REUSEPORT, so reload landed on either process; it now uses admin localhost:2021 (see servers/nh3-dev/README.md).
• A caddy validate of a file under a 0700 staging dir fails "permission denied" as user caddy; a | tail hid it once. Validate from /etc/caddy/Caddyfile.check and check the exit code (positive-controlled with a bogus directive → rc 1).
• secret (Vaultwarden via bw) calls are serialized and can take minutes each when another session is reading the vault; run them in the background.
• Printing ~/.hermes/config.yaml sections exposed the High Seat's Bifrost dispatch key in the local transcript; Prime ruled no rotation (10-11) since the key leaves use with U13.

Moved from the in-flight section on 2026-10-14 (verbatim)

• Acceptance closed 1354: Miranda's reply arrived on the thread through the bridge's own send. ⚠ RE-CORRECTED 2026-10-12: card card:05cf119e604649838e2a4c642c161e26 (Miranda's carded send) WAS approved by prime at 2026-10-11 20:54:25Z (1354:25 PT), per the kernel DB (/var/lib/svos/state/svos.db, read-only). The 1412 "NOT approved" correction was itself wrong, because it came from inference, not a read. Nothing to deny; 0 cards pending. Answered by svos-dev 1400: the thread text was Miranda answering bus mail with her own send (carded) instead of letting the bridge carry her turn's reply; the fix (a bus-tag clause in svos_miranda.turns) SHIPPED in v0.0.10 (adc862e), deployed 1410–1411 (nh3-svos kernel 0.0.10 + UI 0.0.8; nh3-dev miranda 0.0.4, /opt/svos/current → releases/adc862e); bus check passed 1412 with no card. Its manifest still says 0.0.3 (hardcoded), so it did not re-register. Then a static-only release 34476a7 (interface UI 0.0.9, tree strands follow speech) at 1712: /opt/svos/current → releases/34476a7 on nh3-svos, no restart, previous handle = adc862e; then ad255a2 (UI 0.0.10, stronger strands, Prime found 0.0.9 too subtle) at 1719, previous = 34476a7; then 627a9fb (UI 0.0.11: the top bar shows v0.0.<n> · <short commit>, from web/build-id.txt via git-archive export-subst, so ALWAYS build from a clean git archive; a plain copy reads dev) at 1722, previous = ad255a2; 61459d2 (UI 0.0.12: voice chorus/flanger toggle, lighter header) at 1739, previous = 627a9fb; 8fa229d (UI 0.0.13: strands span the tree, deeper voice character) at 1800, previous = 61459d2. 89dad7a (UI 0.0.14: the pane lease race, a frame loading before the attach reply now gets its lease) on 10-13 at 0857, previous = 8fa229d. 7f95e54 (UI 0.0.15: the terminal leaves the app list's Open; a bare embedded terminal can only ever wait for a lease, so it is attached per seat) at 0906:37, previous = 89dad7a. Static-only builds: scratchpad svos-static-release.sh <prev> <new> <ui> pattern (rebuild it from the 61459d2 kit if the scratchpad is gone). Static-only releases: build the full release dir (carried artifacts + new static/), switch current without restarting, regenerate the inventory. ⚠ The kernel + apps still RUN from releases/adc862e (verified via /proc/<pid>/maps): do not delete it before they restart. Same on nh3-dev: svos-terminal, svos-postoffice and svos-agent still run from releases/dc241f8 (svos-miranda from adc862e). The non-autonomous miranda → app:app-postoffice write grant is intended (her own sends); making it autonomous is Prime's call, surfaced by svos-dev. Speech leg unconfirmed (svos-dev asked Prime; n=1 vs the earlier 15 s synthesis_failed). Phase R (24 h soak, doctrine + broadcast texts, High Seat removal) is highseat-dev's (svos docs/u13-cutover-runbook.md).
• Rollback material, keep until the soak ends: nh3-svos releases/b588510, instance.toml.pre-u13-{A,B}, /etc/svos/kernel.env.pre-u13 (absent: there was none); nh3-dev ~/.hermes/config.yaml.pre-u13 and ~/.hermes/plugins/svos_miranda. After the soak: nh3-svos releases 6c6abbe, a737a06, b588510 and nh3-dev /opt/svos/releases/afab123 (unreferenced) can go.