feat(svos): v0.0.28 UI 0.0.33 published on nh3-svos (U24 the 3-D tree); no restart

2026-10-19 10:32:40 -07:00
parent c1867088e4
commit 2929bfc60c
4 changed files with 58 additions and 1 deletions
@@ -125,7 +125,7 @@ _As of 2026-10-18 ~2125 PT. The newest subsection is first; older subsections ca
 - **SVOS task launches were broken since v0.0.11 by OUR front door, FIXED 10-19 0118:** the kernel-origin Caddy (`svos-kernel.phasefinal.com`, path allowlist from U13 B2) never admitted the seat host's U15 reports (POST /tasks/start|progress|checking|exit), so every launch died `start_unreported` ("seat launch failed"). Prime's first two tasks (0110/0111) failed that way. Paths added (`configs/svos-front/Caddyfile`, `playbooks/svos-front-caddy-task-reports.yaml`); probe 401 from the kernel, `/auth/nope` still 404. ⚠ **When SVOS adds a host->kernel endpoint, this allowlist must grow with it.** **VERIFIED 0116:** Prime's relaunch `task:f0e9cc59…` ran hermes/gen-large 28 s, exit 0, check exit 0, settled ok, and produced a 1-page PDF. The result read then 403'd for lack of `prime → app:app-terminal read` (artifact resolve dispatches through app-terminal's read verb); grant 267 was approved 0118:44 and also covers the /tasks scan step.
 - **[2026-10-19] NH3 WAN outage ~0351–0825 (hosts did NOT reboot; WAN IP unchanged 70.230.226.88).** After it, these were re-run and came back clean: headscale-ddns, nemi-vault-pull, claude-memory-sync, ha-backup. backup-freshness reported DEGRADED, which led to: ⚠ **nh3-svos 10-18 21:30 fs-freeze FAILED because the v0.0.16 deploy left `50-svos.pre-judge` (0755) IN `/etc/qemu/fsfreeze-hook.d/`**. qemu's hook runs EVERY executable there (only .bak/.orig/~/.dpkg-* are skipped; a dir passes `-x` too), so the copy ran as a second freeze and refused. Moved to `/root/fsfreeze-hook-retired/`, __pycache__ removed at 0838; infra-hermes' live freeze at 0838:38 ran clean (only 50-svos). **Never back a hook up inside a hook dir.** Tonight's 21:30 is the real check.
 - **althing-core 3.7.1 on nh3-dev since 10-19 1004** (infra-hermes' herald stop condition: a re-poke rung >= 1 asks the post office and is cancelled ONLY on an answered unread 0; fails open on any other answer). That fixes bil-smithy-dev's phantom pokes. Installed with `uv pip --no-deps` into the uv tool env, so the deps are unchanged, but ⚠ the tool receipt still names the 3.7.0 /tmp wheel; the next `uv tool install` should use `~/.local/share/althing-releases/<ver>/`. Rollback wheel: `~/.local/share/althing-releases/3.7.0/`. althing `master` + tag v3.7.1 are LOCAL; the push is Prime's call. nh3-extdev's system wheel is still 3.7.0.
- **SVOS v0.0.27 (d5e15c3) LIVE since 10-19 0954** (interface 0.0.9 restarted, no card; UI 0.0.32). Processes: kernel/knowledge/voice 2f29d0d code, judge d14dc94, interface d5e15c3. Before: v0.0.26 (a83c223) UI 0.0.30 since 0903 (UI-only: the flows' cadence). Before: v0.0.25 (f40bf6d) UI 0.0.29 since 0855. Processes on 2f29d0d (kernel/knowledge/voice) + d14dc94 (judge/interface) code. Before: v0.0.24 (d14dc94) since 10-19 0841 (U22 card summaries: judge 0.0.2 explain, interface 0.0.8 summaries, UI 0.0.28; 4 cards approved by Prime 0839–0841; grants 294/295). → servers/nh3-svos/README.md § Release d14dc94. Before: v0.0.22 (2f29d0d) UI 0.0.26 since 10-19 0125 (task result files by path). Before: v0.0.21 (45d3a48) UI 0.0.25 since 10-19 0104 (UI-only: the /tasks draft survives a hidden tab). Processes still on 15c2d04 (kernel/knowledge/judge/voice) and 6d4cb1c (interface) code — keep those dirs. Before it: v0.0.20 (81cc5a2) UI 0.0.24 since 10-19 0055 (UI-only again: `stronger` look by default; the /tasks draft waits 125 s, after a 10 s browser abort discarded four good drafts). Nothing restarted. → servers/nh3-svos/README.md § Release 81cc5a2. Before it:  (UI-only: U21 working look behind `?look=strong|stronger`, default unchanged; Prime picks, a later patch sets the default). Nothing restarted: the processes run 15c2d04 (kernel/knowledge/judge/voice) and 6d4cb1c (interface) code — keep both dirs. → servers/nh3-svos/README.md § Release 91e9f2d.
+- **SVOS v0.0.28 (d6298f7) UI 0.0.33 LIVE since 10-19 1032** (UI-only: U24 the 3-D tree). Before: v0.0.27 (d5e15c3) since 0954, whose notes follow. ⟶  (interface 0.0.9 restarted, no card; UI 0.0.32). Processes: kernel/knowledge/voice 2f29d0d code, judge d14dc94, interface d5e15c3. Before: v0.0.26 (a83c223) UI 0.0.30 since 0903 (UI-only: the flows' cadence). Before: v0.0.25 (f40bf6d) UI 0.0.29 since 0855. Processes on 2f29d0d (kernel/knowledge/voice) + d14dc94 (judge/interface) code. Before: v0.0.24 (d14dc94) since 10-19 0841 (U22 card summaries: judge 0.0.2 explain, interface 0.0.8 summaries, UI 0.0.28; 4 cards approved by Prime 0839–0841; grants 294/295). → servers/nh3-svos/README.md § Release d14dc94. Before: v0.0.22 (2f29d0d) UI 0.0.26 since 10-19 0125 (task result files by path). Before: v0.0.21 (45d3a48) UI 0.0.25 since 10-19 0104 (UI-only: the /tasks draft survives a hidden tab). Processes still on 15c2d04 (kernel/knowledge/judge/voice) and 6d4cb1c (interface) code — keep those dirs. Before it: v0.0.20 (81cc5a2) UI 0.0.24 since 10-19 0055 (UI-only again: `stronger` look by default; the /tasks draft waits 125 s, after a 10 s browser abort discarded four good drafts). Nothing restarted. → servers/nh3-svos/README.md § Release 81cc5a2. Before it:  (UI-only: U21 working look behind `?look=strong|stronger`, default unchanged; Prime picks, a later patch sets the default). Nothing restarted: the processes run 15c2d04 (kernel/knowledge/judge/voice) and 6d4cb1c (interface) code — keep both dirs. → servers/nh3-svos/README.md § Release 91e9f2d.
 - **SVOS v0.0.18 (6d4cb1c, U20 settings) LIVE since 10-18 2352** (thread `01M59D7NTZNN75SRP0M4ZQF91A`): infra-ops pushed svoperatingsystem main + tag on Prime's in-session "push it" (his own push never landed). Interface 0.0.7 + UI 0.0.22; only svos-interface restarted; card approved by Prime 2352:16 (via Matrix), interface down 30 s. ⚠ The kernel process still runs 15c2d04's code and reports 0.0.17 until its next restart (identical code). Config gen `d750257258f9fb68` since 10-19 0101 (`[[seats.repos]]` + repo-albok 0016, + repo-dread [path dreadnaught; the session handle is dread-dev, and the repo name must match the handle] and repo-svtc 0101, Prime via svos-dev; backups `instance.toml.pre-repo-albok`, `.pre-repo-dread-svtc`). Both `ns:interface-settings` grants are IN (issued by infra-ops as Prime on his word, approved by him 2356; grant ids 207/208). **Open, Prime's:** the /settings and panel checks plus his listening test. Details + rollback: servers/nh3-svos/README.md § Release 6d4cb1c.
 - **SVOS (before v0.0.17): v0.0.16 `b51aa5e` on nh3-svos (10-18 1634–2254)** with app-judge (drafting gen-large; six local task models; `[tasks].judge` on, gen `13241a40`); nh3-dev on `3726d73`. Card audio VERIFIED 10-18 (3/3, 4.5–5.5 s). voice.toml `utterance_seconds` was 30 until v0.0.17 (the old web client refused >30); 60 since. **Open:** one drafted scratch task from /tasks (Draft → launch → approve; expect routing, admitted, routed, launching, started) — verifies v0.0.14's gate, the hermes query-file route and U19; Prime must launch it (every spawn is an owner card). Details + rollback material: servers/nh3-svos/README.md.
 - **Worldtree:** all three on v1.0.0b195 `a416ac01c5a2` since 1112 10-16; Matrix bridge profile-gated (`COMPOSE_PROFILES=matrix` on demo + personal box .env; pinned none, renders api only). Retired-config strips DONE (demo `2240110`, personal `4d1aae2`). ⚠ Pinned's seeded host config dir `/opt/worldtree-pinned/config` is NOT tracked in vh/worldtree-instance-configs and `deploy-wt-config` refuses pinned: capture it. The daily memory gate keeps its harness scratch since 10-18 (`--keep-data`, `~/.local/state/wt-memory-gate-batch/scratch/<batch>/`, 14-day user tmpfiles); 10-18 PASS. Keep defaults `library:` (thread `01M4WA58P2M001K1420NN72WFM`).
@@ -0,0 +1,26 @@
+# nh3-svos: STAGE SVOS v0.0.28 (d6298f7) beside the live release. Installs into /opt/svos/releases/d6298f7 and runs the
+# new kernel's check-config against the live configuration. Does NOT switch current, stop or start anything.
+# svos-dev's request (thread 01M5AKFPPNEW9615GSCPSH8F0H): UI 0.0.33 (U24 the 3-D tree); kernel 0.0.28 is a
+# version bump; everything else carried from d5e15c3. Built from the LOCAL tag before the push; the publish
+# (nh3-svos-d6298f7-v0.0.28.yaml) first needs origin's v0.0.28 to name the same commit.
+steps:
+  - name: Private staging dir
+    shell: install -d -m 0700 /tmp/svos-stage-d6298f7
+  - {name: stage release, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/e0be18fd-0734-4786-b933-904f0a74d2a2/scratchpad/svos-d6298f7/svos-release-d6298f7.tar.gz, dest: /tmp/svos-stage-d6298f7/release.tar.gz, mode: "0644"}}
+  - {name: stage uv, upload: {src: /home/lkraven/bin/uv, dest: /tmp/svos-stage-d6298f7/uv, mode: "0755"}}
+  - {name: stage installer, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/e0be18fd-0734-4786-b933-904f0a74d2a2/scratchpad/svos-d6298f7/install-release.py, dest: /tmp/svos-stage-d6298f7/install-release.py, mode: "0644"}}
+  - name: Verify staged hashes
+    shell: cd /tmp/svos-stage-d6298f7 && printf '%s  %s\n' d046755fd216a74c6be4e5e80e7e66a3a8bda290357fbec3abf89583f8b800f9 release.tar.gz 83aa128da93e168283ed32cf75cd819fe17b135b3fc1f7ccb3c912eec198d9fb uv | sha256sum -c -
+    changed_when: "false"
+  - name: Install release d6298f7 beside the live one (current NOT switched)
+    sudo: true
+    shell: python3 /tmp/svos-stage-d6298f7/install-release.py
+    creates: /opt/svos/releases/d6298f7/apps-env/pyvenv.cfg
+verify:
+  - name: New kernel 0.0.28 accepts the live configuration as it is
+    sudo: true
+    shell: runuser -u svc-svos -- /opt/svos/releases/d6298f7/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos
+    changed_when: "false"
+  - name: current still points at the live release
+    shell: readlink /opt/svos/current
+    changed_when: "false"
@@ -0,0 +1,26 @@
+# nh3-svos -> SVOS v0.0.28 (d6298f7): UI-only publish. UI 0.0.33 (U24: Yggdrasil as a 3-D tree turning with the mote cloud).
+# Kernel = a version bump; no app change, no manifest revision, no card.
+# svos-dev's request, thread 01M5AKFPPNEW9615GSCPSH8F0H. PRE-REQS: the stage playbook; origin's v0.0.28 == d6298f7b465a
+# (git ls-remote on nh3-dev before this runs). current must move because svos-front-inventory generates from
+# /opt/svos/current/static; NOTHING restarts: the kernel, knowledge and voice keep running 2f29d0d's code, the judge
+# d14dc94's and the interface d5e15c3's (all identical in code to d6298f7), so keep those dirs until they restart.
+steps:
+  - name: Pre-reqs present
+    sudo: true
+    shell: test -f /opt/svos/releases/d6298f7/static/index.html && readlink /opt/svos/current
+    changed_when: "false"
+  - name: Switch current to d6298f7
+    sudo: true
+    shell: ln -sfn /opt/svos/releases/d6298f7 /opt/svos/current.new && chgrp -h svc-svos /opt/svos/current.new && mv -T /opt/svos/current.new /opt/svos/current && readlink /opt/svos/current
+  - name: Front door inventory (UI 0.0.33 live; d5e15c3 becomes previous)
+    sudo: true
+    shell: /usr/local/sbin/svos-front-inventory 2>&1 | grep -E 'GENERATED|updated|root \*|path /assets' | head -4
+verify:
+  - name: Front door inventory current; health unchanged
+    sudo: true
+    shell: /usr/local/sbin/svos-front-inventory --check && curl -s http://127.0.0.1:8400/health && echo
+    changed_when: "false"
+  - name: Which release dir each running service maps (nothing restarted)
+    sudo: true
+    shell: for u in svos-kernel svos-interface svos-knowledge svos-judge svos-voice; do pid=$(systemctl show $u -p MainPID --value); printf '%s %s\n' $u "$(grep -oE '/opt/svos/releases/[0-9a-f]+' /proc/$pid/maps | sort -u | tr '\n' ' ')"; done
+    changed_when: "false"
@@ -23,6 +23,11 @@ configuration, enrollment and acceptance.
 | App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events`. **Monitoring:** `scripts/check-backup-freshness.sh` now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict **DEGRADED, exit 4** (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script |
 | Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 |

+## Release d6298f7 / v0.0.28, U24 the 3-D tree — UI-only publish (2026-10-19 1032; Prime "ship it" via svos-dev, thread `01M5AKFPPNEW9615GSCPSH8F0H`)
+
+- The tag was already on origin. Kit `d046755f…`: kernel 0.0.28 (a version bump only) and UI 0.0.33 (`/assets/index-BmaGREHz.js`); d5e15c3 is the previous handle.
+  `current → d6298f7` plus the inventory; nothing restarted; gen `d750257258f9fb68`; health ok, 8 up. Rollback: `current → releases/d5e15c3`, then the inventory.
+
 ## Release d5e15c3 / v0.0.27, interface 0.0.9 + UI 0.0.32 (2026-10-19 0954; Prime 1005 via svos-dev, thread `01M5AH9A7QGJ018HNTSGKNGAA7`)

 - The tag was already on origin. Kit `6d3da96e…`: kernel 0.0.27 (a version bump; not restarted), app-interface 0.0.9 (the stores share `state.py`: