feat(svos): v0.0.24 live on nh3-svos — judge 0.0.2 + interface 0.0.8 (U22 card summaries), UI 0.0.28; four cards approved; grants 294/295

2026-10-19 08:42:42 -07:00
parent 4e3b3b60ea
commit b1a8952812
2 changed files with 20 additions and 1 deletions
@@ -124,7 +124,7 @@ _As of 2026-10-18 ~2125 PT. The newest subsection is first; older subsections ca
 - **SVOS v0.0.17 (15c2d04) LIVE since 10-18 2255** (Prime's "deploy now" 2235 via svos-dev, thread `01M595YP8R62Y665C38MHPH6YX`): kernel 0.0.17 + UI 0.0.21 on nh3-svos, config gen `29bc36bf074b8245` (the six runtimes' prompt/working/window), voice `utterance_seconds = 60`; nh3-dev `current → 15c2d04`, only svos-terminal 0.0.6 restarted. No card: terminal revision 6 changes only `version`, so it pinned silently. Verified: 13 `seat/activity_changed` events at 2255:13 (ops-infra working = the positive control, 12 waiting; dev-ldp stays unknown, its 14-day-old claude matches neither mark). Not verified by infra-ops: GET /seats (needs a principal) and the hero "Fleet · working · N" (Prime's eyes). Rollback: servers/nh3-svos/README.md § Release 15c2d04.
 - **SVOS task launches were broken since v0.0.11 by OUR front door, FIXED 10-19 0118:** the kernel-origin Caddy (`svos-kernel.phasefinal.com`, path allowlist from U13 B2) never admitted the seat host's U15 reports (POST /tasks/start|progress|checking|exit), so every launch died `start_unreported` ("seat launch failed"). Prime's first two tasks (0110/0111) failed that way. Paths added (`configs/svos-front/Caddyfile`, `playbooks/svos-front-caddy-task-reports.yaml`); probe 401 from the kernel, `/auth/nope` still 404. ⚠ **When SVOS adds a host->kernel endpoint, this allowlist must grow with it.** **VERIFIED 0116:** Prime's relaunch `task:f0e9cc59…` ran hermes/gen-large 28 s, exit 0, check exit 0, settled ok, and produced a 1-page PDF. The result read then 403'd for lack of `prime → app:app-terminal read` (artifact resolve dispatches through app-terminal's read verb); grant 267 was approved 0118:44 and also covers the /tasks scan step.
 - **[2026-10-19] NH3 WAN outage ~0351–0825 (hosts did NOT reboot; WAN IP unchanged 70.230.226.88).** After it, these were re-run and came back clean: headscale-ddns, nemi-vault-pull, claude-memory-sync, ha-backup. backup-freshness reported DEGRADED, which led to: ⚠ **nh3-svos 10-18 21:30 fs-freeze FAILED because the v0.0.16 deploy left `50-svos.pre-judge` (0755) IN `/etc/qemu/fsfreeze-hook.d/`**. qemu's hook runs EVERY executable there (only .bak/.orig/~/.dpkg-* are skipped; a dir passes `-x` too), so the copy ran as a second freeze and refused. Moved to `/root/fsfreeze-hook-retired/`, __pycache__ removed at 0838; infra-hermes' live freeze at 0838:38 ran clean (only 50-svos). **Never back a hook up inside a hook dir.** Tonight's 21:30 is the real check.
- **SVOS v0.0.22 (2f29d0d) UI 0.0.26 LIVE since 10-19 0125** (task result files by path). Before: v0.0.21 (45d3a48) UI 0.0.25 since 10-19 0104 (UI-only: the /tasks draft survives a hidden tab). Processes still on 15c2d04 (kernel/knowledge/judge/voice) and 6d4cb1c (interface) code — keep those dirs. Before it: v0.0.20 (81cc5a2) UI 0.0.24 since 10-19 0055 (UI-only again: `stronger` look by default; the /tasks draft waits 125 s, after a 10 s browser abort discarded four good drafts). Nothing restarted. → servers/nh3-svos/README.md § Release 81cc5a2. Before it:  (UI-only: U21 working look behind `?look=strong|stronger`, default unchanged; Prime picks, a later patch sets the default). Nothing restarted: the processes run 15c2d04 (kernel/knowledge/judge/voice) and 6d4cb1c (interface) code — keep both dirs. → servers/nh3-svos/README.md § Release 91e9f2d.
+- **SVOS v0.0.24 (d14dc94) LIVE since 10-19 0841** (U22 card summaries: judge 0.0.2 explain, interface 0.0.8 summaries, UI 0.0.28; 4 cards approved by Prime 0839–0841; grants 294/295). → servers/nh3-svos/README.md § Release d14dc94. Before: v0.0.22 (2f29d0d) UI 0.0.26 since 10-19 0125 (task result files by path). Before: v0.0.21 (45d3a48) UI 0.0.25 since 10-19 0104 (UI-only: the /tasks draft survives a hidden tab). Processes still on 15c2d04 (kernel/knowledge/judge/voice) and 6d4cb1c (interface) code — keep those dirs. Before it: v0.0.20 (81cc5a2) UI 0.0.24 since 10-19 0055 (UI-only again: `stronger` look by default; the /tasks draft waits 125 s, after a 10 s browser abort discarded four good drafts). Nothing restarted. → servers/nh3-svos/README.md § Release 81cc5a2. Before it:  (UI-only: U21 working look behind `?look=strong|stronger`, default unchanged; Prime picks, a later patch sets the default). Nothing restarted: the processes run 15c2d04 (kernel/knowledge/judge/voice) and 6d4cb1c (interface) code — keep both dirs. → servers/nh3-svos/README.md § Release 91e9f2d.
 - **SVOS v0.0.18 (6d4cb1c, U20 settings) LIVE since 10-18 2352** (thread `01M59D7NTZNN75SRP0M4ZQF91A`): infra-ops pushed svoperatingsystem main + tag on Prime's in-session "push it" (his own push never landed). Interface 0.0.7 + UI 0.0.22; only svos-interface restarted; card approved by Prime 2352:16 (via Matrix), interface down 30 s. ⚠ The kernel process still runs 15c2d04's code and reports 0.0.17 until its next restart (identical code). Config gen `d750257258f9fb68` since 10-19 0101 (`[[seats.repos]]` + repo-albok 0016, + repo-dread [path dreadnaught; the session handle is dread-dev, and the repo name must match the handle] and repo-svtc 0101, Prime via svos-dev; backups `instance.toml.pre-repo-albok`, `.pre-repo-dread-svtc`). Both `ns:interface-settings` grants are IN (issued by infra-ops as Prime on his word, approved by him 2356; grant ids 207/208). **Open, Prime's:** the /settings and panel checks plus his listening test. Details + rollback: servers/nh3-svos/README.md § Release 6d4cb1c.
 - **SVOS (before v0.0.17): v0.0.16 `b51aa5e` on nh3-svos (10-18 1634–2254)** with app-judge (drafting gen-large; six local task models; `[tasks].judge` on, gen `13241a40`); nh3-dev on `3726d73`. Card audio VERIFIED 10-18 (3/3, 4.5–5.5 s). voice.toml `utterance_seconds` was 30 until v0.0.17 (the old web client refused >30); 60 since. **Open:** one drafted scratch task from /tasks (Draft → launch → approve; expect routing, admitted, routed, launching, started) — verifies v0.0.14's gate, the hermes query-file route and U19; Prime must launch it (every spawn is an owner card). Details + rollback material: servers/nh3-svos/README.md.
 - **Worldtree:** all three on v1.0.0b195 `a416ac01c5a2` since 1112 10-16; Matrix bridge profile-gated (`COMPOSE_PROFILES=matrix` on demo + personal box .env; pinned none, renders api only). Retired-config strips DONE (demo `2240110`, personal `4d1aae2`). ⚠ Pinned's seeded host config dir `/opt/worldtree-pinned/config` is NOT tracked in vh/worldtree-instance-configs and `deploy-wt-config` refuses pinned: capture it. The daily memory gate keeps its harness scratch since 10-18 (`--keep-data`, `~/.local/state/wt-memory-gate-batch/scratch/<batch>/`, 14-day user tmpfiles); 10-18 PASS. Keep defaults `library:` (thread `01M4WA58P2M001K1420NN72WFM`).
@@ -23,6 +23,25 @@ configuration, enrollment and acceptance.
 | App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events`. **Monitoring:** `scripts/check-backup-freshness.sh` now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict **DEGRADED, exit 4** (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script |
 | Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 |
 ## Release d14dc94 / v0.0.24, U22 card summaries (2026-10-19 0838–0841; Prime's go 0345 via svos-dev, thread `01M59P4Z1VJ4WHJB0N8N1P743K`, delivered at 0836 after the NH3 WAN outage)
 - v0.0.23 (9a0eec8) was superseded before any push. Its staged dir `/opt/svos/releases/9a0eec8` is unused.
 - **Kit** `4259f7f9…`: kernel 0.0.24 (a version bump; no kernel code change since 2f29d0d), app-judge 0.0.2 (U22 `explain`), app-interface 0.0.8 (U22
  `summaries`/`summary`, namespace `interface-summaries`), UI 0.0.28 (`/assets/index-83rJZtc6.js`). Everything else carried from 2f29d0d.
 - **Steps:**
  1. `nh3-svos-d14dc94-v0.0.24.yaml`: backup `pre-d14dc94` (svos.db, briefs, pins, settings; integrity ok); judge.toml `explain_seconds = 60`
     (backup `judge.toml.pre-d14dc94`; canonical `configs/svos-apps/judge.toml`); `current → d14dc94`; front door; restart ONLY svos-judge.
     **Card 1** (`card:8bd70e4a…`, app-judge rev 2) was raised 0838:44 and approved 0839:12.
  2. `…-interface.yaml` (refuses until the judge's revision is pinned): restart ONLY svos-interface. **Card 2** (`card:5fc19cd9…`, app-interface rev 6) was
     raised 0839:43 and approved 0839:54. `summaries.sqlite3` was created.
  3. Grants were raised as Prime over loopback, as before: **cards 3/4** were raised 0840:51 and approved 0841:12/0841:16. Grant 294 = `prime → ns:interface-summaries`
     read (not autonomous); 295 = write (autonomous).
 - ⚠ infra-hermes ran a manual fsfreeze/thaw at 0838:38–40, in the middle of step 1. It stopped and restarted every SVOS service. The thaw brought them up from 2f29d0d
  (before the switch), so the interface was still on 0.0.7 and no extra card was raised. The kernel, knowledge and voice now run 2f29d0d code (identical).
  Manual freezes on nh3-svos are ask-first from now on.
 - Health ok, 8 up, gen `d750257258f9fb68`. Rollback: the v0.0.22 kit (`current → 2f29d0d`, inventory, restart judge + interface: each raises its
  card again). The summaries store is additive.
 ## Release 2f29d0d / v0.0.22, task result files listed by path — UI-only publish (2026-10-19 0125; svos-dev thread `01M59M4T3ZWS0WKQJWZ9RE86B4`)
 - Prime pushed the tag `v0.0.22` → `2f29d0d667d6`. The watcher saw it at 0125:01 and published: `current → 2f29d0d` plus the front-door inventory. Nothing restarted.