vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(svos): v0.0.24 live on nh3-svos — judge 0.0.2 + interface 0.0.8 (U22 card summaries), UI 0.0.28; four cards approved; grants 294/295

Browse Source
This commit is contained in:
vh
2026-10-19 08:42:42 -07:00
parent 4e3b3b60ea
commit b1a8952812
2 changed files with 20 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
servers/nh3-svos/README.md
+19
View File
@@ -23,6 +23,25 @@ configuration, enrollment and acceptance.
| App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events`. **Monitoring:** `scripts/check-backup-freshness.sh` now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict **DEGRADED, exit 4** (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script |
| Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 |
## Release d14dc94 / v0.0.24, U22 card summaries (2026-10-19 0838–0841; Prime's go 0345 via svos-dev, thread `01M59P4Z1VJ4WHJB0N8N1P743K`, delivered at 0836 after the NH3 WAN outage)
- v0.0.23 (9a0eec8) was superseded before any push. Its staged dir `/opt/svos/releases/9a0eec8` is unused.
- **Kit** `4259f7f9…`: kernel 0.0.24 (a version bump; no kernel code change since 2f29d0d), app-judge 0.0.2 (U22 `explain`), app-interface 0.0.8 (U22
`summaries`/`summary`, namespace `interface-summaries`), UI 0.0.28 (`/assets/index-83rJZtc6.js`). Everything else carried from 2f29d0d.
- **Steps:**
1. `nh3-svos-d14dc94-v0.0.24.yaml`: backup `pre-d14dc94` (svos.db, briefs, pins, settings; integrity ok); judge.toml `explain_seconds = 60`
(backup `judge.toml.pre-d14dc94`; canonical `configs/svos-apps/judge.toml`); `current → d14dc94`; front door; restart ONLY svos-judge.
**Card 1** (`card:8bd70e4a…`, app-judge rev 2) was raised 0838:44 and approved 0839:12.
2. `…-interface.yaml` (refuses until the judge's revision is pinned): restart ONLY svos-interface. **Card 2** (`card:5fc19cd9…`, app-interface rev 6) was
raised 0839:43 and approved 0839:54. `summaries.sqlite3` was created.
3. Grants were raised as Prime over loopback, as before: **cards 3/4** were raised 0840:51 and approved 0841:12/0841:16. Grant 294 = `prime → ns:interface-summaries`
read (not autonomous); 295 = write (autonomous).
- ⚠ infra-hermes ran a manual fsfreeze/thaw at 0838:38–40, in the middle of step 1. It stopped and restarted every SVOS service. The thaw brought them up from 2f29d0d
(before the switch), so the interface was still on 0.0.7 and no extra card was raised. The kernel, knowledge and voice now run 2f29d0d code (identical).
Manual freezes on nh3-svos are ask-first from now on.
- Health ok, 8 up, gen `d750257258f9fb68`. Rollback: the v0.0.22 kit (`current → 2f29d0d`, inventory, restart judge + interface: each raises its
card again). The summaries store is additive.
## Release 2f29d0d / v0.0.22, task result files listed by path — UI-only publish (2026-10-19 0125; svos-dev thread `01M59M4T3ZWS0WKQJWZ9RE86B4`)
- Prime pushed the tag `v0.0.22` → `2f29d0d667d6`. The watcher saw it at 0125:01 and published: `current → 2f29d0d` plus the front-door inventory. Nothing restarted.