From 9c9d6a777ab0ab93fe4e489f1b64225861795e63 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 18 Oct 2026 16:42:06 -0700 Subject: [PATCH] feat(svos): [tasks].judge = app-judge after Prime's grants; card audio verified (3/3 in 4.5-5.5 s) --- persistent-memory.md | 2 +- .../svos-releases/nh3-svos-tasks-judge.yaml | 36 +++++++++++++++++++ servers/nh3-svos/README.md | 24 +++++++++++++ 3 files changed, 61 insertions(+), 1 deletion(-) create mode 100644 playbooks/svos-releases/nh3-svos-tasks-judge.yaml diff --git a/persistent-memory.md b/persistent-memory.md index 0d1cdde..387104b 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -121,7 +121,7 @@ _As of 2026-10-14 ~0005 PT. The newest subsection is first; older subsections ca ### 2026-10-14: live now (as of 2026-10-14 ~0005 PT) - **Nothing is mid-action.** No ops-log claims held. eshpfi `main` is pushed through `d1192ca` (infra-ops has STANDING push on main since 2026-10-13, CLAUDE.md § Working rules); this snapshot's commit follows and is pushed. ⚠ The working tree and index are shared with infra-hermes, albok-dev and subagents: commit with `git commit -- ` and read `git log origin/main..main` before pushing. `graphify-out/GRAPH_REPORT.md` stays modified on purpose. -- **SVOS: v0.0.15 `57d90d6` on nh3-svos since 2026-10-17 1101** (kernel 0.0.15, the short spoken summons with no ids; voice 0.0.2; `utterance_seconds = 60`; `[notifications.matrix] request_timeout_seconds = 30`; config gen `40f469b75b5d0782`, schema 14). It sits on **v0.0.14 `3726d73` (0908–0910)**: kernel 0.0.14, migration 0014; terminal 0.0.5 with `scan_patterns`; interface 0.0.6 with briefs v2; both cards approved by Prime 0909. nh3-dev `current → 3726d73`, and only svos-terminal was restarted. **Hermes task runtime on the query-file route** (Prime 10-17; backup `instance.toml.pre-queryfile`). ⚠ **Two checks still open:** (1) one scratch task from /tasks with check `exit 0`; Prime must launch it, since every spawn is an owner card. It verifies the success gate and the query route: task.md 0400 under the tasks root, the exit report's first_line, check.exit.code 0. (2) card audio on the FIRST card after 1101. The watcher saw no card through 10-18 0303. Read `audio_finished` + synthesis seconds; before the fix, all 12 failures took exactly 15.0 s. Rollback material: `backup/pre-3726d73`, `backup/pre-57d90d6`, `instance.toml.pre-{3726d73,57d90d6,queryfile,matrix-timeout}`, `voice.toml.pre-57d90d6`, nh3-dev `terminal.toml.pre-3726d73`. Details: servers/nh3-svos/README.md. (Before: main `33a7077` (v0.0.13+2: kernel 0.0.13, interface 0.0.5 with briefs, postoffice 0.0.3, UI 0.0.18) since 2026-10-14 1108–1111; config_gen `33f001f4e2dbc7dc`, apps up 7.) ⚠ Lesson (paid 10-14): a manifest revision whose verbs change is HELD for the owner, and the app reads DOWN (`refused wrong-revision`) until Prime approves its card (15-min expiry). Roll such apps only while he is at the board, and the post-office bridge last (held, it carries no Miranda mail). The interface unit needs the drop-in `svos-interface.service.d/20-artifact-host.conf` (`SVOS_ARTIFACT_HOST=nh3-svos`), and the briefs mapping host must equal it. Before that, kernel 0.0.12 (`d95078c`, 0416) on top of v0.0.11 (452a592, U15 task seats slice 1). The `hermes` runtime is the only task-capable one (one-off `hermes -z`, `process = "python3"`, no `--yolo`: Prime's call); its old argv is in `instance.toml.pre-hermes-task`. Earlier: v0.0.11 is live on nh3-svos (schema 13, config_gen `5ade279d2fb2dd1c`, apps up 7) and nh3-dev (`/opt/svos/current → releases/452a592`; svos-terminal 0.0.4 runs from it). svos-dev accepted it. The task command is ruled with svos (Prime, 2026-10-14): infra-ops edits runtimes only at svos-dev's ask. `GET /tasks` as Prime is his acceptance check. Static-only interface releases since: UI 0.0.14 `89dad7a`, 0.0.15 `7f95e54` (the web build carried into 452a592). ⚠ Process→release map: nh3-svos kernel + local apps run `33a7077`; on nh3-dev svos-postoffice runs `33a7077`, svos-terminal `452a592`; svos-miranda runs `adc862e`, svos-agent still runs `dc241f8` (check `/proc//maps` before deleting any release dir). Rollback material: nh3-svos `/var/lib/svos/backup/pre-452a592/` (schema-12 DB + config tar), `instance.toml.pre-452a592`, `instance.toml.pre-a171`; nh3-dev `terminal.toml.pre-452a592`. U13 history and the static-release chain: → `persistent-memory.d/2026-10-11-u13-cutover-executed.md`. Phase R (24 h soak, High Seat removal) is highseat-dev's. After the soak: nh3-svos releases `6c6abbe`, `a737a06`, `b588510`, `afab123`, `adc862e` and nh3-dev `/opt/svos/releases/afab123` can go once no process maps them. +- **SVOS: v0.0.16 `b51aa5e` on nh3-svos since 2026-10-18 1634, with app-judge (drafting gen-large; six local task models; `[tasks].judge` on since 1642, gen `13241a40`). Card audio VERIFIED 1640 (3/3 accepted, 4.5–5.5 s).** Open: one drafted scratch task from /tasks (Draft → launch → approve; expect events routing, admitted, routed, launching, started), which also verifies v0.0.14's gate. Before that, **v0.0.15 `57d90d6` on nh3-svos since 2026-10-17 1101** (kernel 0.0.15, the short spoken summons with no ids; voice 0.0.2; `utterance_seconds = 60`; `[notifications.matrix] request_timeout_seconds = 30`; config gen `40f469b75b5d0782`, schema 14). It sits on **v0.0.14 `3726d73` (0908–0910)**: kernel 0.0.14, migration 0014; terminal 0.0.5 with `scan_patterns`; interface 0.0.6 with briefs v2; both cards approved by Prime 0909. nh3-dev `current → 3726d73`, and only svos-terminal was restarted. **Hermes task runtime on the query-file route** (Prime 10-17; backup `instance.toml.pre-queryfile`). ⚠ **Two checks still open:** (1) one scratch task from /tasks with check `exit 0`; Prime must launch it, since every spawn is an owner card. It verifies the success gate and the query route: task.md 0400 under the tasks root, the exit report's first_line, check.exit.code 0. (2) card audio: VERIFIED 10-18 1640. Rollback material: `backup/pre-3726d73`, `backup/pre-57d90d6`, `instance.toml.pre-{3726d73,57d90d6,queryfile,matrix-timeout}`, `voice.toml.pre-57d90d6`, nh3-dev `terminal.toml.pre-3726d73`. Details: servers/nh3-svos/README.md. (Before: main `33a7077` (v0.0.13+2: kernel 0.0.13, interface 0.0.5 with briefs, postoffice 0.0.3, UI 0.0.18) since 2026-10-14 1108–1111; config_gen `33f001f4e2dbc7dc`, apps up 7.) ⚠ Lesson (paid 10-14): a manifest revision whose verbs change is HELD for the owner, and the app reads DOWN (`refused wrong-revision`) until Prime approves its card (15-min expiry). Roll such apps only while he is at the board, and the post-office bridge last (held, it carries no Miranda mail). The interface unit needs the drop-in `svos-interface.service.d/20-artifact-host.conf` (`SVOS_ARTIFACT_HOST=nh3-svos`), and the briefs mapping host must equal it. Before that, kernel 0.0.12 (`d95078c`, 0416) on top of v0.0.11 (452a592, U15 task seats slice 1). The `hermes` runtime is the only task-capable one (one-off `hermes -z`, `process = "python3"`, no `--yolo`: Prime's call); its old argv is in `instance.toml.pre-hermes-task`. Earlier: v0.0.11 is live on nh3-svos (schema 13, config_gen `5ade279d2fb2dd1c`, apps up 7) and nh3-dev (`/opt/svos/current → releases/452a592`; svos-terminal 0.0.4 runs from it). svos-dev accepted it. The task command is ruled with svos (Prime, 2026-10-14): infra-ops edits runtimes only at svos-dev's ask. `GET /tasks` as Prime is his acceptance check. Static-only interface releases since: UI 0.0.14 `89dad7a`, 0.0.15 `7f95e54` (the web build carried into 452a592). ⚠ Process→release map: nh3-svos kernel + local apps run `33a7077`; on nh3-dev svos-postoffice runs `33a7077`, svos-terminal `452a592`; svos-miranda runs `adc862e`, svos-agent still runs `dc241f8` (check `/proc//maps` before deleting any release dir). Rollback material: nh3-svos `/var/lib/svos/backup/pre-452a592/` (schema-12 DB + config tar), `instance.toml.pre-452a592`, `instance.toml.pre-a171`; nh3-dev `terminal.toml.pre-452a592`. U13 history and the static-release chain: → `persistent-memory.d/2026-10-11-u13-cutover-executed.md`. Phase R (24 h soak, High Seat removal) is highseat-dev's. After the soak: nh3-svos releases `6c6abbe`, `a737a06`, `b588510`, `afab123`, `adc862e` and nh3-dev `/opt/svos/releases/afab123` can go once no process maps them. - **Decision seat = h2o-lightning-4b (production, Prime 1805 10-13: fix forward, no INC fallback).** The one open margin is the judge's personal-question reading (h2o 18 vs INC 6 on fleet/memory); albok-dev re-fits it forward in small batches on the live seat. - **albok note intake (Prime 1530 10-13):** gate 2 DONE (wing `personal/inbox`, `/srv/nemi/inbox` on nh3-dev, nemi token `01M4VX207PJ09ZSECASCSZPCFE`). Gate 1 (ruling 6) DONE 2026-10-15 0140: Prime's notes are `/home/lkraven/kb` = gitea `vh/VuongKB` (he syncs them with git, not Syncthing), so Nemi's copy is a fresh clone at `/srv/nemi/vault`, pulled hourly at :50 by the lkraven user timer `nemi-vault-pull` (`configs/nemi-vault/`). It holds 129 md outside dot-dirs; the untracked Muninn output (`main/`, `fiction/`) is not in it. **Walked 0143–0147 10-15:** 232 objects (129 notes + 103 children), 2 judged personal, 0 adult. The nemi token was re-minted again (`01M4ZBJG3DMRF9Q9XEVZ8Q20GH`, + personal/notes) and the old one revoked (thread `01M4VWV5FEF64EP71W60N6Q1N0`). Prime says the kb needs cleaning up (its last commit is 2026-04-16). - **Worldtree:** demo was deploying `beba08a4` (albok 4.7 the rest; Worldtree is a pure albok-service HTTP customer) at 2120 10-13. Retired-config strip: **demo DONE 2026-10-15 2339 on Prime's go** (instance-configs `2240110`: providers `graph_store` + `library`, plus the b193-retired `memory.*` keys; the boot shows 0 retired-key warnings, down from 7). (0823 10-16: all three moved to v1.0.0b194 `daf587396f7a`; pinned via Deploy-pinned 11136, from its May 446e5807 build.) **Personal's strip DONE 2026-10-16 1036 on Prime's yes** (instance-configs `4d1aae2`, the same patch as demo; boot shows 0 retired-key warnings, down from 7). **The Matrix bridge becomes profile-gated** in worldtree-dev's next release: `COMPOSE_PROFILES=matrix` is set on the demo and personal box .env (and in env.public, `5bc796a`); pinned sets NOTHING. **All three on v1.0.0b195 (`a416ac01c5a2`) since 1112 10-16** (demo/personal via CI 11145/11146, pinned via Deploy-pinned 11147). Pinned renders api only, and its dead bridge container was removed. The image no longer carries KB_PATH. ⚠ Pinned's new compose bind-mounts `/opt/worldtree-pinned/config`: the entrypoint SEEDED it from the image's config-defaults on first start, so pinned now HAS a host config dir that vh/worldtree-instance-configs does not yet track (pinned was out of scope while its config was baked in). Capture it. Keep defaults `library:` (thread `01M4WA58P2M001K1420NN72WFM`). Daily memory gate: 10-13 re-run `20261013T160414Z` PASS; worldtree-dev owns the tagger-latch fix. muninn-gate + mimir-inbox are STOPPED. **Albok unit 6 teardown (thread `01M4WAEFZEA1YY03ZR31AEHGTS`): steps 1-4 DONE 2026-10-16 0836.** 8 tars (3 worldtree-kb volumes, mimir-inbox staging, personal `state/ingestion`, muninn-gate conf, both compose dirs) were test-restored with per-file sha256 identical, and are now on `nh3-nas:/volume1/Backup/retired/albok-u6-corviduo-dev-20261016` (0700, re-verified there; local copy `/var/tmp/albok-u6`). **Deletion (step 5) is HELD for Prime's direct go**: albok-dev relayed "land all of u6", and a relayed go does not authorize a delete. The images muninn-gate:0.0.14 + mimir-inbox:0.0.1 were already gone at 0830, cause unknown. The in-process ingestion watcher ran until the b194 deploy (last heartbeat 0821:45). worldtree-dev cleared the ingestion subtree from the b194 source (0845). I removed personal .env's no-op `COMPOSE_PROFILES=muninn` (0846). The two 07-27 `.bak-muninn` files go in step 5 with no archive (the .env one holds live secrets). personal's providers.yaml `write_agents: ["muninn"]` goes with personal's retired-config strip (on Prime's yes). **albok-service 0.1.24 (sha256:75be71c1) LIVE since 0917 on nh3-albok, with personal/sealed configured (8 wings ready).** Harness (a) PASSED 0922: LOCK 0920:08-0920:38 (423s, search `locked`), live UNLOCK 0921:53 with no restart; the synthetic object was hard-deleted. The sealed wing stays MOUNTED. **Step 5 DONE 2026-10-16 1031 on Prime's direct go:** the 3 worldtree-kb volumes, personal `state/ingestion`, the muninn/mimir dirs and staging, and the 2 .bak files are all deleted. I also pruned the build cache (345.7MB) and removed /var/tmp/albok-u6; the NAS archive is the only copy. Found during the readback: **pinned's Matrix bridge had crash-looped since the 0822 b194 deploy** (no matrix.yaml, empty MATRIX tokens: pinned never had a bridge). I stopped it at 1033. The next pinned deploy recreates it; the durable fix is worldtree-dev's compose. diff --git a/playbooks/svos-releases/nh3-svos-tasks-judge.yaml b/playbooks/svos-releases/nh3-svos-tasks-judge.yaml new file mode 100644 index 0000000..223d0d5 --- /dev/null +++ b/playbooks/svos-releases/nh3-svos-tasks-judge.yaml @@ -0,0 +1,36 @@ +# nh3-svos: [tasks].judge = "app-judge" (U19 step 4), after Prime's three owner grants landed (1641 2026-10-18: +# app-judge -> app:app-interface read, app-judge -> app:app-terminal read, prime -> app:app-judge read). +# From now every root task's approval runs the judge's admit before any seat exists. judge_timeout_seconds stays at its +# default 10 (the app's admit_seconds 8 sits inside). Revert = instance.toml.pre-tasks-judge + reload. +steps: + - name: back up instance.toml + sudo: true + shell: runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml /var/lib/svos/config/instance.toml.pre-tasks-judge + creates: /var/lib/svos/config/instance.toml.pre-tasks-judge + - name: add judge = "app-judge" under [tasks] + sudo: true + shell: | + runuser -u svc-svos -- python3 - <<'PY' + import os + p = "/var/lib/svos/config/instance.toml"; s = open(p).read() + anchor = '[tasks]\nscratch_root = "/var/lib/svos/apps/terminal/tasks"\n' + assert s.count(anchor) == 1, s.count(anchor) + tail = s[s.index(anchor):]; nxt = tail.find("\n[", len(anchor) - 1) + assert "judge =" not in (tail if nxt < 0 else tail[:nxt]) + s = s.replace(anchor, anchor + 'judge = "app-judge" # U19 (v0.0.16, 2026-10-18): admission + routing; judge_timeout_seconds default 10\n') + t = p + ".tmp"; fd = os.open(t, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600) + with os.fdopen(fd, "w") as f: f.write(s) + os.replace(t, p); print("edited") + PY + - name: the kernel accepts it (else restore) + sudo: true + shell: | + if runuser -u svc-svos -- /opt/svos/current/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos; then exit 0; fi + runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml.pre-tasks-judge /var/lib/svos/config/instance.toml; echo RESTORED; exit 1 + - name: reload + sudo: true + shell: systemctl reload svos-kernel; sleep 3; curl -s http://127.0.0.1:8400/health +verify: + - name: health ok, nothing down, new generation + shell: h=$(curl -s http://127.0.0.1:8400/health); echo "$h"; echo "$h" | grep -q '"status":"ok"' && echo "$h" | grep -q '"down":0' && ! echo "$h" | grep -q f5330a9978aedc48 + changed_when: "false" diff --git a/servers/nh3-svos/README.md b/servers/nh3-svos/README.md index 6ec857f..61c8cc3 100644 --- a/servers/nh3-svos/README.md +++ b/servers/nh3-svos/README.md @@ -23,6 +23,30 @@ configuration, enrollment and acceptance. | App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events`. **Monitoring:** `scripts/check-backup-freshness.sh` now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict **DEGRADED, exit 4** (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script | | Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 | +## Release b51aa5e / v0.0.16, the judge app (2026-10-18 1633–1642; Prime: "go with your picks, cut over now"; svos-dev thread `01M57XAEZVQ39DKNYXRTMC29S5`) + +- **Kit:** kernel 0.0.16, app-judge 0.0.1 and UI 0.0.20 from a clean archive of the tag. The rest was carried from 57d90d6. Kit sha256 `25f78195…`. + Playbooks: `nh3-svos-b51aa5e-v0.0.16-stage.yaml`, then `…-v0.0.16.yaml`, then `nh3-svos-tasks-judge.yaml`. +- **app-judge:** + - Listens on 127.0.0.1:8409, unit `svos-judge` (`configs/svos-apps/svos-judge.service`), and is in the quiesce hook (`50-svos.pre-judge` kept). + - Config `config/apps/judge.toml` (`configs/svos-apps/judge.toml`): drafting **gen-large**; route `*` → hermes/gen-large. + - `/etc/svos/judge.env` (root 0600) holds two values: + - `SVOS_JUDGE_GATEWAY_KEY`: LiteLLM alias `svos-app-judge`, scoped to exactly gen-large, gen, gen-reasoning, gen-small, sec and + sec-reasoning; vault `nh3-svos/svos-judge-gateway-key`. + - `SVOS_APP_CREDENTIAL`: enrolled 1633 via `svos-kernel mint-secret` + `POST /auth/enroll`; vault `nh3-svos/svos-judge-credential`. + - Manifest revision 1 pinned, no card. +- **Config:** + - `[[apps]] app-judge`, and the hermes runtime gets `models = [the same six]`, which is the /tasks select ∩ the judge key's catalogue + (gen `f5330a99`). + - After Prime's three owner grants (approved 1641:04–1641:10: app-judge → app:app-interface read, app-judge → + app:app-terminal read, prime → app:app-judge read), `[tasks].judge = "app-judge"` (gen `13241a403080788c`). All 8 apps are up. +- **Card audio VERIFIED with the v0.0.15 spoken form:** the three grant cards spoke in 5.5, 4.5 and 5.4 s, all `accepted`. + Before the fix: 12 of 12 failed at 15.0 s. +- **Rollback:** + - Remove `[tasks].judge` (`instance.toml.pre-tasks-judge`), stop and disable `svos-judge`. + - For the full release: stop the apps and the kernel, restore `backup/pre-b51aa5e/svos.db` + `instance.toml.pre-b51aa5e`, + `current → releases/57d90d6`, then start. + ## Release 57d90d6 / v0.0.15, the short spoken summons (2026-10-17 1100–1102; Prime's wording ruling via svos-dev, thread `01M5558E19Q3ZK4RRZXQG6A2Y9`) - **Kit:** kernel 0.0.15 and app-voice 0.0.2 wheels from a clean archive of the tag. Everything else, the web build included, was carried byte-identical