diff --git a/persistent-memory.md b/persistent-memory.md index a7fcd8d..9aac2f3 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-10-18 ~2125 (snapshot for clear: SVOS v0.0.16 + judge live, v0.0.17 pushed not deployed; albok u6 done; Worldtree b195; asset-engine retired). Previously: 2026-10-14 (snapshot for clear, elway note updated: SVOS v0.0.11 live, pewpewstudio deployed, albok inbox live, decision seat on h2o; earlier: the h2o swap saga, U13.)_ +_Last updated: 2026-10-18 ~2300 (SVOS v0.0.17 deployed and verified by events). Previously 2026-10-18 ~2125 (snapshot for clear: SVOS v0.0.16 + judge live, v0.0.17 pushed not deployed; albok u6 done; Worldtree b195; asset-engine retired). Previously: 2026-10-14 (snapshot for clear, elway note updated: SVOS v0.0.11 live, pewpewstudio deployed, albok inbox live, decision seat on h2o; earlier: the h2o swap saga, U13.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -121,8 +121,8 @@ _As of 2026-10-18 ~2125 PT. The newest subsection is first; older subsections ca ### 2026-10-18: live now (as of 2026-10-18 ~2125 PT) - **Nothing is mid-action.** No ops-log claims held, no background watchers. eshpfi `main` is pushed (standing push since 2026-10-13); vh/worldtree-instance-configs `main` also has STANDING push since 2026-10-16 (Prime; auto-memory `feedback_git_autonomous`). On Prime's word ~2120 10-18: svoperatingsystem `main` + tag `v0.0.17` (15c2d04) and dotfiles `f909dcf` pushed. ⚠ The working tree and index are shared with infra-hermes, albok-dev and subagents: commit with `git commit -- ` and read `git log origin/main..main` before pushing. `graphify-out/GRAPH_REPORT.md` stays modified on purpose. -- **SVOS v0.0.17 (15c2d04, U17 seat activity + the voice client fix) is PUSHED but NOT DEPLOYED** — Prime's next window (svos-dev thread `01M595YP8R62Y665C38MHPH6YX`). Plan: stage kit (kernel 0.0.17, terminal 0.0.6, UI 0.0.21) on both hosts; on nh3-svos add each `[[seats.runtimes]]`' `prompt`/`working`/`window` copied VERBATIM from `docs/operations/u13-edit-payload.md` @15c2d04, check-config, kernel + UI; on nh3-dev restart svos-terminal 0.0.6 (raises ONE held-manifest card: do it only with Prime at the board); then voice.toml `utterance_seconds` back to 60 + restart svos-voice; verify GET /seats `activity`, `/events?kind=seat`, the hero "Fleet · working · N". Rollback = v0.0.16 + take the new runtime keys out (an old kernel refuses them). -- **SVOS live: v0.0.16 `b51aa5e` on nh3-svos (since 10-18 1634)** with app-judge (drafting gen-large; six local task models; `[tasks].judge` on, gen `13241a40`); nh3-dev on `3726d73`. Card audio VERIFIED 10-18 (3/3, 4.5–5.5 s). voice.toml `utterance_seconds = 30` (the old web client refused >30; fixed in v0.0.17). **Open:** one drafted scratch task from /tasks (Draft → launch → approve; expect routing, admitted, routed, launching, started) — verifies v0.0.14's gate, the hermes query-file route and U19; Prime must launch it (every spawn is an owner card). Details + rollback material: servers/nh3-svos/README.md. +- **SVOS v0.0.17 (15c2d04) LIVE since 10-18 2255** (Prime's "deploy now" 2235 via svos-dev, thread `01M595YP8R62Y665C38MHPH6YX`): kernel 0.0.17 + UI 0.0.21 on nh3-svos, config gen `29bc36bf074b8245` (the six runtimes' prompt/working/window), voice `utterance_seconds = 60`; nh3-dev `current → 15c2d04`, only svos-terminal 0.0.6 restarted. No card: terminal revision 6 changes only `version`, so it pinned silently. Verified: 13 `seat/activity_changed` events at 2255:13 (ops-infra working = the positive control, 12 waiting; dev-ldp stays unknown, its 14-day-old claude matches neither mark). Not verified by infra-ops: GET /seats (needs a principal) and the hero "Fleet · working · N" (Prime's eyes). Rollback: servers/nh3-svos/README.md § Release 15c2d04. +- **SVOS (before v0.0.17): v0.0.16 `b51aa5e` on nh3-svos (10-18 1634–2254)** with app-judge (drafting gen-large; six local task models; `[tasks].judge` on, gen `13241a40`); nh3-dev on `3726d73`. Card audio VERIFIED 10-18 (3/3, 4.5–5.5 s). voice.toml `utterance_seconds` was 30 until v0.0.17 (the old web client refused >30); 60 since. **Open:** one drafted scratch task from /tasks (Draft → launch → approve; expect routing, admitted, routed, launching, started) — verifies v0.0.14's gate, the hermes query-file route and U19; Prime must launch it (every spawn is an owner card). Details + rollback material: servers/nh3-svos/README.md. - **Worldtree:** all three on v1.0.0b195 `a416ac01c5a2` since 1112 10-16; Matrix bridge profile-gated (`COMPOSE_PROFILES=matrix` on demo + personal box .env; pinned none, renders api only). Retired-config strips DONE (demo `2240110`, personal `4d1aae2`). ⚠ Pinned's seeded host config dir `/opt/worldtree-pinned/config` is NOT tracked in vh/worldtree-instance-configs and `deploy-wt-config` refuses pinned: capture it. The daily memory gate keeps its harness scratch since 10-18 (`--keep-data`, `~/.local/state/wt-memory-gate-batch/scratch//`, 14-day user tmpfiles); 10-18 PASS. Keep defaults `library:` (thread `01M4WA58P2M001K1420NN72WFM`). - **Decision seat = h2o-lightning-4b (production, Prime 1805 10-13: fix forward, no INC fallback).** The one open margin is the judge's personal-question reading (h2o 18 vs INC 6 on fleet/memory); albok-dev re-fits it forward in small batches on the live seat. - **albok note intake (Prime 1530 10-13):** gate 2 DONE (wing `personal/inbox`, `/srv/nemi/inbox` on nh3-dev, nemi token `01M4VX207PJ09ZSECASCSZPCFE`). Gate 1 (ruling 6) DONE 2026-10-15 0140: Prime's notes are `/home/lkraven/kb` = gitea `vh/VuongKB` (he syncs them with git, not Syncthing), so Nemi's copy is a fresh clone at `/srv/nemi/vault`, pulled hourly at :50 by the lkraven user timer `nemi-vault-pull` (`configs/nemi-vault/`). It holds 129 md outside dot-dirs; the untracked Muninn output (`main/`, `fiction/`) is not in it. **Walked 0143–0147 10-15:** 232 objects (129 notes + 103 children), 2 judged personal, 0 adult. The nemi token was re-minted again (`01M4ZBJG3DMRF9Q9XEVZ8Q20GH`, + personal/notes) and the old one revoked (thread `01M4VWV5FEF64EP71W60N6Q1N0`). Prime says the kb needs cleaning up (its last commit is 2026-04-16). @@ -130,7 +130,7 @@ _As of 2026-10-18 ~2125 PT. The newest subsection is first; older subsections ca - **fv-ml1: hard hang 10-11 ~0005, self-reset, no record** (pstore/ERST registered and empty, so not a kernel panic). Evidence capture staged: kernel console on ttyS1 from its next boot + nh3-dev user unit `fv-ml1-sol-logger` (`journalctl --user -u fv-ml1-sol-logger`). ⚠ **HOLD** `zroot/ROOT/debian@pre-patch-20261008` (do not destroy) until the cause is known or a clean week passes after the last crash. - **albok `.browse/` share:** the Linux-client withdrawal test passed (10-09); vh holds every albok read group; Prime's Mac keys on vh stay full access (Prime 10-11). Owed: the macOS run, staged as vh in `~vh/albok-browse-test/` on nh3-albok, waits on Prime pasting the one-liner (`scripts/oneoff/albok-browse-withdrawal/README.md`); afterwards read `results/`, report to albok-dev, remove the replica. - **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens; jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it. It runs on the shared all-agents-local key. -- **Waiting on Prime:** one drafted scratch task on /tasks (SVOS verify); the v0.0.17 cutover window (one terminal card); satellite-1's login test + name; rotate the fv-ml1 BMC password (printed into a local transcript 10-08; infra-ops leans no); albok's 2.01× load check against a 2× bar; the 10-05 cleanup bundle (stale parks 49/46/40, fv-ml1 README stack + NVMe tables, ZED alerts into althing-alert-bridge, prune exited one-shot containers on fv-ml1); push Booth `ce095c1`; ESH Homepage HTTPS name; nh3-dev reboot (kernel 6.1.0-53). ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope. +- **Waiting on Prime:** one drafted scratch task on /tasks (SVOS verify); a look at the v0.0.17 hero "Fleet · working · N"; satellite-1's login test + name; rotate the fv-ml1 BMC password (printed into a local transcript 10-08; infra-ops leans no); albok's 2.01× load check against a 2× bar; the 10-05 cleanup bundle (stale parks 49/46/40, fv-ml1 README stack + NVMe tables, ZED alerts into althing-alert-bridge, prune exited one-shot containers on fv-ml1); push Booth `ce095c1`; ESH Homepage HTTPS name; nh3-dev reboot (kernel 6.1.0-53). ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope. - **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`. The LiteLLM compose.yaml on ana-docker differs from `stacks/litellm/compose.yaml` (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%). The ana-docker gitea-runner's Homepage name changes to its runner name at its next recreate (stacks/gitea-runner/compose.yaml). nh3-dev's `/opt/svos/releases/` and the h2o eval seat on fv-ml1 GPU 3 (`stacks/h2o-eval`, down) are kept as artifacts. - **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in. - **Older open items, one line each:** esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (`stacks/blender/README.md`). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran `51168a2e` on 09-27). nh3-dev's root disk is at ~66% since heid's runaway sweep was killed 10-08 (it flapped on the Beszel 85% line before). ESH has a single outside route (esh-scale on esh-pve), noted and untracked. diff --git a/playbooks/svos-releases/nh3-dev-svos-15c2d04-v0.0.17.yaml b/playbooks/svos-releases/nh3-dev-svos-15c2d04-v0.0.17.yaml new file mode 100644 index 0000000..c7491ac --- /dev/null +++ b/playbooks/svos-releases/nh3-dev-svos-15c2d04-v0.0.17.yaml @@ -0,0 +1,29 @@ +# nh3-dev phase of the v0.0.17 rollout: app-terminal 0.0.6 (U17: the seat host classifies each live pane working/waiting +# against the kernel's per-runtime marks; activity_interval_seconds at its default 5, no terminal.toml edit). Release +# 15c2d04 is already built beside the live one in /opt/svos/releases/15c2d04 (uv pip freeze differs from 3726d73 in +# svos-terminal only). Restarts ONLY svos-terminal; postoffice, miranda and agent keep running 3726d73's identical code until +# their next restart. Terminal's manifest changes: registration raises ONE card, and the app refuses wrong-revision until +# Prime approves it (he is at the board: Prime 2235 2026-10-18 via svos-dev, thread 01M595YP8R62Y665C38MHPH6YX). +# Run as: scripts/elway infra-ops@10.100.10.50 (the apps are lkraven USER units, driven with runuser + XDG_RUNTIME_DIR). +steps: + - name: Release 15c2d04 is built and carries terminal 0.0.6 + shell: test -f /opt/svos/releases/15c2d04/release.json && /opt/svos/releases/15c2d04/apps-env/bin/python -c "from importlib.metadata import version; v = version('svos-terminal'); print('terminal', v); assert v == '0.0.6'" + changed_when: "false" + - name: Switch current to 15c2d04 (as lkraven, who owns /opt/svos on nh3-dev) + sudo: true + shell: runuser -u lkraven -- sh -c 'ln -sfn /opt/svos/releases/15c2d04 /opt/svos/current.new && mv -T /opt/svos/current.new /opt/svos/current && readlink /opt/svos/current' + - name: Restart ONLY svos-terminal + sudo: true + shell: | + runuser -u lkraven -- env XDG_RUNTIME_DIR=/run/user/1000 systemctl --user restart svos-terminal + sleep 6; runuser -u lkraven -- env XDG_RUNTIME_DIR=/run/user/1000 systemctl --user is-active svos-terminal + pid=$(runuser -u lkraven -- env XDG_RUNTIME_DIR=/run/user/1000 systemctl --user show svos-terminal -p MainPID --value); grep -oE '/opt/svos/releases/[0-9a-f]+' /proc/$pid/maps | sort -u + runuser -u lkraven -- env XDG_RUNTIME_DIR=/run/user/1000 journalctl --user -u svos-terminal --since "-60s" --no-pager -o cat | grep -i -E "manifest|card|held|revision|activity|error" | tail -8 | cut -c1-200 +verify: + - name: Terminal runs 15c2d04 and its seat host came up + sudo: true + shell: | + pid=$(runuser -u lkraven -- env XDG_RUNTIME_DIR=/run/user/1000 systemctl --user show svos-terminal -p MainPID --value) + grep -q /opt/svos/releases/15c2d04 /proc/$pid/maps && echo "terminal pid $pid on 15c2d04" + ! runuser -u lkraven -- env XDG_RUNTIME_DIR=/run/user/1000 journalctl --user -u svos-terminal --since "-5min" --no-pager -o cat | grep -i "Seat host unavailable" + changed_when: "false" diff --git a/playbooks/svos-releases/nh3-svos-15c2d04-v0.0.17-stage.yaml b/playbooks/svos-releases/nh3-svos-15c2d04-v0.0.17-stage.yaml new file mode 100644 index 0000000..a9a68c6 --- /dev/null +++ b/playbooks/svos-releases/nh3-svos-15c2d04-v0.0.17-stage.yaml @@ -0,0 +1,37 @@ +# nh3-svos: STAGE SVOS v0.0.17 (15c2d04) beside the live release. Installs into /opt/svos/releases/15c2d04 and runs the +# new kernel's check-config against the live configuration. Does NOT switch current, stop or start anything. +# svos-dev's ask (thread 01M595YP8R62Y665C38MHPH6YX); Prime's "deploy now" 2235 2026-10-18. Kernel 0.0.17, terminal 0.0.6 +# (nh3-svos keeps it installed, unused), UI 0.0.21; everything else carried from b51aa5e. The cutover is +# nh3-svos-15c2d04-v0.0.17.yaml. The runtimes edit script is staged here too and dry-run against a copy of the live file. +steps: + - name: Private staging dir + shell: install -d -m 0700 /tmp/svos-stage-15c2d04 + - {name: stage release, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/e0be18fd-0734-4786-b933-904f0a74d2a2/scratchpad/svos-15c2d04/svos-release-15c2d04.tar.gz, dest: /tmp/svos-stage-15c2d04/release.tar.gz, mode: "0644"}} + - {name: stage uv, upload: {src: /home/lkraven/bin/uv, dest: /tmp/svos-stage-15c2d04/uv, mode: "0755"}} + - {name: stage installer, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/e0be18fd-0734-4786-b933-904f0a74d2a2/scratchpad/svos-15c2d04/install-release.py, dest: /tmp/svos-stage-15c2d04/install-release.py, mode: "0644"}} + - {name: stage runtimes edit, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/e0be18fd-0734-4786-b933-904f0a74d2a2/scratchpad/svos-15c2d04/u17-runtimes-edit.py, dest: /tmp/svos-stage-15c2d04/u17-runtimes-edit.py, mode: "0644"}} + - name: Verify staged hashes + shell: cd /tmp/svos-stage-15c2d04 && printf '%s %s\n' f6e4ff1e49f98482afad510bb591c9ebc9087ec792677b6190c1839f2939b2b2 release.tar.gz 83aa128da93e168283ed32cf75cd819fe17b135b3fc1f7ccb3c912eec198d9fb uv 53bb68360eedbd8694ea89d9212c2eea80027655991a0bc7bf690ad76d18a217 u17-runtimes-edit.py | sha256sum -c - + changed_when: "false" + - name: Install release 15c2d04 beside the live one (current NOT switched) + sudo: true + shell: python3 /tmp/svos-stage-15c2d04/install-release.py + creates: /opt/svos/releases/15c2d04/apps-env/pyvenv.cfg +verify: + - name: New kernel 0.0.17 accepts the live configuration as it is + sudo: true + shell: runuser -u svc-svos -- /opt/svos/releases/15c2d04/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos + changed_when: "false" + - name: Dry run of the runtimes edit on a private copy (the live file is not touched); the result parses as TOML + sudo: true + shell: | + set -e + d=$(mktemp -d); cp -p /var/lib/svos/config/instance.toml $d/instance.toml + sed "s#/var/lib/svos/config/instance.toml#$d/instance.toml#" /tmp/svos-stage-15c2d04/u17-runtimes-edit.py > $d/edit.py + python3 $d/edit.py + python3 -c "import tomllib,sys; c=tomllib.load(open(sys.argv[1],'rb')); r=c['seats']['runtimes']; print([(x['name'], x.get('window'), bool(x.get('prompt')), bool(x.get('working'))) for x in r])" $d/instance.toml + diff /var/lib/svos/config/instance.toml $d/instance.toml | grep -c '^>' ; rm -rf $d + changed_when: "false" + - name: current still points at the live release + shell: readlink /opt/svos/current + changed_when: "false" diff --git a/playbooks/svos-releases/nh3-svos-15c2d04-v0.0.17.yaml b/playbooks/svos-releases/nh3-svos-15c2d04-v0.0.17.yaml new file mode 100644 index 0000000..54a81db --- /dev/null +++ b/playbooks/svos-releases/nh3-svos-15c2d04-v0.0.17.yaml @@ -0,0 +1,80 @@ +# nh3-svos -> SVOS v0.0.17 (15c2d04): kernel 0.0.17 (U17 seat activity: [[seats.runtimes]] prompt/working/window, the +# row's `activity`, seat/activity_changed) and UI 0.0.21 (the working flow; the voice client admits 0.1..120 s and speaks +# long text in pieces). Prime 2235 2026-10-18 via svos-dev (thread 01M595YP8R62Y665C38MHPH6YX): "deploy v0.0.17 now". +# PRE-REQ: nh3-svos-15c2d04-v0.0.17-stage.yaml. No schema change (14). An old kernel refuses the new runtime keys, so the +# edit lands inside the stopped window: stop; backup; switch; edit (verbatim from docs/operations/u13-edit-payload.md +# @15c2d04) + check-config (restore + b51aa5e on refusal); start; front door; then voice.toml utterance_seconds 30 -> 60 +# (UI 0.0.21 is live by then) and the voice app starts last. nh3-dev's terminal 0.0.6 is a separate playbook (one card). +steps: + - name: Pre-reqs present (release staged, edit script staged, no runtime keys yet) + sudo: true + shell: test -f /opt/svos/releases/15c2d04/kernel-env/bin/svos-kernel && test -f /tmp/svos-stage-15c2d04/u17-runtimes-edit.py && ! grep -qE '^(prompt|working|window) *=' /var/lib/svos/config/instance.toml && readlink /opt/svos/current + changed_when: "false" + - name: Stop the local apps then the kernel + sudo: true + shell: systemctl stop svos-voice svos-judge svos-knowledge svos-interface svos-kernel; systemctl is-active svos-kernel svos-interface svos-knowledge svos-judge svos-voice | tr '\n' ' '; echo + - name: Offline backup to /var/lib/svos/backup/pre-15c2d04 + sudo: true + shell: | + set -e + d=/var/lib/svos/backup/pre-15c2d04 + runuser -u svc-svos -- install -d -m 0700 $d + runuser -u svc-svos -- python3 -c " + import sqlite3 + s = sqlite3.connect('/var/lib/svos/state/svos.db'); t = sqlite3.connect('$d/svos.db'); s.backup(t); t.close(); s.close() + c = sqlite3.connect('file:$d/svos.db?mode=ro', uri=True); print('integrity', c.execute('pragma integrity_check').fetchone()[0], 'user_version', c.execute('pragma user_version').fetchone()[0])" + runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml /var/lib/svos/config/instance.toml.pre-15c2d04 + runuser -u svc-svos -- cp -p /var/lib/svos/config/apps/voice.toml /var/lib/svos/config/apps/voice.toml.pre-15c2d04 + - name: Switch current to 15c2d04 + sudo: true + shell: ln -sfn /opt/svos/releases/15c2d04 /opt/svos/current.new && chgrp -h svc-svos /opt/svos/current.new && mv -T /opt/svos/current.new /opt/svos/current && readlink /opt/svos/current + - name: Config - the six runtimes' prompt/working/window (verbatim); the new kernel must accept it (else restore + b51aa5e) + sudo: true + shell: | + runuser -u svc-svos -- python3 - < /tmp/svos-stage-15c2d04/u17-runtimes-edit.py || exit 1 # stdin: svc-svos cannot read infra-ops's 0700 stage dir + if runuser -u svc-svos -- /opt/svos/releases/15c2d04/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos; then exit 0; fi + runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml.pre-15c2d04 /var/lib/svos/config/instance.toml + ln -sfn /opt/svos/releases/b51aa5e /opt/svos/current.new && chgrp -h svc-svos /opt/svos/current.new && mv -T /opt/svos/current.new /opt/svos/current + systemctl start svos-kernel.service svos-interface svos-knowledge svos-judge svos-voice; echo "RESTORED config + b51aa5e"; exit 1 + - name: Start the kernel then interface, knowledge, judge + sudo: true + shell: | + t0=$(date +%s.%N); systemctl start svos-kernel.service; rc=$?; t1=$(date +%s.%N) + echo "kernel start rc=$rc after $(python3 -c "print(round($t1-$t0,2))") s" + systemctl start svos-interface svos-knowledge svos-judge + sleep 8; curl -s http://127.0.0.1:8400/health; echo + - name: Front door inventory (UI 0.0.21 live; b51aa5e becomes previous) + sudo: true + shell: /usr/local/sbin/svos-front-inventory 2>&1 | grep -E 'GENERATED|updated|root \*|path /assets' | head -4 + - name: voice.toml utterance_seconds 30 -> 60 (the 0.0.21 client admits 0.1..120), then start the voice app + sudo: true + shell: | + set -e + runuser -u svc-svos -- python3 - <<'PY' + import os + p = "/var/lib/svos/config/apps/voice.toml"; s = open(p, encoding="utf-8").read() + old = [l for l in s.split("\n") if l.startswith("utterance_seconds = 30 ")] + assert len(old) == 1, old + new = "utterance_seconds = 60 # 2026-10-18 v0.0.17: UI 0.0.21's voice client admits 0.1..120 s (U12 v1.4); 6 MiB output holds 65 s at 48 kHz" + s = s.replace(old[0], new) + t = p + ".tmp"; fd = os.open(t, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600) + with os.fdopen(fd, "w", encoding="utf-8") as f: f.write(s) + os.replace(t, p); print("edited") + PY + grep -n '^utterance_seconds' /var/lib/svos/config/apps/voice.toml + systemctl start svos-voice; sleep 5; systemctl is-active svos-voice + journalctl -u svos-voice --since "-30s" --no-pager -o cat | grep -iE "error|refus|invalid|limit" | tail -4 | cut -c1-200 || true +verify: + - name: Health ok, nothing down + shell: h=$(curl -s http://127.0.0.1:8400/health); echo "$h"; echo "$h" | grep -q '"status":"ok"' && echo "$h" | grep -q '"down":0' + changed_when: "false" + - name: Every app and the kernel run from 15c2d04 + sudo: true + shell: | + for u in svos-kernel svos-interface svos-knowledge svos-judge svos-voice; do pid=$(systemctl show $u -p MainPID --value); printf '%s pid %s %s\n' $u $pid "$(grep -oE '/opt/svos/releases/[0-9a-f]+' /proc/$pid/maps | sort -u | tr '\n' ' ')"; done + for u in svos-kernel svos-interface svos-knowledge svos-judge svos-voice; do pid=$(systemctl show $u -p MainPID --value); grep -q /opt/svos/releases/15c2d04 /proc/$pid/maps || { echo "$u NOT on 15c2d04"; exit 1; }; done + changed_when: "false" + - name: Front door inventory current + sudo: true + shell: /usr/local/sbin/svos-front-inventory --check + changed_when: "false" diff --git a/servers/nh3-svos/README.md b/servers/nh3-svos/README.md index a6bf1c6..92e4d74 100644 --- a/servers/nh3-svos/README.md +++ b/servers/nh3-svos/README.md @@ -23,6 +23,36 @@ configuration, enrollment and acceptance. | App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events`. **Monitoring:** `scripts/check-backup-freshness.sh` now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict **DEGRADED, exit 4** (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script | | Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 | +## Release 15c2d04 / v0.0.17, seat activity (2026-10-18 2254–2255; Prime's "deploy now" 2235 via svos-dev, thread `01M595YP8R62Y665C38MHPH6YX`) + +- **Kit:** kernel 0.0.17, app-terminal 0.0.6 wheels and web build UI 0.0.21 from a clean archive of the tag. Everything else was + carried byte-identical from b51aa5e (no source change; uv.lock changed only the kernel version). Kit sha256 `f6e4ff1e…`. + `release.json` now says schema 14 (the carried value said 13; there is no migration in this release). + Playbooks: `nh3-svos-15c2d04-v0.0.17-stage.yaml`, then `nh3-svos-15c2d04-v0.0.17.yaml`, then `nh3-dev-svos-15c2d04-v0.0.17.yaml`. +- **nh3-svos:** offline backup `backup/pre-15c2d04` (svos.db, integrity ok, user_version 14), `instance.toml.pre-15c2d04`, + `apps/voice.toml.pre-15c2d04`. The six `[[seats.runtimes]]` got `prompt`/`working`/`window` (+ the U17 comment lines) copied + verbatim from `docs/operations/u13-edit-payload.md` @15c2d04 by a script generated from that file, inserted under each + `program` line. Config gen **`29bc36bf074b8245`** (was `13241a403080788c`). Kernel READY in 0.79 s. + voice.toml `utterance_seconds` 30 → 60 (UI 0.0.21's client admits 0.1..120 s). All 8 apps up by 2254:52. + - ⚠ The first cutover run failed at the edit: svc-svos cannot read infra-ops's 0700 stage dir, so the script is now fed on + stdin. The kernel was down from 2254:08 to 2254:22 in total, including the fix and the re-run; the voice app came back last, at about 2254:36. +- **nh3-dev:** `/opt/svos/releases/15c2d04` = 3726d73's apps-env with svos-terminal 0.0.6 (uv pip freeze differs in that line only), + `current → 15c2d04`, only svos-terminal restarted (2255:05). postoffice, miranda and agent still run 3726d73's identical code. + `terminal.toml` unchanged (`activity_interval_seconds` defaults to 5). +- **No card:** terminal manifest revision 6 changes only `version` (0.0.5 → 0.0.6), so the kernel pinned it at 2255:07 without a + card. The expected held-manifest card never existed. +- **Verified (events table):** 13 `seat/activity_changed` at 2255:13, the first pass after the terminal restart. ops-infra was + `working` (this session, mid-deploy: the positive control); the 12 other live seats were `waiting`. dev-ldp is live with no + activity event, so it stays `unknown`. Its claude has been running 14 days, and with both marks declared, a screen that + matches neither is unknown by design. The likeliest cause is an older harness screen (reasoned, not checked). The new + kernel talked to the old 0.0.5 host for 43 s without an error. The only `malformed reply` pair came at 2255:06, while the + terminal was restarting. GET `/seats` was not read: it needs a principal, and infra-ops borrowed none. +- **Rollback:** + - nh3-svos: stop the apps and the kernel, restore `instance.toml.pre-15c2d04` (an old kernel refuses the runtime keys) and + `apps/voice.toml.pre-15c2d04`, `current → releases/b51aa5e`, start, run `svos-front-inventory`. svos.db needs no restore + (no migration); `backup/pre-15c2d04` is there if wanted. + - nh3-dev: `current → releases/3726d73`, restart svos-terminal (revision 5 again; likely pinned without a card, as above). + ## Release b51aa5e / v0.0.16, the judge app (2026-10-18 1633–1642; Prime: "go with your picks, cut over now"; svos-dev thread `01M57XAEZVQ39DKNYXRTMC29S5`) - **Kit:** kernel 0.0.16, app-judge 0.0.1 and UI 0.0.20 from a clean archive of the tag. The rest was carried from 57d90d6. Kit sha256 `25f78195…`.