From d2f9f561c2f96e4ab35e67ad99cf1be6926c75e4 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Thu, 8 Oct 2026 15:17:56 -0700 Subject: [PATCH] =?UTF-8?q?docs(nh3-svos):=20refreshed=20to=20SVOS=20ac42a?= =?UTF-8?q?ce=20=E2=80=94=20kernel=200.0.7=20+=20Type=3Dnotify=20pair,=20t?= =?UTF-8?q?haw=20race=20fixed=20(drill=202=20->=200)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- persistent-memory.md | 2 +- servers/nh3-svos/README.md | 20 ++++++++++++++++++++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/persistent-memory.md b/persistent-memory.md index 576fe66..020037a 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -124,7 +124,6 @@ _As of 2026-10-08 ~0740 PT. The newest subsection is first; older subsections ca - Destroy `zroot/ROOT/debian@pre-patch-20261008` on fv-ml1 ~10-15 if nothing regressed. - sfsrv-ana `pre-pbs4` snapshot delete is unverified (infra-ops has no key there; ask infra-hermes). - Re-read nh3-nas usage after the 10-08 GC (79% at 10-07 2350; the mirror prune ran OK). - - Install SVOS kernel 0.0.7 (the thaw-race fix Prime accepted; on svoperatingsystem main at `b672fda` since 10-08) with the NEXT nh3-svos refresh, as a PAIR or not at all: the 0.0.7 wheel (sends `READY=1` once bound) plus `svos-kernel.service` with `Type=notify`, `NotifyAccess=main`, `TimeoutStartSec=300`. `Type=notify` on the old kernel times the unit out. Afterwards run one freeze/thaw drill: the interface must NOT restart. Edge: if the notify cannot reach systemd, a healthy kernel is killed at 300 s, with the cause on stderr. Not urgent; until then the interface restarts once per thaw, which is known. - **Verified 2026-10-07/08 (closed):** the 10-07 0800 backup check read all fresh with no DESTROYED/DEGRADED lines; the pbs-nh3 mirror prune ran OK at 00:00 on 10-07 and 10-08; `pre-pve9` snapshots are gone on pfi-pve and esh-pve. - **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens. jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it, as gen-large did (preemptions; decode down to 2–16 tok/s). jekyll's context grows ~1k/call toward the 262k ceiling. It runs on the shared all-agents-local key; a key of its own would make its traffic attributable. - **Waiting on Prime:** @@ -187,6 +186,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache- ## Recent decisions +- `[2026-10-08]` **nh3-svos refreshed to ac42ace (interface UI 0.0.7 polish + kernel 0.0.7 with the `Type=notify` unit as a pair); thaw race FIXED**: the pre-refresh drill (old kernel) made two apps fail at thaw, the post-refresh drill made 0. Previous release is now a737a06; 6c6abbe is deletable. → `servers/nh3-svos/README.md` - `[2026-10-08]` **SVOS voice ON (Prime, via svos-dev): app-voice on 127.0.0.1:8402, apps up 4 + terminal degraded; Connect → "Voice connected", Enable lists the gateway's 30 voices (Chromium over the LAN door).** `/voice/*` is routed BEFORE Basic auth, because the browser sends `Authorization: Bearer ` there (`web/src/lib/voice.ts`) and Basic auth would reject it. All four app credentials are now vaulted as `nh3-svos/svos--credential`. → `servers/nh3-svos/README.md` - `[2026-10-08]` **SVOS served on the LAN too (Prime):** Caddy binds `10.100.50.43:443` as well as `100.64.0.9:443`; the internal AdGuards answer `svos`/`svos-terminal.phasefinal.com` → `10.100.50.43` (MagicDNS still answers 100.64.0.9). Verified from LAN hosts at all three sites. → `servers/nh3-svos/README.md` - `[2026-10-08]` **fv-ml1 is OFF the mesh again, by design**: its tailscaled is disabled, and the break-glass watchdog joins only on fleet loss (09-15 design, archived). The 0030 reboot closed a door that had been left open, so mesh-side tests need another peer, not fv-ml1. The watchdog timer is active. diff --git a/servers/nh3-svos/README.md b/servers/nh3-svos/README.md index fd374b6..f08d561 100644 --- a/servers/nh3-svos/README.md +++ b/servers/nh3-svos/README.md @@ -23,6 +23,26 @@ configuration, enrollment and acceptance. | App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events`. **Monitoring:** `scripts/check-backup-freshness.sh` now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict **DEGRADED, exit 4** (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script | | Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 | +## Release ac42ace (2026-10-08 1515, svos-dev thread `01M4ES8Q110HNH8J4WWCGE6S4K`) + +- `/opt/svos/current` → `releases/ac42ace`: kernel **0.0.7** (schema 11, no migration), connector base 0.0.6, + interface UI **0.0.7** (top bar with the Pacific clock; the tree as the desk's stage), svos-agent 0.0.2 (staged). + Every other app wheel is unchanged, so there was no re-enrollment and no config change. Built from a clean archive, + the same way as a737a06; third-party pins are unchanged. **Previous = `a737a06`** (the front door's previous-asset + handle points at it). `6c6abbe` is no longer referenced and is safe to delete (it cannot read schema 11). +- **Kernel readiness pair installed together:** `svos-kernel.service` now has `Type=notify`, `NotifyAccess=main` and + `TimeoutStartSec=300` (copied from the release's `deployment/nh3-svos/svos-kernel.service`; the old unit is kept as + `.pre-notify`). `systemctl start svos-kernel` now returns only after `READY=1` (0.61 s, `/health` 200 at once). ⚠ + Install a kernel ≥ 0.0.7 and this unit as a pair, never one alone: `Type=notify` on an older kernel times the start + out at 300 s. +- **Freeze/thaw drill, with a control:** the pre-refresh vzdump on the OLD kernel (15:14) made terminal and knowledge + fail once at thaw (the known race). The post-refresh vzdump (`vm/111/2026-10-08T22:16:12Z`, 15:16) showed **0** + app startup failures; all six units came back, and `/health` read up 4, degraded 1 within ~2 s. The one-poll + "down" for the terminal right after the restart is its first health poll, not a fault. +- Acceptance: `/health` ok, up 4, degraded 1. Chromium phone + desk over the LAN door: the top bar is present with + clock `1517`, the desk stage shows the tree (4 stage layers + hero), no overflow, 0 page errors; desk Connect voice + shows "Voice connected". + ## Release a737a06 + apps (2026-10-07, svos-dev thread `01M4D1GNYBW7QVY7RF05R412V2`, Prime's 2240 decision) - **Release:** `/opt/svos/current` → `releases/a737a06` (previous `6c6abbe` kept: it cannot read schema 11, and the