docs(nh3-svos): kernel 0.0.8 reader pool live; LimitNOFILE stopgap removed; soak flat

2026-10-09 05:45:14 -07:00
parent 457707c9ff
commit 51da9ff5ea
2 changed files with 16 additions and 1 deletions
@@ -123,7 +123,6 @@ _As of 2026-10-08 ~0740 PT. The newest subsection is first; older subsections ca
  - fv-ml1 GPU 0/1 plateau read (Beszel `stats.g.<n>.mu` × 1.024), now ~10-10, since every seat restarted at 0034 10-08.
  - Destroy `zroot/ROOT/debian@pre-patch-20261008` on fv-ml1 ~10-15 if nothing regressed.
  - sfsrv-ana `pre-pbs4` snapshot delete is unverified (infra-ops has no key there; ask infra-hermes).
-  - ⚠ **SVOS kernel 0.0.7 leaks SQLite connections while a browser polls** (EMFILE at 22:55 10-08 took SVOS dark until 2325). STOPGAP on nh3-svos: `/etc/systemd/system/svos-kernel.service.d/10-nofile-stopgap.conf` (LimitNOFILE=65536); the nightly 21:30 thaw restarts the kernel. Root cause (svos-dev, 10-09): `Store._reader` kept one read-only connection per anyio worker thread, and those threads expire after 10 s idle, so a browser's polling bursts leaked one connection per dead worker. The fix is kernel **0.0.8** (svoperatingsystem `b588510`, a bounded reader pool), committed locally and **waiting on Prime's push**. When it is on main: refresh (kernel wheel only; no migration, no config change), DROP the drop-in, then sample svos.db handles with Prime's browser open for a few minutes; flat is the acceptance. until then watch the kernel's fds AND RSS (`ls /proc/<pid>/fd | wc -l`, `VmRSS`).
  - SVOS thaw fix needs repeats (one clean drill is evidence, not proof): after the nightly 21:30 nh3-pve-2 vzdumps, grep nh3-svos `journalctl -u 'svos-*' --since 21:29` for `startup or service failed`; if one reappears, send svos-dev one line.
  - Re-read nh3-nas usage after the 10-08 GC (79% at 10-07 2350; the mirror prune ran OK).
 - **Verified 2026-10-07/08 (closed):** the 10-07 0800 backup check read all fresh with no DESTROYED/DEGRADED lines; the pbs-nh3 mirror prune ran OK at 00:00 on 10-07 and 10-08; `pre-pve9` snapshots are gone on pfi-pve and esh-pve.
@@ -188,6 +187,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-

 ## Recent decisions

+- `[2026-10-09]` **nh3-svos → kernel 0.0.8 (`b588510`, a bounded SQLite reader pool): the 10-08 22:55 EMFILE outage is fixed at the source and the LimitNOFILE stopgap is removed.** A 4-min browser soak held `svos.db` handles flat at 9–12 (0.0.7 grew +22 in 2 min). Root cause: one cached connection per short-lived anyio worker thread. → `servers/nh3-svos/README.md`
 - `[2026-10-08]` **nh3-svos refreshed to ac42ace (interface UI 0.0.7 polish + kernel 0.0.7 with the `Type=notify` unit as a pair); thaw race FIXED**: the pre-refresh drill (old kernel) made two apps fail at thaw, the post-refresh drill made 0. Previous release is now a737a06; 6c6abbe is deletable. → `servers/nh3-svos/README.md`
 - `[2026-10-08]` **SVOS voice ON (Prime, via svos-dev): app-voice on 127.0.0.1:8402, apps up 4 + terminal degraded; Connect → "Voice connected", Enable lists the gateway's 30 voices (Chromium over the LAN door).** `/voice/*` is routed BEFORE Basic auth, because the browser sends `Authorization: Bearer <lease>` there (`web/src/lib/voice.ts`) and Basic auth would reject it. All four app credentials are now vaulted as `nh3-svos/svos-<app>-credential`. → `servers/nh3-svos/README.md`
 - `[2026-10-08]` **SVOS served on the LAN too (Prime):** Caddy binds `10.100.50.43:443` as well as `100.64.0.9:443`; the internal AdGuards answer `svos`/`svos-terminal.phasefinal.com` → `10.100.50.43` (MagicDNS still answers 100.64.0.9). Verified from LAN hosts at all three sites. → `servers/nh3-svos/README.md`
@@ -23,6 +23,21 @@ configuration, enrollment and acceptance.
 | App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events`. **Monitoring:** `scripts/check-backup-freshness.sh` now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict **DEGRADED, exit 4** (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script |
 | Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 |

+## Release b588510 (2026-10-09 0540, kernel 0.0.8: the SQLite reader-pool fix)
+
+- `/opt/svos/current` → `releases/b588510`. Only the kernel wheel changed (0.0.8). Every other artifact was carried
+  byte-identical from ac42ace, the notify unit is unchanged, and there was no migration and no config change.
+  Previous-asset handle: none, because the static build is identical to ac42ace's.
+- **The fd outage of 2026-10-08 22:55–23:25 is fixed at the source.** Root cause (svos-dev): `Store._reader` kept
+  one read-only SQLite connection per anyio worker thread, and those threads die after 10 s idle, so a browser's
+  polling bursts leaked a connection per dead worker. Kernel 0.0.7 hit EMFILE at the default soft limit of 1024
+  (509 `svos.db` + 506 `-wal` fds). The stopgap drop-in `svos-kernel.service.d/10-nofile-stopgap.conf`
+  (LimitNOFILE=65536, 2325 10-08 → 0540 10-09) is **removed**; the limit is back to 1024.
+- Acceptance: `/health` ok, up 4, degraded 1. A 4-min headless-Chromium desk soak (211 `/api` requests) gave
+  `svos.db` handles 9 → 12, with one step during the first burst and then flat for 2.5 min, and `-wal` flat at 9.
+  0.0.7 under a real browser had grown +22 in 2 min. ⚠ Harness differs: headless Chromium here, Prime's browser then.
+- Thaw-fix repeats: 0 app startup failures on the 10-08 21:30 nightly thaw and on the 10-09 05:39 pre-refresh vzdump.
+
 ## Release ac42ace (2026-10-08 1515, svos-dev thread `01M4ES8Q110HNH8J4WWCGE6S4K`)

 - `/opt/svos/current` → `releases/ac42ace`: kernel **0.0.7** (schema 11, no migration), connector base 0.0.6,