docs(nh3-svos): kernel 0.0.8 reader pool live; LimitNOFILE stopgap removed; soak flat

2026-10-09 05:45:14 -07:00
parent 457707c9ff
commit 51da9ff5ea
2 changed files with 16 additions and 1 deletions
@@ -123,7 +123,6 @@ _As of 2026-10-08 ~0740 PT. The newest subsection is first; older subsections ca
  - fv-ml1 GPU 0/1 plateau read (Beszel `stats.g.<n>.mu` × 1.024), now ~10-10, since every seat restarted at 0034 10-08.
  - Destroy `zroot/ROOT/debian@pre-patch-20261008` on fv-ml1 ~10-15 if nothing regressed.
  - sfsrv-ana `pre-pbs4` snapshot delete is unverified (infra-ops has no key there; ask infra-hermes).
-  - ⚠ **SVOS kernel 0.0.7 leaks SQLite connections while a browser polls** (EMFILE at 22:55 10-08 took SVOS dark until 2325). STOPGAP on nh3-svos: `/etc/systemd/system/svos-kernel.service.d/10-nofile-stopgap.conf` (LimitNOFILE=65536); the nightly 21:30 thaw restarts the kernel. Root cause (svos-dev, 10-09): `Store._reader` kept one read-only connection per anyio worker thread, and those threads expire after 10 s idle, so a browser's polling bursts leaked one connection per dead worker. The fix is kernel **0.0.8** (svoperatingsystem `b588510`, a bounded reader pool), committed locally and **waiting on Prime's push**. When it is on main: refresh (kernel wheel only; no migration, no config change), DROP the drop-in, then sample svos.db handles with Prime's browser open for a few minutes; flat is the acceptance. until then watch the kernel's fds AND RSS (`ls /proc/<pid>/fd | wc -l`, `VmRSS`).
  - SVOS thaw fix needs repeats (one clean drill is evidence, not proof): after the nightly 21:30 nh3-pve-2 vzdumps, grep nh3-svos `journalctl -u 'svos-*' --since 21:29` for `startup or service failed`; if one reappears, send svos-dev one line.
  - Re-read nh3-nas usage after the 10-08 GC (79% at 10-07 2350; the mirror prune ran OK).
 - **Verified 2026-10-07/08 (closed):** the 10-07 0800 backup check read all fresh with no DESTROYED/DEGRADED lines; the pbs-nh3 mirror prune ran OK at 00:00 on 10-07 and 10-08; `pre-pve9` snapshots are gone on pfi-pve and esh-pve.
@@ -188,6 +187,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-

 ## Recent decisions

+- `[2026-10-09]` **nh3-svos → kernel 0.0.8 (`b588510`, a bounded SQLite reader pool): the 10-08 22:55 EMFILE outage is fixed at the source and the LimitNOFILE stopgap is removed.** A 4-min browser soak held `svos.db` handles flat at 9–12 (0.0.7 grew +22 in 2 min). Root cause: one cached connection per short-lived anyio worker thread. → `servers/nh3-svos/README.md`
 - `[2026-10-08]` **nh3-svos refreshed to ac42ace (interface UI 0.0.7 polish + kernel 0.0.7 with the `Type=notify` unit as a pair); thaw race FIXED**: the pre-refresh drill (old kernel) made two apps fail at thaw, the post-refresh drill made 0. Previous release is now a737a06; 6c6abbe is deletable. → `servers/nh3-svos/README.md`
 - `[2026-10-08]` **SVOS voice ON (Prime, via svos-dev): app-voice on 127.0.0.1:8402, apps up 4 + terminal degraded; Connect → "Voice connected", Enable lists the gateway's 30 voices (Chromium over the LAN door).** `/voice/*` is routed BEFORE Basic auth, because the browser sends `Authorization: Bearer <lease>` there (`web/src/lib/voice.ts`) and Basic auth would reject it. All four app credentials are now vaulted as `nh3-svos/svos-<app>-credential`. → `servers/nh3-svos/README.md`
 - `[2026-10-08]` **SVOS served on the LAN too (Prime):** Caddy binds `10.100.50.43:443` as well as `100.64.0.9:443`; the internal AdGuards answer `svos`/`svos-terminal.phasefinal.com` → `10.100.50.43` (MagicDNS still answers 100.64.0.9). Verified from LAN hosts at all three sites. → `servers/nh3-svos/README.md`