docs(nh3-svos): kernel 0.0.8 reader pool live; LimitNOFILE stopgap removed; soak flat

2026-10-09 05:45:14 -07:00
parent 457707c9ff
commit 51da9ff5ea
2 changed files with 16 additions and 1 deletions
@@ -23,6 +23,21 @@ configuration, enrollment and acceptance.
 | App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events`. **Monitoring:** `scripts/check-backup-freshness.sh` now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict **DEGRADED, exit 4** (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script |
 | Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 |

+## Release b588510 (2026-10-09 0540, kernel 0.0.8: the SQLite reader-pool fix)
+
+- `/opt/svos/current` → `releases/b588510`. Only the kernel wheel changed (0.0.8). Every other artifact was carried
+  byte-identical from ac42ace, the notify unit is unchanged, and there was no migration and no config change.
+  Previous-asset handle: none, because the static build is identical to ac42ace's.
+- **The fd outage of 2026-10-08 22:55–23:25 is fixed at the source.** Root cause (svos-dev): `Store._reader` kept
+  one read-only SQLite connection per anyio worker thread, and those threads die after 10 s idle, so a browser's
+  polling bursts leaked a connection per dead worker. Kernel 0.0.7 hit EMFILE at the default soft limit of 1024
+  (509 `svos.db` + 506 `-wal` fds). The stopgap drop-in `svos-kernel.service.d/10-nofile-stopgap.conf`
+  (LimitNOFILE=65536, 2325 10-08 → 0540 10-09) is **removed**; the limit is back to 1024.
+- Acceptance: `/health` ok, up 4, degraded 1. A 4-min headless-Chromium desk soak (211 `/api` requests) gave
+  `svos.db` handles 9 → 12, with one step during the first burst and then flat for 2.5 min, and `-wal` flat at 9.
+  0.0.7 under a real browser had grown +22 in 2 min. ⚠ Harness differs: headless Chromium here, Prime's browser then.
+- Thaw-fix repeats: 0 app startup failures on the 10-08 21:30 nightly thaw and on the 10-09 05:39 pre-refresh vzdump.
+
 ## Release ac42ace (2026-10-08 1515, svos-dev thread `01M4ES8Q110HNH8J4WWCGE6S4K`)

 - `/opt/svos/current` → `releases/ac42ace`: kernel **0.0.7** (schema 11, no migration), connector base 0.0.6,