vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: SVOS kernel 0.0.7 SQLite connection leak — EMFILE outage, LimitNOFILE stopgap on nh3-svos

Browse Source
This commit is contained in:
vh
2026-10-08 23:25:24 -07:00
parent 00d990d485
commit 6b2de2b356
1 changed files with 1 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+1
View File
@@ -123,6 +123,7 @@ _As of 2026-10-08 ~0740 PT. The newest subsection is first; older subsections ca
- fv-ml1 GPU 0/1 plateau read (Beszel `stats.g.<n>.mu` × 1.024), now ~10-10, since every seat restarted at 0034 10-08. - fv-ml1 GPU 0/1 plateau read (Beszel `stats.g.<n>.mu` × 1.024), now ~10-10, since every seat restarted at 0034 10-08.
- Destroy `zroot/ROOT/debian@pre-patch-20261008` on fv-ml1 ~10-15 if nothing regressed. - Destroy `zroot/ROOT/debian@pre-patch-20261008` on fv-ml1 ~10-15 if nothing regressed.
- sfsrv-ana `pre-pbs4` snapshot delete is unverified (infra-ops has no key there; ask infra-hermes). - sfsrv-ana `pre-pbs4` snapshot delete is unverified (infra-ops has no key there; ask infra-hermes).
- ⚠ **SVOS kernel 0.0.7 leaks SQLite connections while a browser polls** (EMFILE at 22:55 10-08 took SVOS dark until 2325). STOPGAP on nh3-svos: `/etc/systemd/system/svos-kernel.service.d/10-nofile-stopgap.conf` (LimitNOFILE=65536); the nightly 21:30 thaw restarts the kernel. Root cause is svos-dev's (evidence sent). Remove the drop-in when the fix lands; until then watch the kernel's fds AND RSS (`ls /proc/<pid>/fd | wc -l`, `VmRSS`).
- SVOS thaw fix needs repeats (one clean drill is evidence, not proof): after the nightly 21:30 nh3-pve-2 vzdumps, grep nh3-svos `journalctl -u 'svos-*' --since 21:29` for `startup or service failed`; if one reappears, send svos-dev one line. - SVOS thaw fix needs repeats (one clean drill is evidence, not proof): after the nightly 21:30 nh3-pve-2 vzdumps, grep nh3-svos `journalctl -u 'svos-*' --since 21:29` for `startup or service failed`; if one reappears, send svos-dev one line.
- Re-read nh3-nas usage after the 10-08 GC (79% at 10-07 2350; the mirror prune ran OK). - Re-read nh3-nas usage after the 10-08 GC (79% at 10-07 2350; the mirror prune ran OK).
- **Verified 2026-10-07/08 (closed):** the 10-07 0800 backup check read all fresh with no DESTROYED/DEGRADED lines; the pbs-nh3 mirror prune ran OK at 00:00 on 10-07 and 10-08; `pre-pve9` snapshots are gone on pfi-pve and esh-pve. - **Verified 2026-10-07/08 (closed):** the 10-07 0800 backup check read all fresh with no DESTROYED/DEGRADED lines; the pbs-nh3 mirror prune ran OK at 00:00 on 10-07 and 10-08; `pre-pve9` snapshots are gone on pfi-pve and esh-pve.