esh-pfi-infrastructure/servers/nh3-svos/README.md

nh3-svos: host for the NEW SVOS operating system (VM 111 on nh3-pve-2, NH3)

Built 2026-10-05 at svos-dev's request, relaying Prime ("stand up a dedicated VM for it on nh3-pve-2"). It runs the new SVOS (repo corviduo/svoperatingsystem): a FastAPI/uvicorn kernel (SQLite WAL, synchronous FULL, one writer), an interface connector, an optional voice adapter and a static phone/desktop UI. It is a separate instance. The existing High Seat (svos.service on nh3-dev, :8770) and Miranda are NOT migrated by this host.

Ownership split: infra-ops owns the VM, OS, access, backups and ingress. svos-dev owns the application: install, configuration, enrollment and acceptance.

Address	10.100.50.43, VLAN 50 (nh3-servers), static via cloud-init; nh3-svos.nh3.internal
Guest	VM 111 on nh3-pve-2, onboot 1, tag svos
Shape	4 vCPU x86-64-v3, 8 GiB, 100 GB on vmstore (local NVMe LVM-thin, discard on). nh3-pve-2 had 47 GB RAM and 946 GB vmstore free at build
OS	Debian 13 genericcloud, Python 3.13 + venv, SQLite 3.46, systemd-timesyncd (synced), qemu-guest-agent
Identities	vh 1000 (operator keys + lkraven@nh3-dev, so svos-dev's session can ssh in; NOPASSWD sudo from cloud-init, plus /etc/sudoers.d/91-vh-verifypw so sudo -v agrees), infra-ops 850, svc-svos 801 (nologin; the only writer of the data root)
Data root	/var/lib/svos, svc-svos 0700, on the local root ext4. Never NFS, never shared
Releases	/opt/svos root:svc-svos 2775, with releases/ for svos-dev's layout. The git checkout is not a runtime dependency
Docker	none, by design
Backups	nh3-pve-2 vzdump job, nightly 21:30, all 1 → pbs-ana namespace nh3-pve-2 (mirrored to pbs-nh3 at 06:30), snapshot mode with guest-agent fs-freeze
App-consistent hook	svos-dev's /etc/qemu/fsfreeze-hook.d/50-svos stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ Debian's qemu-ga does NOT run fsfreeze hooks unless configured (no -F in the unit, no qemu-ga.conf by default). Here /etc/qemu/qemu-ga.conf points it at /usr/local/sbin/fsfreeze-dispatch (configs/qemu-ga/fsfreeze-dispatch), which unlike the stock /etc/qemu/fsfreeze-hook propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. Live drill 2026-10-05 1952–1953 (services running): quiesced backup vm/111/2026-10-06T02:52:11Z → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary 60-drill-refuse after 50-svos, since removed) vm/111/2026-10-06T02:53:03Z → the dispatcher unwound and restarted the services, and then ⚠ vzdump did NOT abort: it logged ERROR: unable to freeze guest fs, backed up UNFROZEN (crash-consistent, app live), and reported Backup job finished successfully, rc 0. Both images mapped from PBS and mounted read-only (ro,noload) on nh3-pve-2, never booted: svos.db integrity ok (each image still carries its ~0.9 MB WAL: keep -wal with the db when restoring), interface pins.sqlite3 integrity ok, config/key/config-generation hashes equal live, table counts equal live except events. Monitoring: scripts/check-backup-freshness.sh now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict DEGRADED, exit 4 (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script
Monitoring	Beszel native agent 0.18.7 (playbooks/beszel-native.yaml, configs/beszel-agent/hosts/nh3-svos.env), hub system t6y646ifa3x7xxv, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05

Release afab123 / v0.0.9 + U13 phase B (2026-10-10 1646–1648, Prime-approved 1639; svos-dev thread 01M4M2Z2H3BQH375M2G30GQHDM)

• /opt/svos/current → releases/afab123: kernel 0.0.9, schema 11 → 12 (migration 0012, the page verb's tables; start returned READY in 0.64 s). Built from a clean archive of afab123: kernel, connector 0.0.7, postoffice 0.0.2, terminal 0.0.3, agent 0.0.3 and miranda 0.0.2 wheels; interface 0.0.4, knowledge 0.0.1 and voice 0.0.1 wheels, static, vendor, hermes-plugin, the kernel unit and apps-requirements.txt carried byte-identical from b588510 (sources unchanged, verified). /health ok, schema 12, up 4, degraded 1 (the terminal, by design). Inventory regenerated; the front door's previous-asset handle is now b588510.
• Pre-change backups: vzdump vm/111/2026-10-10T23:45:41Z on pbs-ana (freeze/thaw clean), and an OFFLINE copy taken with every service stopped: /var/lib/svos/backup/pre-afab123/ (svos.db integrity ok, user_version 11, plus config-keys-apps.tar). Rollback: stop all, remove the kernel drop-in, restore that svos.db (keep no stale -wal/-shm), switch current back to b588510, start.
• ⚠ app-terminal and app-postoffice are HELD on b588510 by drop-ins svos-{terminal,postoffice}.service.d/10-hold-b588510.conf (configs/svos-apps/): their v0.0.9 manifests changed, and a new version would be held for approval and refuse the kernel's calls until then (svos-dev's sequencing). Both move to nh3-dev at the U13 edit: remove the drop-ins there, and do NOT delete releases/b588510 before that. interface, knowledge and voice run afab123.
• B2, the kernel's app origin: drop-in svos-kernel.service.d/20-app-listen.conf (configs/svos-kernel/) adds --app-listen 127.0.0.1:8410; Caddy serves https://svos-kernel.phasefinal.com on 10.100.50.43 only (own LE cert, DNS-01), no Basic auth, bearer passed, identity headers stripped, source allowlisted to nh3-dev (10.100.10.50), paths method-pinned to svoperatingsystem dc241f8 u13-svos-edit.md item 1 (the apps' and Miranda's agent connector's calls: GET /auth/kernel-key /auth/whoami /registry/apps[/*] /principals/*/grants /seats /events /artifacts/resolve; POST /auth/enroll /auth/proofs /seats/enroll /registry/apps/*/manifest; /verbs/*), else 404. The three internal AdGuards answer the name with 10.100.50.43. Tested from nh3-dev: all 15 allowed method+path pairs → 401 from the kernel (JSON detail); /health, /stream, /cards, POST /auth/kernel-key, DELETE /registry/apps/x, /principals/a/b/grants → Caddy's empty 404; from nh3-svos and nh3-docker → 403. Backups /etc/caddy/Caddyfile.pre-kernel-origin, .pre-kernel-paths. ⚠ Validate from a caddy-readable path (/etc/caddy/Caddyfile.check): the first deploy validated /tmp/svos-stage-*/ (0700 infra-ops), got "permission denied", and a | tail hid it. The config was valid (Caddy's own reload accepted it), and the validator was then positive-controlled (a bogus directive → rc 1).
• B5 + enrollment secrets (2026-10-10 1654): svos-kernel mint-secret (run as infra-ops without sudo, so no sudo-io copy) minted three tokens straight into the vault, read back by hash: nh3-svos/app-high-seat-enrollment-secret, nh3-svos/app-miranda-enrollment-secret, nh3-svos/miranda-persona-enrollment-secret. Only app-high-seat is declared so far ([[apps]] with enrollment_sha256, one reload, config_gen 22a05c20e17f93ce → 1bd948c754e571da, backup instance.toml.pre-high-seat); the other two are declared in the edit.
• nh3-svos stays on afab123: svoperatingsystem dc241f8 changes only svos-miranda (0.0.3, the charter), which runs on nh3-dev, so a dc241f8 release here would change nothing that runs.
• The app-side origins on nh3-dev (svos-seathost, svos-highseat) and the bridge env are in servers/nh3-dev/README.md.

Release b588510 (2026-10-09 0540, kernel 0.0.8: the SQLite reader-pool fix)

• /opt/svos/current → releases/b588510. Only the kernel wheel changed (0.0.8). Every other artifact was carried byte-identical from ac42ace, the notify unit is unchanged, and there was no migration and no config change. Previous-asset handle: none, because the static build is identical to ac42ace's.
• The fd outage of 2026-10-08 22:55–23:25 is fixed at the source. Root cause (svos-dev): Store._reader kept one read-only SQLite connection per anyio worker thread, and those threads die after 10 s idle, so a browser's polling bursts leaked a connection per dead worker. Kernel 0.0.7 hit EMFILE at the default soft limit of 1024 (509 svos.db + 506 -wal fds). The stopgap drop-in svos-kernel.service.d/10-nofile-stopgap.conf (LimitNOFILE=65536, 2325 10-08 → 0540 10-09) is removed; the limit is back to 1024.
• Acceptance: /health ok, up 4, degraded 1. A 4-min headless-Chromium desk soak (211 /api requests) gave svos.db handles 9 → 12, with one step during the first burst and then flat for 2.5 min, and -wal flat at 9. 0.0.7 under a real browser had grown +22 in 2 min. ⚠ Harness differs: headless Chromium here, Prime's browser then.
• Thaw-fix repeats: 0 app startup failures on the 10-08 21:30 nightly thaw and on the 10-09 05:39 pre-refresh vzdump.

Release ac42ace (2026-10-08 1515, svos-dev thread 01M4ES8Q110HNH8J4WWCGE6S4K)

• /opt/svos/current → releases/ac42ace: kernel 0.0.7 (schema 11, no migration), connector base 0.0.6, interface UI 0.0.7 (top bar with the Pacific clock; the tree as the desk's stage), svos-agent 0.0.2 (staged). Every other app wheel is unchanged, so there was no re-enrollment and no config change. Built from a clean archive, the same way as a737a06; third-party pins are unchanged. Previous = a737a06 (the front door's previous-asset handle points at it). 6c6abbe is no longer referenced and is safe to delete (it cannot read schema 11).
• Kernel readiness pair installed together: svos-kernel.service now has Type=notify, NotifyAccess=main and TimeoutStartSec=300 (copied from the release's deployment/nh3-svos/svos-kernel.service; the old unit is kept as .pre-notify). systemctl start svos-kernel now returns only after READY=1 (0.61 s, /health 200 at once). ⚠ Install a kernel ≥ 0.0.7 and this unit as a pair, never one alone: Type=notify on an older kernel times the start out at 300 s.
• Freeze/thaw drill, with a control: the pre-refresh vzdump on the OLD kernel (15:14) made terminal and knowledge fail once at thaw (the known race). The post-refresh vzdump (vm/111/2026-10-08T22:16:12Z, 15:16) showed 0 app startup failures; all six units came back, and /health read up 4, degraded 1 within ~2 s. The one-poll "down" for the terminal right after the restart is its first health poll, not a fault.
• Acceptance: /health ok, up 4, degraded 1. Chromium phone + desk over the LAN door: the top bar is present with clock 1517, the desk stage shows the tree (4 stage layers + hero), no overflow, 0 page errors; desk Connect voice shows "Voice connected".

Release a737a06 + apps (2026-10-07, svos-dev thread 01M4D1GNYBW7QVY7RF05R412V2, Prime's 2240 decision)

• Release: /opt/svos/current → releases/a737a06 (previous 6c6abbe kept: it cannot read schema 11, and the front door's previous-asset handle still points at it). Built by infra-ops from a clean git archive: 9 wheels, the web/ 0.0.6 static build, kernel-requirements.txt (uv export) and apps-requirements.txt (uv pip compile against the kernel pins, hashed), vendor/bifrost-1.2.1 (gitea vh index, sha256-verified, because PyPI has an unrelated bifrost), and hermes-plugin/svos (staged for U13). Kernel 0.0.5 migrated schema 9 → 11 on first start.
• Pre-change backups: vzdump vm/111/2026-10-08T06:02:14Z (freeze/thaw clean) and an online sqlite .backup plus config, keys and app state at /var/lib/svos/backup/pre-a737a06/ (integrity ok, user_version 9).
• Apps:

  slug	listener	unit	state
  app-interface	127.0.0.1:8403	svos-interface	up
  app-knowledge	127.0.0.1:8405	svos-knowledge	up; Albok token 01M4D22HHJ7AJHX30WPDE02DFC, vault nh3-svos/albok-token-knowledge, grants per A111 read:fleet/memory, read:fleet/stash, read:personal/notes (no wildcard, no restricted wing, no class grant)
  app-terminal	127.0.0.1:8404, base_url https://svos-terminal.phasefinal.com	svos-terminal	degraded by design: no zellij and no fleet session on this box
  app-postoffice	127.0.0.1:8406	svos-postoffice (installed, NOT enabled)	declared + enrolled, stopped until Prime declares the bridge handle; handle = "svos" in its config is the example's placeholder
  app-voice	127.0.0.1:8402, front door /voice/* (same origin, BEFORE Basic auth; the lease bearer is the auth)	svos-voice	up since 2026-10-08 (Prime: voice on); engines in config/apps/voice.toml: STT Parakeet http://10.251.50.54:8300, TTS fleet gateway http://irv-ml1.nh3.internal:8198; owner grants app:app-voice write (autonomous) + read
  app-miranda, svos-agent	—	—	packages in apps-env only (U13)
  Credentials /etc/svos/<app>.env (root 0600), vaulted as nh3-svos/svos-<app>-credential (terminal, knowledge, postoffice, voice; 2026-10-08); units in configs/svos-apps/; the quiesce hook lists every app unit.

• Terminal origin: svos-terminal.phasefinal.com = headscale MagicDNS extra record → 100.64.0.9, its own Caddy site (LE via DNS-01). No Basic auth (Chromium cancels Basic prompts in cross-origin frames; the lease is the page's auth). /verbs/* and /health are refused unless they come from the box itself. The kernel's signed calls reach the app through Caddy, so nh3-svos resolves the name via /etc/hosts (100.64.0.9 svos-terminal.phasefinal.com svos.phasefinal.com): it has no MagicDNS (accept-dns=false). The terminal is up only while Caddy is.
• Acceptance 2026-10-07 ~2315: 17/17 HTTPS controls through an SSH forward to the real listener (the 12 from 10-05, plus: health schema 11 with 2 up and 1 degraded; the terminal registered on its own origin; knowledge wings = exactly the 3 token wings; knowledge search hits; terminal attach refused 404 unknown seat). Chromium phone + desk: shell, inbox, stream, reload, /status, /graph, /feed, no overflow, no page errors; desk "Open Terminal here" → pane open, framed from https://svos-terminal.phasefinal.com/, body waiting (no lease). From mesh peer fv-ml1, the terminal origin returns: / 200 with frame-ancestors https://svos.phasefinal.com; lease routes 401; /verbs/* and /health 403; anything else 404.

Build (reproducible)

scripts/elway infra-ops@10.100.250.62 --playbook playbooks/nh3-pve-2-cloudinit-vm.yaml --var vmid=111 \
  --var name=nh3-svos --var ip=10.100.50.43/24 --var cores=4 --var memory=8192 --var disk=100G --var tags=svos \
  --var "description=..."
scripts/elway vh@10.100.50.43        --playbook playbooks/bootstrap-infra-ops-user.yaml
scripts/elway infra-ops@10.100.50.43 --playbook playbooks/nh3-svos-host.yaml

Front door: mesh-only https://svos.phasefinal.com, HTTP Basic auth (Prime, 2026-10-05)

• Mesh: tailscaled joined headscale as nh3-svos, 100.64.0.9, --accept-dns=false --accept-routes=false.
• Name: svos.phasefinal.com (and svos-terminal.phasefinal.com) has no public record. Mesh clients get 100.64.0.9 from the headscale MagicDNS extra records (/var/lib/headscale/extra-records.json on nh3-headscale, hot-reloaded). Since 2026-10-08 (Prime: "serve it on the LAN too and point the dns at the internal LAN address") the three internal AdGuards (ana 10.250.50.70:8053, esh 10.0.50.45, nh3 10.100.50.40) carry hand-made rewrites for both names → 10.100.50.43, outside dns-sync.py's .internal authority (like *.nh3.phasefinal.com). Verified 10-08 from LAN hosts at all three sites (nh3-dev, nh3-docker, esh-docker-vm, ana-docker): / 401, /sw.js 200, terminal / 200, terminal /health 403. nh3-svos itself still resolves both names to 100.64.0.9 via /etc/hosts, so the kernel's signed calls to the terminal keep arriving from 100.64.0.9.
• TLS: Caddy 2.11.4 (the fleet caddy-cf build with the Cloudflare DNS module) at /usr/local/bin/caddy, unit caddy-svos.service, config /etc/caddy/Caddyfile (canonical configs/svos-front/Caddyfile). It listens on 100.64.0.9:443 and, since 2026-10-08, 10.100.50.43:443 (both site blocks), with no :80 and nothing on the internet. The LE cert for svos.phasefinal.com comes via DNS-01, token nh3-svos-caddy-dns01 (phasefinal.com only), in /etc/caddy/env (root:caddy 0640), vaulted nh3-svos/cloudflare-caddy-dns01-token.
• Identity: HTTP Basic auth, one account Vuong (Prime: "Basic auth is fine, saved in browser is fine. Access is via mesh because we don't host externally." SVOS addresses him as Prime; the login is Vuong). The argon2id hash is in /etc/caddy/svos-basic-auth.accounts (root:caddy 0640, not in git), imported into basic_auth argon2id SVOS. The plaintext is vaulted at nh3-svos/basic-auth-vuong. To rotate: pipe the new password to caddy hash-password --algorithm argon2id on stdin (never --plaintext), rewrite the accounts file, secret put, then systemctl reload caddy-svos.
• Credential-free offline shell (svos-dev, 2026-10-05): the service worker fetches its hash-pinned INVENTORY with credentials:'omit', so exactly those files plus /sw.js are served BEFORE Basic auth (GET/HEAD, exact files, no fallback). Public build bytes only; /, /status, /api/*, the connector and everything else stay gated. The path list is generated into /etc/caddy/svos-public-inventory by /usr/local/sbin/svos-front-inventory (configs/svos-front/): it hash-checks every file against INVENTORY, validates, then reloads. ⚠ Run it as root after every SVOS release, or the next build's hashed asset names stay gated and worker install fails. --check changes nothing.
• Previous release's assets across a rollout: the same tool writes /etc/caddy/svos-public-previous: when /opt/svos/current has moved, the release named in the old inventory header becomes "previous", and its hashed /assets/* files that the new build lacks are served credential-free from that release's own static dir (hash-checked). One previous release at most; comments only when there is none (true at 2026-10-05, one release). Replica-tested: old asset 200 with the old INVENTORY hash, new asset with the new hash, POST to the old asset 401, unknown asset 401/404; tampered files and an unsafe path make the tool refuse.
• Hosting bounds (svos-dev, per the web contract): request bodies capped at exactly 1 MiB on every route (413 over); headers within 5 s, whole body within 15 s (Caddy read_header / read_body); no write, response or idle deadline, so /api/stream (SSE) stays open. Live 2026-10-05 2014–2021 from fv-ml1: 1 MiB+1 → 413 on /api/verbs and /verbs/pin; exactly 1 MiB passes Caddy (kernel then answers); unfinished headers closed at 5.0 s; stalled body cut at 15.0 s; a signed app-interface ping returned pong before and DURING a stalled body; an SSE stream stayed open 41–45 s (client hung up) alongside all of it. ⚠ A stalled body over HTTP/1.1 can come back as Caddy's empty 200 OK (Content-Length 0) instead of 504: replica 3/3 buffered, 2/3 unbuffered; live 2 of 3. Over h2, which browsers use, it is 502 (3/3 replica, 2/2 live). The app never gets a complete request either way; an empty 200 is not success. handle_errors 499 and request_buffers do not change it (tested).
• Static routes (gated), per svoperatingsystem web/README.md Hosting contract: / and /status → index.html; /assets/*, /icons/*, /favicon.svg, /manifest.webmanifest, /index.html, /sw.js → exact file, missing → 404; other HTML document requests (Sec-Fetch-Dest: document or Accept: text/html) → index.html; anything else → 404. Unversioned files get Cache-Control: no-cache.
• Acceptance matrix: configs/svos-front/acceptance-matrix.sh (35 checks; usage in its header). 35/35 pass live at 1957 2026-10-05 from fv-ml1; against the previous config it reports 16 FAIL (positive control).
• Headers: client copies of Tailscale-User-*, X-Forwarded-*, Tailscale-Headers-Info, X-Real-IP, Forwarded and X-SVOS-Peer are deleted before auth. After auth, Tailscale-User-Login is SET to the authenticated user, and Authorization is deleted. That deletion is load-bearing: the kernel's human path (svoperatingsystem u1-principals INV-U1-06) refuses any request carrying Authorization. As a result the front door is human-only; a bearer client cannot come through it.
• Retired 2026-10-05: the WhoIs forward_auth gate (svos-whois-auth, 127.0.0.1:8405). Every mesh node has WhoIs login pfi, so it could not tell Prime's devices from the servers without restructuring the mesh, and Prime declined that. The unit, script, /etc/svos-front/ and the repo files are removed; git history has them.
• Config changes: validate WITHOUT putting the token on a command line, then reload: sudo systemd-run --wait --pipe --collect -p User=caddy -p Group=caddy -p EnvironmentFile=/etc/caddy/env -p "Environment=XDG_DATA_HOME=/var/lib/caddy XDG_CONFIG_HOME=/var/lib/caddy" /usr/local/bin/caddy validate --config <file> --adapter caddyfile, then systemctl reload caddy-svos. (env $(cat /etc/caddy/env) caddy … leaked the first token into sudo's argv log on 2026-10-05; it was revoked.)
• Tested 2026-10-05 ~1940–1946, from mesh peer fv-ml1, against a temporary header echo on 127.0.0.1:8400 (removed after):
  • no credentials → 401 Basic realm="SVOS"; wrong password → 401; username vuong (wrong case) → 401;
  • right credentials with forged Tailscale-User-Login/-Whatever, X-Forwarded-For/Proto/Zzz, Forwarded and X-Real-IP → 200. Upstream saw exactly one Tailscale-User-Login: Vuong, no Authorization, none of the forged headers, X-Forwarded-Proto: https, Host: svos.phasefinal.com, and path /health (the /api prefix stripped);
  • connector route with right credentials → 502 (auth passed; the connector was not running); cert verifies; LAN 10.100.50.43:443 → connection refused.

Original requirements (svos-dev)

App side, per svos-dev 2026-10-05: kernel on 127.0.0.1:8400 --trust-loopback (its Unix-socket mode needs a 0700 kernel-owned dir and creates a 0666 socket, so no shared-group socket in this build), interface connector on 127.0.0.1:8403. Static UI at /opt/svos/current/static. /api/* → kernel with the prefix stripped (/api/health is the kernel's); /health, /verbs/ping|pins|pin|unpin, /openapi.json → connector unchanged. ⚠ With --trust-loopback, every local process is inside the identity trust boundary, so run no untrusted workloads here. Agreed design (option A): Caddy on the tailnet address only, forward_auth to a loopback WhoIs responder fed the real socket peer ({http.request.remote.host}:{port}), fail closed, allow-list of logins, client identity/forwarding headers stripped, Origin passed through untouched.

App listeners stay private (loopback or a Unix socket). The front door has to give the browser HTTPS: the UI and the voice adapter need a secure context for the microphone. It also has to give the kernel's adapter a verified identity in Tailscale-User-Login, with any browser-supplied copy stripped. See the svos-dev thread of 2026-10-05.