esh-pfi-infrastructure/servers/nh3-svos

nh3-svos: host for the NEW SVOS operating system (VM 111 on nh3-pve-2, NH3)

Built 2026-10-05 at svos-dev's request, relaying Prime ("stand up a dedicated VM for it on nh3-pve-2"). It runs the new SVOS (repo corviduo/svoperatingsystem): a FastAPI/uvicorn kernel (SQLite WAL, synchronous FULL, one writer), an interface connector, an optional voice adapter and a static phone/desktop UI. It is a separate instance. The existing High Seat (svos.service on nh3-dev, :8770) and Miranda are NOT migrated by this host.

Ownership split: infra-ops owns the VM, OS, access, backups and ingress. svos-dev owns the application: install, configuration, enrollment and acceptance.

Address	10.100.50.43, VLAN 50 (nh3-servers), static via cloud-init; nh3-svos.nh3.internal
Guest	VM 111 on nh3-pve-2, onboot 1, tag svos
Shape	4 vCPU x86-64-v3, 8 GiB, 100 GB on vmstore (local NVMe LVM-thin, discard on). nh3-pve-2 had 47 GB RAM and 946 GB vmstore free at build
OS	Debian 13 genericcloud, Python 3.13 + venv, SQLite 3.46, systemd-timesyncd (synced), qemu-guest-agent
Identities	vh 1000 (operator keys + lkraven@nh3-dev, so svos-dev's session can ssh in; NOPASSWD sudo from cloud-init, plus /etc/sudoers.d/91-vh-verifypw so sudo -v agrees), infra-ops 850, svc-svos 801 (nologin; the only writer of the data root)
Data root	/var/lib/svos, svc-svos 0700, on the local root ext4. Never NFS, never shared
Releases	/opt/svos root:svc-svos 2775, with releases/ for svos-dev's layout. The git checkout is not a runtime dependency
Docker	none, by design
Backups	nh3-pve-2 vzdump job, nightly 21:30, all 1 → pbs-ana namespace nh3-pve-2 (mirrored to pbs-nh3 at 06:30), snapshot mode with guest-agent fs-freeze
App-consistent hook	svos-dev's /etc/qemu/fsfreeze-hook.d/50-svos stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ Debian's qemu-ga does NOT run fsfreeze hooks unless configured (no -F in the unit, no qemu-ga.conf by default). Here /etc/qemu/qemu-ga.conf points it at /usr/local/sbin/fsfreeze-dispatch (configs/qemu-ga/fsfreeze-dispatch), which unlike the stock /etc/qemu/fsfreeze-hook propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. Live drill 2026-10-05 1952–1953 (services running): quiesced backup vm/111/2026-10-06T02:52:11Z → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary 60-drill-refuse after 50-svos, since removed) vm/111/2026-10-06T02:53:03Z → the dispatcher unwound and restarted the services, and then ⚠ vzdump did NOT abort: it logged ERROR: unable to freeze guest fs, backed up UNFROZEN (crash-consistent, app live), and reported Backup job finished successfully, rc 0. Both images mapped from PBS and mounted read-only (ro,noload) on nh3-pve-2, never booted: svos.db integrity ok (each image still carries its ~0.9 MB WAL: keep -wal with the db when restoring), interface pins.sqlite3 integrity ok, config/key/config-generation hashes equal live, table counts equal live except events. Monitoring: scripts/check-backup-freshness.sh now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict DEGRADED, exit 4 (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script
Monitoring	Beszel native agent 0.18.7 (playbooks/beszel-native.yaml, configs/beszel-agent/hosts/nh3-svos.env), hub system t6y646ifa3x7xxv, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05

Release a737a06 + apps (2026-10-07, svos-dev thread 01M4D1GNYBW7QVY7RF05R412V2, Prime's 2240 decision)

• Release: /opt/svos/current → releases/a737a06 (previous 6c6abbe kept: it cannot read schema 11, and the front door's previous-asset handle still points at it). Built by infra-ops from a clean git archive: 9 wheels, the web/ 0.0.6 static build, kernel-requirements.txt (uv export) and apps-requirements.txt (uv pip compile against the kernel pins, hashed), vendor/bifrost-1.2.1 (gitea vh index, sha256-verified, because PyPI has an unrelated bifrost), and hermes-plugin/svos (staged for U13). Kernel 0.0.5 migrated schema 9 → 11 on first start.
• Pre-change backups: vzdump vm/111/2026-10-08T06:02:14Z (freeze/thaw clean) and an online sqlite .backup plus config, keys and app state at /var/lib/svos/backup/pre-a737a06/ (integrity ok, user_version 9).
• Apps:

  slug	listener	unit	state
  app-interface	127.0.0.1:8403	svos-interface	up
  app-knowledge	127.0.0.1:8405	svos-knowledge	up; Albok token 01M4D22HHJ7AJHX30WPDE02DFC, vault nh3-svos/albok-token-knowledge, grants per A111 read:fleet/memory, read:fleet/stash, read:personal/notes (no wildcard, no restricted wing, no class grant)
  app-terminal	127.0.0.1:8404, base_url https://svos-terminal.phasefinal.com	svos-terminal	degraded by design: no zellij and no fleet session on this box
  app-postoffice	127.0.0.1:8406	svos-postoffice (installed, NOT enabled)	declared + enrolled, stopped until Prime declares the bridge handle; handle = "svos" in its config is the example's placeholder
  app-miranda, svos-agent	—	—	packages in apps-env only (U13)
  Credentials /etc/svos/<app>.env (root 0600); units in configs/svos-apps/; the quiesce hook lists all three new units.

• Terminal origin: svos-terminal.phasefinal.com = headscale MagicDNS extra record → 100.64.0.9, its own Caddy site (LE via DNS-01). No Basic auth (Chromium cancels Basic prompts in cross-origin frames; the lease is the page's auth). /verbs/* and /health are refused unless they come from the box itself. The kernel's signed calls reach the app through Caddy, so nh3-svos resolves the name via /etc/hosts (100.64.0.9 svos-terminal.phasefinal.com svos.phasefinal.com): it has no MagicDNS (accept-dns=false). The terminal is up only while Caddy is.
• Acceptance 2026-10-07 ~2315: 17/17 HTTPS controls through an SSH forward to the real listener (the 12 from 10-05, plus: health schema 11 with 2 up and 1 degraded; the terminal registered on its own origin; knowledge wings = exactly the 3 token wings; knowledge search hits; terminal attach refused 404 unknown seat). Chromium phone + desk: shell, inbox, stream, reload, /status, /graph, /feed, no overflow, no page errors; desk "Open Terminal here" → pane open, framed from https://svos-terminal.phasefinal.com/, body waiting (no lease). From mesh peer fv-ml1, the terminal origin returns: / 200 with frame-ancestors https://svos.phasefinal.com; lease routes 401; /verbs/* and /health 403; anything else 404.

Build (reproducible)

scripts/elway infra-ops@10.100.250.62 --playbook playbooks/nh3-pve-2-cloudinit-vm.yaml --var vmid=111 \
  --var name=nh3-svos --var ip=10.100.50.43/24 --var cores=4 --var memory=8192 --var disk=100G --var tags=svos \
  --var "description=..."
scripts/elway vh@10.100.50.43        --playbook playbooks/bootstrap-infra-ops-user.yaml
scripts/elway infra-ops@10.100.50.43 --playbook playbooks/nh3-svos-host.yaml

Front door: mesh-only https://svos.phasefinal.com, HTTP Basic auth (Prime, 2026-10-05)

• Mesh: tailscaled joined headscale as nh3-svos, 100.64.0.9, --accept-dns=false --accept-routes=false.
• Name: svos.phasefinal.com (and svos-terminal.phasefinal.com) has no public record. Mesh clients get 100.64.0.9 from the headscale MagicDNS extra records (/var/lib/headscale/extra-records.json on nh3-headscale, hot-reloaded). Since 2026-10-08 (Prime: "serve it on the LAN too and point the dns at the internal LAN address") the three internal AdGuards (ana 10.250.50.70:8053, esh 10.0.50.45, nh3 10.100.50.40) carry hand-made rewrites for both names → 10.100.50.43, outside dns-sync.py's .internal authority (like *.nh3.phasefinal.com). Verified 10-08 from LAN hosts at all three sites (nh3-dev, nh3-docker, esh-docker-vm, ana-docker): / 401, /sw.js 200, terminal / 200, terminal /health 403. nh3-svos itself still resolves both names to 100.64.0.9 via /etc/hosts, so the kernel's signed calls to the terminal keep arriving from 100.64.0.9.
• TLS: Caddy 2.11.4 (the fleet caddy-cf build with the Cloudflare DNS module) at /usr/local/bin/caddy, unit caddy-svos.service, config /etc/caddy/Caddyfile (canonical configs/svos-front/Caddyfile). It listens on 100.64.0.9:443 and, since 2026-10-08, 10.100.50.43:443 (both site blocks), with no :80 and nothing on the internet. The LE cert for svos.phasefinal.com comes via DNS-01, token nh3-svos-caddy-dns01 (phasefinal.com only), in /etc/caddy/env (root:caddy 0640), vaulted nh3-svos/cloudflare-caddy-dns01-token.
• Identity: HTTP Basic auth, one account Vuong (Prime: "Basic auth is fine, saved in browser is fine. Access is via mesh because we don't host externally." SVOS addresses him as Prime; the login is Vuong). The argon2id hash is in /etc/caddy/svos-basic-auth.accounts (root:caddy 0640, not in git), imported into basic_auth argon2id SVOS. The plaintext is vaulted at nh3-svos/basic-auth-vuong. To rotate: pipe the new password to caddy hash-password --algorithm argon2id on stdin (never --plaintext), rewrite the accounts file, secret put, then systemctl reload caddy-svos.
• Credential-free offline shell (svos-dev, 2026-10-05): the service worker fetches its hash-pinned INVENTORY with credentials:'omit', so exactly those files plus /sw.js are served BEFORE Basic auth (GET/HEAD, exact files, no fallback). Public build bytes only; /, /status, /api/*, the connector and everything else stay gated. The path list is generated into /etc/caddy/svos-public-inventory by /usr/local/sbin/svos-front-inventory (configs/svos-front/): it hash-checks every file against INVENTORY, validates, then reloads. ⚠ Run it as root after every SVOS release, or the next build's hashed asset names stay gated and worker install fails. --check changes nothing.
• Previous release's assets across a rollout: the same tool writes /etc/caddy/svos-public-previous: when /opt/svos/current has moved, the release named in the old inventory header becomes "previous", and its hashed /assets/* files that the new build lacks are served credential-free from that release's own static dir (hash-checked). One previous release at most; comments only when there is none (true at 2026-10-05, one release). Replica-tested: old asset 200 with the old INVENTORY hash, new asset with the new hash, POST to the old asset 401, unknown asset 401/404; tampered files and an unsafe path make the tool refuse.
• Hosting bounds (svos-dev, per the web contract): request bodies capped at exactly 1 MiB on every route (413 over); headers within 5 s, whole body within 15 s (Caddy read_header / read_body); no write, response or idle deadline, so /api/stream (SSE) stays open. Live 2026-10-05 2014–2021 from fv-ml1: 1 MiB+1 → 413 on /api/verbs and /verbs/pin; exactly 1 MiB passes Caddy (kernel then answers); unfinished headers closed at 5.0 s; stalled body cut at 15.0 s; a signed app-interface ping returned pong before and DURING a stalled body; an SSE stream stayed open 41–45 s (client hung up) alongside all of it. ⚠ A stalled body over HTTP/1.1 can come back as Caddy's empty 200 OK (Content-Length 0) instead of 504: replica 3/3 buffered, 2/3 unbuffered; live 2 of 3. Over h2, which browsers use, it is 502 (3/3 replica, 2/2 live). The app never gets a complete request either way; an empty 200 is not success. handle_errors 499 and request_buffers do not change it (tested).
• Static routes (gated), per svoperatingsystem web/README.md Hosting contract: / and /status → index.html; /assets/*, /icons/*, /favicon.svg, /manifest.webmanifest, /index.html, /sw.js → exact file, missing → 404; other HTML document requests (Sec-Fetch-Dest: document or Accept: text/html) → index.html; anything else → 404. Unversioned files get Cache-Control: no-cache.
• Acceptance matrix: configs/svos-front/acceptance-matrix.sh (35 checks; usage in its header). 35/35 pass live at 1957 2026-10-05 from fv-ml1; against the previous config it reports 16 FAIL (positive control).
• Headers: client copies of Tailscale-User-*, X-Forwarded-*, Tailscale-Headers-Info, X-Real-IP, Forwarded and X-SVOS-Peer are deleted before auth. After auth, Tailscale-User-Login is SET to the authenticated user, and Authorization is deleted. That deletion is load-bearing: the kernel's human path (svoperatingsystem u1-principals INV-U1-06) refuses any request carrying Authorization. As a result the front door is human-only; a bearer client cannot come through it.
• Retired 2026-10-05: the WhoIs forward_auth gate (svos-whois-auth, 127.0.0.1:8405). Every mesh node has WhoIs login pfi, so it could not tell Prime's devices from the servers without restructuring the mesh, and Prime declined that. The unit, script, /etc/svos-front/ and the repo files are removed; git history has them.
• Config changes: validate WITHOUT putting the token on a command line, then reload: sudo systemd-run --wait --pipe --collect -p User=caddy -p Group=caddy -p EnvironmentFile=/etc/caddy/env -p "Environment=XDG_DATA_HOME=/var/lib/caddy XDG_CONFIG_HOME=/var/lib/caddy" /usr/local/bin/caddy validate --config <file> --adapter caddyfile, then systemctl reload caddy-svos. (env $(cat /etc/caddy/env) caddy … leaked the first token into sudo's argv log on 2026-10-05; it was revoked.)
• Tested 2026-10-05 ~1940–1946, from mesh peer fv-ml1, against a temporary header echo on 127.0.0.1:8400 (removed after):
  • no credentials → 401 Basic realm="SVOS"; wrong password → 401; username vuong (wrong case) → 401;
  • right credentials with forged Tailscale-User-Login/-Whatever, X-Forwarded-For/Proto/Zzz, Forwarded and X-Real-IP → 200. Upstream saw exactly one Tailscale-User-Login: Vuong, no Authorization, none of the forged headers, X-Forwarded-Proto: https, Host: svos.phasefinal.com, and path /health (the /api prefix stripped);
  • connector route with right credentials → 502 (auth passed; the connector was not running); cert verifies; LAN 10.100.50.43:443 → connection refused.

Original requirements (svos-dev)

App side, per svos-dev 2026-10-05: kernel on 127.0.0.1:8400 --trust-loopback (its Unix-socket mode needs a 0700 kernel-owned dir and creates a 0666 socket, so no shared-group socket in this build), interface connector on 127.0.0.1:8403. Static UI at /opt/svos/current/static. /api/* → kernel with the prefix stripped (/api/health is the kernel's); /health, /verbs/ping|pins|pin|unpin, /openapi.json → connector unchanged. ⚠ With --trust-loopback, every local process is inside the identity trust boundary, so run no untrusted workloads here. Agreed design (option A): Caddy on the tailnet address only, forward_auth to a loopback WhoIs responder fed the real socket peer ({http.request.remote.host}:{port}), fail closed, allow-list of logins, client identity/forwarding headers stripped, Origin passed through untouched.

App listeners stay private (loopback or a Unix socket). The front door has to give the browser HTTPS: the UI and the voice adapter need a secure context for the microphone. It also has to give the kernel's adapter a verified identity in Tailscale-User-Login, with any browser-supplied copy stripped. See the svos-dev thread of 2026-10-05.