esh-pfi-infrastructure/servers/nh3-svos/README.md

nh3-svos: host for the NEW SVOS operating system (VM 111 on nh3-pve-2, NH3)

Built 2026-10-05 at svos-dev's request, relaying Prime ("stand up a dedicated VM for it on nh3-pve-2"). It runs the new SVOS (repo corviduo/svoperatingsystem): a FastAPI/uvicorn kernel (SQLite WAL, synchronous FULL, one writer), an interface connector, an optional voice adapter and a static phone/desktop UI. It is a separate instance. The existing High Seat (svos.service on nh3-dev, :8770) and Miranda are NOT migrated by this host.

Ownership split: infra-ops owns the VM, OS, access, backups and ingress. svos-dev owns the application: install, configuration, enrollment and acceptance.

Address	10.100.50.43, VLAN 50 (nh3-servers), static via cloud-init; nh3-svos.nh3.internal
Guest	VM 111 on nh3-pve-2, onboot 1, tag svos
Shape	4 vCPU x86-64-v3, 8 GiB, 100 GB on vmstore (local NVMe LVM-thin, discard on). nh3-pve-2 had 47 GB RAM and 946 GB vmstore free at build
OS	Debian 13 genericcloud, Python 3.13 + venv, SQLite 3.46, systemd-timesyncd (synced), qemu-guest-agent
Identities	vh 1000 (operator keys + lkraven@nh3-dev, so svos-dev's session can ssh in; NOPASSWD sudo from cloud-init, plus /etc/sudoers.d/91-vh-verifypw so sudo -v agrees), infra-ops 850, svc-svos 801 (nologin; the only writer of the data root)
Data root	/var/lib/svos, svc-svos 0700, on the local root ext4. Never NFS, never shared
Releases	/opt/svos root:svc-svos 2775, with releases/ for svos-dev's layout. The git checkout is not a runtime dependency
Docker	none, by design
Backups	nh3-pve-2 vzdump job, nightly 21:30, all 1 → pbs-ana namespace nh3-pve-2 (mirrored to pbs-nh3 at 06:30), snapshot mode with guest-agent fs-freeze
App-consistent hook	svos-dev's /etc/qemu/fsfreeze-hook.d/50-svos stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ Debian's qemu-ga does NOT run fsfreeze hooks unless configured (no -F in the unit, no qemu-ga.conf by default). Here /etc/qemu/qemu-ga.conf points it at /usr/local/sbin/fsfreeze-dispatch (configs/qemu-ga/fsfreeze-dispatch), which unlike the stock /etc/qemu/fsfreeze-hook propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. Live drill 2026-10-05 1952–1953 (services running): quiesced backup vm/111/2026-10-06T02:52:11Z → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary 60-drill-refuse after 50-svos, since removed) vm/111/2026-10-06T02:53:03Z → the dispatcher unwound and restarted the services, and then ⚠ vzdump did NOT abort: it logged ERROR: unable to freeze guest fs, backed up UNFROZEN (crash-consistent, app live), and reported Backup job finished successfully, rc 0. Both images mapped from PBS and mounted read-only (ro,noload) on nh3-pve-2, never booted: svos.db integrity ok (each image still carries its ~0.9 MB WAL: keep -wal with the db when restoring), interface pins.sqlite3 integrity ok, config/key/config-generation hashes equal live, table counts equal live except events. Monitoring: scripts/check-backup-freshness.sh now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict DEGRADED, exit 4 (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script
Monitoring	Beszel native agent 0.18.7 (playbooks/beszel-native.yaml, configs/beszel-agent/hosts/nh3-svos.env), hub system t6y646ifa3x7xxv, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05

Build (reproducible)

scripts/elway infra-ops@10.100.250.62 --playbook playbooks/nh3-pve-2-cloudinit-vm.yaml --var vmid=111 \
  --var name=nh3-svos --var ip=10.100.50.43/24 --var cores=4 --var memory=8192 --var disk=100G --var tags=svos \
  --var "description=..."
scripts/elway vh@10.100.50.43        --playbook playbooks/bootstrap-infra-ops-user.yaml
scripts/elway infra-ops@10.100.50.43 --playbook playbooks/nh3-svos-host.yaml

Front door: mesh-only https://svos.phasefinal.com, HTTP Basic auth (Prime, 2026-10-05)

• Mesh: tailscaled joined headscale as nh3-svos, 100.64.0.9, --accept-dns=false --accept-routes=false.
• Name: svos.phasefinal.com exists ONLY as a headscale MagicDNS extra record (/var/lib/headscale/extra-records.json on nh3-headscale, hot-reloaded) → 100.64.0.9. There is no public record and no internal AdGuard record.
• TLS: Caddy 2.11.4 (the fleet caddy-cf build with the Cloudflare DNS module) at /usr/local/bin/caddy, unit caddy-svos.service, config /etc/caddy/Caddyfile (canonical configs/svos-front/Caddyfile). It listens only on 100.64.0.9:443, with no :80. The LE cert for svos.phasefinal.com comes via DNS-01, token nh3-svos-caddy-dns01 (phasefinal.com only), in /etc/caddy/env (root:caddy 0640), vaulted nh3-svos/cloudflare-caddy-dns01-token.
• Identity: HTTP Basic auth, one account Vuong (Prime: "Basic auth is fine, saved in browser is fine. Access is via mesh because we don't host externally." SVOS addresses him as Prime; the login is Vuong). The argon2id hash is in /etc/caddy/svos-basic-auth.accounts (root:caddy 0640, not in git), imported into basic_auth argon2id SVOS. The plaintext is vaulted at nh3-svos/basic-auth-vuong. To rotate: pipe the new password to caddy hash-password --algorithm argon2id on stdin (never --plaintext), rewrite the accounts file, secret put, then systemctl reload caddy-svos.
• Credential-free offline shell (svos-dev, 2026-10-05): the service worker fetches its hash-pinned INVENTORY with credentials:'omit', so exactly those files plus /sw.js are served BEFORE Basic auth (GET/HEAD, exact files, no fallback). Public build bytes only; /, /status, /api/*, the connector and everything else stay gated. The path list is generated into /etc/caddy/svos-public-inventory by /usr/local/sbin/svos-front-inventory (configs/svos-front/): it hash-checks every file against INVENTORY, validates, then reloads. ⚠ Run it as root after every SVOS release, or the next build's hashed asset names stay gated and worker install fails. --check changes nothing.
• Previous release's assets across a rollout: the same tool writes /etc/caddy/svos-public-previous: when /opt/svos/current has moved, the release named in the old inventory header becomes "previous", and its hashed /assets/* files that the new build lacks are served credential-free from that release's own static dir (hash-checked). One previous release at most; comments only when there is none (true at 2026-10-05, one release). Replica-tested: old asset 200 with the old INVENTORY hash, new asset with the new hash, POST to the old asset 401, unknown asset 401/404; tampered files and an unsafe path make the tool refuse.
• Hosting bounds (svos-dev, per the web contract): request bodies capped at exactly 1 MiB on every route (413 over); headers within 5 s, whole body within 15 s (Caddy read_header / read_body); no write, response or idle deadline, so /api/stream (SSE) stays open. Live 2026-10-05 2014–2021 from fv-ml1: 1 MiB+1 → 413 on /api/verbs and /verbs/pin; exactly 1 MiB passes Caddy (kernel then answers); unfinished headers closed at 5.0 s; stalled body cut at 15.0 s; a signed app-interface ping returned pong before and DURING a stalled body; an SSE stream stayed open 41–45 s (client hung up) alongside all of it. ⚠ A stalled body over HTTP/1.1 can come back as Caddy's empty 200 OK (Content-Length 0) instead of 504: replica 3/3 buffered, 2/3 unbuffered; live 2 of 3. Over h2, which browsers use, it is 502 (3/3 replica, 2/2 live). The app never gets a complete request either way; an empty 200 is not success. handle_errors 499 and request_buffers do not change it (tested).
• Static routes (gated), per svoperatingsystem web/README.md Hosting contract: / and /status → index.html; /assets/*, /icons/*, /favicon.svg, /manifest.webmanifest, /index.html, /sw.js → exact file, missing → 404; other HTML document requests (Sec-Fetch-Dest: document or Accept: text/html) → index.html; anything else → 404. Unversioned files get Cache-Control: no-cache.
• Acceptance matrix: configs/svos-front/acceptance-matrix.sh (35 checks; usage in its header). 35/35 pass live at 1957 2026-10-05 from fv-ml1; against the previous config it reports 16 FAIL (positive control).
• Headers: client copies of Tailscale-User-*, X-Forwarded-*, Tailscale-Headers-Info, X-Real-IP, Forwarded and X-SVOS-Peer are deleted before auth. After auth, Tailscale-User-Login is SET to the authenticated user, and Authorization is deleted. That deletion is load-bearing: the kernel's human path (svoperatingsystem u1-principals INV-U1-06) refuses any request carrying Authorization. As a result the front door is human-only; a bearer client cannot come through it.
• Retired 2026-10-05: the WhoIs forward_auth gate (svos-whois-auth, 127.0.0.1:8405). Every mesh node has WhoIs login pfi, so it could not tell Prime's devices from the servers without restructuring the mesh, and Prime declined that. The unit, script, /etc/svos-front/ and the repo files are removed; git history has them.
• Config changes: validate WITHOUT putting the token on a command line, then reload: sudo systemd-run --wait --pipe --collect -p User=caddy -p Group=caddy -p EnvironmentFile=/etc/caddy/env -p "Environment=XDG_DATA_HOME=/var/lib/caddy XDG_CONFIG_HOME=/var/lib/caddy" /usr/local/bin/caddy validate --config <file> --adapter caddyfile, then systemctl reload caddy-svos. (env $(cat /etc/caddy/env) caddy … leaked the first token into sudo's argv log on 2026-10-05; it was revoked.)
• Tested 2026-10-05 ~1940–1946, from mesh peer fv-ml1, against a temporary header echo on 127.0.0.1:8400 (removed after):
  • no credentials → 401 Basic realm="SVOS"; wrong password → 401; username vuong (wrong case) → 401;
  • right credentials with forged Tailscale-User-Login/-Whatever, X-Forwarded-For/Proto/Zzz, Forwarded and X-Real-IP → 200. Upstream saw exactly one Tailscale-User-Login: Vuong, no Authorization, none of the forged headers, X-Forwarded-Proto: https, Host: svos.phasefinal.com, and path /health (the /api prefix stripped);
  • connector route with right credentials → 502 (auth passed; the connector was not running); cert verifies; LAN 10.100.50.43:443 → connection refused.

Original requirements (svos-dev)

App side, per svos-dev 2026-10-05: kernel on 127.0.0.1:8400 --trust-loopback (its Unix-socket mode needs a 0700 kernel-owned dir and creates a 0666 socket, so no shared-group socket in this build), interface connector on 127.0.0.1:8403. Static UI at /opt/svos/current/static. /api/* → kernel with the prefix stripped (/api/health is the kernel's); /health, /verbs/ping|pins|pin|unpin, /openapi.json → connector unchanged. ⚠ With --trust-loopback, every local process is inside the identity trust boundary, so run no untrusted workloads here. Agreed design (option A): Caddy on the tailnet address only, forward_auth to a loopback WhoIs responder fed the real socket peer ({http.request.remote.host}:{port}), fail closed, allow-list of logins, client identity/forwarding headers stripped, Origin passed through untouched.

App listeners stay private (loopback or a Unix socket). The front door has to give the browser HTTPS: the UI and the voice adapter need a secure context for the microphone. It also has to give the kernel's adapter a verified identity in Tailscale-User-Login, with any browser-supplied copy stripped. See the svos-dev thread of 2026-10-05.