esh-pfi-infrastructure/servers/nh3-svos/README.md

nh3-svos: host for the NEW SVOS operating system (VM 111 on nh3-pve-2, NH3)

Built 2026-10-05 at svos-dev's request, relaying Prime ("stand up a dedicated VM for it on nh3-pve-2"). It runs the new SVOS (repo corviduo/svoperatingsystem): a FastAPI/uvicorn kernel (SQLite WAL, synchronous FULL, one writer), an interface connector, an optional voice adapter and a static phone/desktop UI. It is a separate instance. The existing High Seat (svos.service on nh3-dev, :8770) and Miranda are NOT migrated by this host.

Ownership split: infra-ops owns the VM, OS, access, backups and ingress. svos-dev owns the application: install, configuration, enrollment and acceptance.

Address	10.100.50.43, VLAN 50 (nh3-servers), static via cloud-init; nh3-svos.nh3.internal
Guest	VM 111 on nh3-pve-2, onboot 1, tag svos
Shape	4 vCPU x86-64-v3, 8 GiB, 100 GB on vmstore (local NVMe LVM-thin, discard on). nh3-pve-2 had 47 GB RAM and 946 GB vmstore free at build
OS	Debian 13 genericcloud, Python 3.13 + venv, SQLite 3.46, systemd-timesyncd (synced), qemu-guest-agent
Identities	vh 1000 (operator keys + lkraven@nh3-dev, so svos-dev's session can ssh in; NOPASSWD sudo from cloud-init, plus /etc/sudoers.d/91-vh-verifypw so sudo -v agrees), infra-ops 850, svc-svos 801 (nologin; the only writer of the data root)
Data root	/var/lib/svos, svc-svos 0700, on the local root ext4. Never NFS, never shared
Releases	/opt/svos root:svc-svos 2775, with releases/ for svos-dev's layout. The git checkout is not a runtime dependency
Docker	none, by design
Backups	nh3-pve-2 vzdump job, nightly 21:30, all 1 → pbs-ana namespace nh3-pve-2 (mirrored to pbs-nh3 at 06:30), snapshot mode with guest-agent fs-freeze
App-consistent hook	svos-dev's /etc/qemu/fsfreeze-hook.d/50-svos stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ Debian's qemu-ga does NOT run fsfreeze hooks unless configured (no -F in the unit, no qemu-ga.conf by default). Here /etc/qemu/qemu-ga.conf points it at /usr/local/sbin/fsfreeze-dispatch (configs/qemu-ga/fsfreeze-dispatch), which unlike the stock /etc/qemu/fsfreeze-hook propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. Live drill 2026-10-05 1952–1953 (services running): quiesced backup vm/111/2026-10-06T02:52:11Z → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary 60-drill-refuse after 50-svos, since removed) vm/111/2026-10-06T02:53:03Z → the dispatcher unwound and restarted the services, and then ⚠ vzdump did NOT abort: it logged ERROR: unable to freeze guest fs, backed up UNFROZEN (crash-consistent, app live), and reported Backup job finished successfully, rc 0. Both images mapped from PBS and mounted read-only (ro,noload) on nh3-pve-2, never booted: svos.db integrity ok (each image still carries its ~0.9 MB WAL: keep -wal with the db when restoring), interface pins.sqlite3 integrity ok, config/key/config-generation hashes equal live, table counts equal live except events. Monitoring: scripts/check-backup-freshness.sh now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict DEGRADED, exit 4 (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script
Monitoring	Beszel native agent 0.18.7 (playbooks/beszel-native.yaml, configs/beszel-agent/hosts/nh3-svos.env), hub system t6y646ifa3x7xxv, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05

Release 3726d73 / v0.0.14, the task-seat success gate (2026-10-17 0908–0910; Prime's in-session go, svos-dev thread 01M5558E19Q3ZK4RRZXQG6A2Y9)

• Kit: kernel 0.0.14, connector base 0.0.10, app-interface 0.0.6, app-terminal 0.0.5 wheels and web build UI 0.0.19, all from a clean archive of the tag. Everything else was carried byte-identical from 33a7077 (no source change; uv.lock changed only versions). Kit sha256 380e64f6…. Staged first (nh3-svos-3726d73-v0.0.14-stage.yaml), then three phases in svos-dev's order:
  • A (nh3-svos-3726d73-v0.0.14.yaml): offline backup to backup/pre-3726d73 (schema 13 svos.db plus the v1 briefs.sqlite3, both via the sqlite backup API with integrity ok, plus the config tar and instance.toml.pre-3726d73). Kernel READY in 0.73 s, with migration 0014, now schema 14.
  • B (nh3-dev-svos-3726d73-v0.0.14.yaml): nh3-dev current → 3726d73, plus [seat_host].scan_patterns pointing at hermes's tools/threat_patterns.py (backup terminal.toml.pre-3726d73). Only svos-terminal was restarted.
  • C (…-interface.yaml): interface start (briefs v1→v2), then the front-door inventory (33a7077 becomes previous).
• Cards: terminal revision 5 (9822cb2f…, raised 0908:53) and interface revision 4 (898902b4…, raised 0909:04). Prime approved both, at 0909:20 and 0909:27. All 7 apps were up by 0909:33.
• Hermes query-file route (Prime, same go): task_argv is now hermes chat --query-file {task} --oneshot -Q --ignore-rules -t {toolsets} --max-turns {max_turns} --run-budget {deadline_seconds} -m {model}, with task_toolsets = ["terminal", "file"] and task_max_turns = 200 (nh3-svos-hermes-query-file-route.yaml, backup instance.toml.pre-queryfile). Config gen is now c0e9beefe9bde65b.
• Rollback:
  • nh3-svos: stop the apps and the kernel, then restore svos.db and state/apps/interface/briefs.sqlite3 from backup/pre-3726d73, then restore instance.toml.pre-3726d73, then current → releases/33a7077, then start.
  • nh3-dev: current → releases/33a7077, then restore terminal.toml.pre-3726d73, then restart svos-terminal.
  • Both apps' older revisions would raise cards again.
• ⚠ Card audio has failed every time: 12 of 12 synthesis_failed since 10-10, each at exactly 15.0 s. That is the kernel's [notifications.matrix] request_timeout_seconds (default 15) running out while synthesis is still going. The kernel speaks the Matrix summons text, which includes three card ids spelled aloud, so it runs far past what 15 s covers (miranda-is ≈ 28 chars of text per second; 167 chars took 5.85 s median, n=3). Stopgap 2026-10-17 0917 (svos-dev's word): request_timeout_seconds = 60 (range 15..60; config gen d69e7ae87d14ed45; revert = instance.toml.pre-matrix-timeout). Long cards still fail at the voice app's 30 s audio cap (utterance_seconds, max 30, about 480 chars). The real fix is a shorter spoken summons, a kernel patch whose wording is Prime's call. Restore the 15 s default once it ships.

Release 452a592 / v0.0.11, U15 task seats slice 1 (2026-10-13 2131–2133; Prime's 2010 go via svos-dev, thread 01M4WAGAS2Z0P8J6GZRSCRPH2Q)

• Order: tonight's 21:30 vzdump of VM 111 finished 21:31:30, then playbooks/svos-releases/nh3-svos-452a592-v0.0.11.yaml (the kit was built in-session: kernel 0.0.11, connector base 0.0.8 and app-terminal 0.0.4 wheels from a clean archive of the tag; everything else, including the web build of 7f95e54, carried byte-identical).
• Backup before the migration: /var/lib/svos/backup/pre-452a592/ (the schema 12 svos.db via the sqlite backup API, integrity ok, plus the config/keys/apps tar) and config/instance.toml.pre-452a592.
• Config: a [tasks] table with scratch_root = "/var/lib/svos/apps/terminal/tasks" (all ceilings at their defaults), plus a [[notifications.artifacts]] for that root. Its host is nh3-svos, not nh3-dev: the terminal's artifact_host stayed nh3-svos through U13, and its ResultReader names results by that host, so the mapping must match it, as the injects mapping does. No runtime has task_argv yet, so a task-spawn is refused runtime_not_task_capable until Prime names the command.
• Result: kernel 0.0.11 READY in 1.17 s, migration 0013 run, /health schema 13, config_gen 5ade279d2fb2dd1c, apps up 7.
• nh3-dev: /opt/svos/current → releases/452a592 (terminal 0.0.4, connector 0.0.8). /var/lib/svos/apps/terminal/tasks (0700) and terminal.toml [seat_host].tasks_root (backup .pre-452a592). Only svos-terminal was restarted. Its manifest revision 4 (0.0.4, seat-host) was pinned directly, with no card. The seats read unknown for 6 s, then live. svos-miranda, svos-postoffice and svos-agent still run adc862e's code until their next restart.
• Rollback: stop the kernel, put svos.db back from backup/pre-452a592 (schema 12), restore instance.toml.pre-452a592, current → releases/7f95e54, then start. On nh3-dev: current → releases/adc862e, restore terminal.toml.pre-452a592, then restart svos-terminal.

Release afab123 / v0.0.9 + U13 phase B (2026-10-10 1646–1648, Prime-approved 1639; svos-dev thread 01M4M2Z2H3BQH375M2G30GQHDM)

• /opt/svos/current → releases/afab123: kernel 0.0.9, schema 11 → 12 (migration 0012, the page verb's tables; start returned READY in 0.64 s). Built from a clean archive of afab123: kernel, connector 0.0.7, postoffice 0.0.2, terminal 0.0.3, agent 0.0.3 and miranda 0.0.2 wheels; interface 0.0.4, knowledge 0.0.1 and voice 0.0.1 wheels, static, vendor, hermes-plugin, the kernel unit and apps-requirements.txt carried byte-identical from b588510 (sources unchanged, verified). /health ok, schema 12, up 4, degraded 1 (the terminal, by design). Inventory regenerated; the front door's previous-asset handle is now b588510.
• Pre-change backups: vzdump vm/111/2026-10-10T23:45:41Z on pbs-ana (freeze/thaw clean), and an OFFLINE copy taken with every service stopped: /var/lib/svos/backup/pre-afab123/ (svos.db integrity ok, user_version 11, plus config-keys-apps.tar). Rollback: stop all, remove the kernel drop-in, restore that svos.db (keep no stale -wal/-shm), switch current back to b588510, start.
• ⚠ app-terminal and app-postoffice are HELD on b588510 by drop-ins svos-{terminal,postoffice}.service.d/10-hold-b588510.conf (configs/svos-apps/): their v0.0.9 manifests changed, and a new version would be held for approval and refuse the kernel's calls until then (svos-dev's sequencing). Both move to nh3-dev at the U13 edit: remove the drop-ins there, and do NOT delete releases/b588510 before that. interface, knowledge and voice run afab123.
• B2, the kernel's app origin: drop-in svos-kernel.service.d/20-app-listen.conf (configs/svos-kernel/) adds --app-listen 127.0.0.1:8410; Caddy serves https://svos-kernel.phasefinal.com on 10.100.50.43 only (own LE cert, DNS-01), no Basic auth, bearer passed, identity headers stripped, source allowlisted to nh3-dev (10.100.10.50), paths method-pinned to svoperatingsystem dc241f8 u13-svos-edit.md item 1 (the apps' and Miranda's agent connector's calls: GET /auth/kernel-key /auth/whoami /registry/apps[/*] /principals/*/grants /seats /events /artifacts/resolve; POST /auth/enroll /auth/proofs /seats/enroll /registry/apps/*/manifest; /verbs/*), else 404. The three internal AdGuards answer the name with 10.100.50.43. Tested from nh3-dev: all 15 allowed method+path pairs → 401 from the kernel (JSON detail); /health, /stream, /cards, POST /auth/kernel-key, DELETE /registry/apps/x, /principals/a/b/grants → Caddy's empty 404; from nh3-svos and nh3-docker → 403. Backups /etc/caddy/Caddyfile.pre-kernel-origin, .pre-kernel-paths. ⚠ Validate from a caddy-readable path (/etc/caddy/Caddyfile.check): the first deploy validated /tmp/svos-stage-*/ (0700 infra-ops), got "permission denied", and a | tail hid it. The config was valid (Caddy's own reload accepted it), and the validator was then positive-controlled (a bogus directive → rc 1).
• B5 + enrollment secrets (2026-10-10 1654): svos-kernel mint-secret (run as infra-ops without sudo, so no sudo-io copy) minted three tokens straight into the vault, read back by hash: nh3-svos/app-high-seat-enrollment-secret, nh3-svos/app-miranda-enrollment-secret, nh3-svos/miranda-persona-enrollment-secret. Only app-high-seat is declared so far ([[apps]] with enrollment_sha256, one reload, config_gen 22a05c20e17f93ce → 1bd948c754e571da, backup instance.toml.pre-high-seat); the other two are declared in the edit.
• nh3-svos stays on afab123: svoperatingsystem dc241f8 changes only svos-miranda (0.0.3, the charter), which runs on nh3-dev, so a dc241f8 release here would change nothing that runs.
• The app-side origins on nh3-dev (svos-seathost, svos-highseat) and the bridge env are in servers/nh3-dev/README.md.

Release b588510 (2026-10-09 0540, kernel 0.0.8: the SQLite reader-pool fix)

• /opt/svos/current → releases/b588510. Only the kernel wheel changed (0.0.8). Every other artifact was carried byte-identical from ac42ace, the notify unit is unchanged, and there was no migration and no config change. Previous-asset handle: none, because the static build is identical to ac42ace's.
• The fd outage of 2026-10-08 22:55–23:25 is fixed at the source. Root cause (svos-dev): Store._reader kept one read-only SQLite connection per anyio worker thread, and those threads die after 10 s idle, so a browser's polling bursts leaked a connection per dead worker. Kernel 0.0.7 hit EMFILE at the default soft limit of 1024 (509 svos.db + 506 -wal fds). The stopgap drop-in svos-kernel.service.d/10-nofile-stopgap.conf (LimitNOFILE=65536, 2325 10-08 → 0540 10-09) is removed; the limit is back to 1024.
• Acceptance: /health ok, up 4, degraded 1. A 4-min headless-Chromium desk soak (211 /api requests) gave svos.db handles 9 → 12, with one step during the first burst and then flat for 2.5 min, and -wal flat at 9. 0.0.7 under a real browser had grown +22 in 2 min. ⚠ Harness differs: headless Chromium here, Prime's browser then.
• Thaw-fix repeats: 0 app startup failures on the 10-08 21:30 nightly thaw and on the 10-09 05:39 pre-refresh vzdump.

Release ac42ace (2026-10-08 1515, svos-dev thread 01M4ES8Q110HNH8J4WWCGE6S4K)

• /opt/svos/current → releases/ac42ace: kernel 0.0.7 (schema 11, no migration), connector base 0.0.6, interface UI 0.0.7 (top bar with the Pacific clock; the tree as the desk's stage), svos-agent 0.0.2 (staged). Every other app wheel is unchanged, so there was no re-enrollment and no config change. Built from a clean archive, the same way as a737a06; third-party pins are unchanged. Previous = a737a06 (the front door's previous-asset handle points at it). 6c6abbe is no longer referenced and is safe to delete (it cannot read schema 11).
• Kernel readiness pair installed together: svos-kernel.service now has Type=notify, NotifyAccess=main and TimeoutStartSec=300 (copied from the release's deployment/nh3-svos/svos-kernel.service; the old unit is kept as .pre-notify). systemctl start svos-kernel now returns only after READY=1 (0.61 s, /health 200 at once). ⚠ Install a kernel ≥ 0.0.7 and this unit as a pair, never one alone: Type=notify on an older kernel times the start out at 300 s.
• Freeze/thaw drill, with a control: the pre-refresh vzdump on the OLD kernel (15:14) made terminal and knowledge fail once at thaw (the known race). The post-refresh vzdump (vm/111/2026-10-08T22:16:12Z, 15:16) showed 0 app startup failures; all six units came back, and /health read up 4, degraded 1 within ~2 s. The one-poll "down" for the terminal right after the restart is its first health poll, not a fault.
• Acceptance: /health ok, up 4, degraded 1. Chromium phone + desk over the LAN door: the top bar is present with clock 1517, the desk stage shows the tree (4 stage layers + hero), no overflow, 0 page errors; desk Connect voice shows "Voice connected".

Release a737a06 + apps (2026-10-07, svos-dev thread 01M4D1GNYBW7QVY7RF05R412V2, Prime's 2240 decision)

• Release: /opt/svos/current → releases/a737a06 (previous 6c6abbe kept: it cannot read schema 11, and the front door's previous-asset handle still points at it). Built by infra-ops from a clean git archive: 9 wheels, the web/ 0.0.6 static build, kernel-requirements.txt (uv export) and apps-requirements.txt (uv pip compile against the kernel pins, hashed), vendor/bifrost-1.2.1 (gitea vh index, sha256-verified, because PyPI has an unrelated bifrost), and hermes-plugin/svos (staged for U13). Kernel 0.0.5 migrated schema 9 → 11 on first start.
• Pre-change backups: vzdump vm/111/2026-10-08T06:02:14Z (freeze/thaw clean) and an online sqlite .backup plus config, keys and app state at /var/lib/svos/backup/pre-a737a06/ (integrity ok, user_version 9).
• Apps:

  slug	listener	unit	state
  app-interface	127.0.0.1:8403	svos-interface	up
  app-knowledge	127.0.0.1:8405	svos-knowledge	up; Albok token 01M4D22HHJ7AJHX30WPDE02DFC, vault nh3-svos/albok-token-knowledge, grants per A111 read:fleet/memory, read:fleet/stash, read:personal/notes (no wildcard, no restricted wing, no class grant)
  app-terminal	127.0.0.1:8404, base_url https://svos-terminal.phasefinal.com	svos-terminal	degraded by design: no zellij and no fleet session on this box
  app-postoffice	127.0.0.1:8406	svos-postoffice (installed, NOT enabled)	declared + enrolled, stopped until Prime declares the bridge handle; handle = "svos" in its config is the example's placeholder
  app-voice	127.0.0.1:8402, front door /voice/* (same origin, BEFORE Basic auth; the lease bearer is the auth)	svos-voice	up since 2026-10-08 (Prime: voice on); engines in config/apps/voice.toml: STT Parakeet http://10.251.50.54:8300, TTS fleet gateway http://irv-ml1.nh3.internal:8198; owner grants app:app-voice write (autonomous) + read
  app-miranda, svos-agent	—	—	packages in apps-env only (U13)
  Credentials /etc/svos/<app>.env (root 0600), vaulted as nh3-svos/svos-<app>-credential (terminal, knowledge, postoffice, voice; 2026-10-08); units in configs/svos-apps/; the quiesce hook lists every app unit.

• Terminal origin: svos-terminal.phasefinal.com = headscale MagicDNS extra record → 100.64.0.9, its own Caddy site (LE via DNS-01). No Basic auth (Chromium cancels Basic prompts in cross-origin frames; the lease is the page's auth). /verbs/* and /health are refused unless they come from the box itself. The kernel's signed calls reach the app through Caddy, so nh3-svos resolves the name via /etc/hosts (100.64.0.9 svos-terminal.phasefinal.com svos.phasefinal.com): it has no MagicDNS (accept-dns=false). The terminal is up only while Caddy is.
• Acceptance 2026-10-07 ~2315: 17/17 HTTPS controls through an SSH forward to the real listener (the 12 from 10-05, plus: health schema 11 with 2 up and 1 degraded; the terminal registered on its own origin; knowledge wings = exactly the 3 token wings; knowledge search hits; terminal attach refused 404 unknown seat). Chromium phone + desk: shell, inbox, stream, reload, /status, /graph, /feed, no overflow, no page errors; desk "Open Terminal here" → pane open, framed from https://svos-terminal.phasefinal.com/, body waiting (no lease). From mesh peer fv-ml1, the terminal origin returns: / 200 with frame-ancestors https://svos.phasefinal.com; lease routes 401; /verbs/* and /health 403; anything else 404.

Build (reproducible)

scripts/elway infra-ops@10.100.250.62 --playbook playbooks/nh3-pve-2-cloudinit-vm.yaml --var vmid=111 \
  --var name=nh3-svos --var ip=10.100.50.43/24 --var cores=4 --var memory=8192 --var disk=100G --var tags=svos \
  --var "description=..."
scripts/elway vh@10.100.50.43        --playbook playbooks/bootstrap-infra-ops-user.yaml
scripts/elway infra-ops@10.100.50.43 --playbook playbooks/nh3-svos-host.yaml

Front door: mesh-only https://svos.phasefinal.com, HTTP Basic auth (Prime, 2026-10-05)

• Mesh: tailscaled joined headscale as nh3-svos, 100.64.0.9, --accept-dns=false --accept-routes=false.
• Name: svos.phasefinal.com (and svos-terminal.phasefinal.com) has no public record. Mesh clients get 100.64.0.9 from the headscale MagicDNS extra records (/var/lib/headscale/extra-records.json on nh3-headscale, hot-reloaded). Since 2026-10-08 (Prime: "serve it on the LAN too and point the dns at the internal LAN address") the three internal AdGuards (ana 10.250.50.70:8053, esh 10.0.50.45, nh3 10.100.50.40) carry hand-made rewrites for both names → 10.100.50.43, outside dns-sync.py's .internal authority (like *.nh3.phasefinal.com). Verified 10-08 from LAN hosts at all three sites (nh3-dev, nh3-docker, esh-docker-vm, ana-docker): / 401, /sw.js 200, terminal / 200, terminal /health 403. nh3-svos itself still resolves both names to 100.64.0.9 via /etc/hosts, so the kernel's signed calls to the terminal keep arriving from 100.64.0.9.
• TLS: Caddy 2.11.4 (the fleet caddy-cf build with the Cloudflare DNS module) at /usr/local/bin/caddy, unit caddy-svos.service, config /etc/caddy/Caddyfile (canonical configs/svos-front/Caddyfile). It listens on 100.64.0.9:443 and, since 2026-10-08, 10.100.50.43:443 (both site blocks), with no :80 and nothing on the internet. The LE cert for svos.phasefinal.com comes via DNS-01, token nh3-svos-caddy-dns01 (phasefinal.com only), in /etc/caddy/env (root:caddy 0640), vaulted nh3-svos/cloudflare-caddy-dns01-token.
• Identity: HTTP Basic auth, one account Vuong (Prime: "Basic auth is fine, saved in browser is fine. Access is via mesh because we don't host externally." SVOS addresses him as Prime; the login is Vuong). The argon2id hash is in /etc/caddy/svos-basic-auth.accounts (root:caddy 0640, not in git), imported into basic_auth argon2id SVOS. The plaintext is vaulted at nh3-svos/basic-auth-vuong. To rotate: pipe the new password to caddy hash-password --algorithm argon2id on stdin (never --plaintext), rewrite the accounts file, secret put, then systemctl reload caddy-svos.
• Credential-free offline shell (svos-dev, 2026-10-05): the service worker fetches its hash-pinned INVENTORY with credentials:'omit', so exactly those files plus /sw.js are served BEFORE Basic auth (GET/HEAD, exact files, no fallback). Public build bytes only; /, /status, /api/*, the connector and everything else stay gated. The path list is generated into /etc/caddy/svos-public-inventory by /usr/local/sbin/svos-front-inventory (configs/svos-front/): it hash-checks every file against INVENTORY, validates, then reloads. ⚠ Run it as root after every SVOS release, or the next build's hashed asset names stay gated and worker install fails. --check changes nothing.
• Previous release's assets across a rollout: the same tool writes /etc/caddy/svos-public-previous: when /opt/svos/current has moved, the release named in the old inventory header becomes "previous", and its hashed /assets/* files that the new build lacks are served credential-free from that release's own static dir (hash-checked). One previous release at most; comments only when there is none (true at 2026-10-05, one release). Replica-tested: old asset 200 with the old INVENTORY hash, new asset with the new hash, POST to the old asset 401, unknown asset 401/404; tampered files and an unsafe path make the tool refuse.
• Hosting bounds (svos-dev, per the web contract): request bodies capped at exactly 1 MiB on every route (413 over); headers within 5 s, whole body within 15 s (Caddy read_header / read_body); no write, response or idle deadline, so /api/stream (SSE) stays open. Live 2026-10-05 2014–2021 from fv-ml1: 1 MiB+1 → 413 on /api/verbs and /verbs/pin; exactly 1 MiB passes Caddy (kernel then answers); unfinished headers closed at 5.0 s; stalled body cut at 15.0 s; a signed app-interface ping returned pong before and DURING a stalled body; an SSE stream stayed open 41–45 s (client hung up) alongside all of it. ⚠ A stalled body over HTTP/1.1 can come back as Caddy's empty 200 OK (Content-Length 0) instead of 504: replica 3/3 buffered, 2/3 unbuffered; live 2 of 3. Over h2, which browsers use, it is 502 (3/3 replica, 2/2 live). The app never gets a complete request either way; an empty 200 is not success. handle_errors 499 and request_buffers do not change it (tested).
• Static routes (gated), per svoperatingsystem web/README.md Hosting contract: / and /status → index.html; /assets/*, /icons/*, /favicon.svg, /manifest.webmanifest, /index.html, /sw.js → exact file, missing → 404; other HTML document requests (Sec-Fetch-Dest: document or Accept: text/html) → index.html; anything else → 404. Unversioned files get Cache-Control: no-cache.
• Acceptance matrix: configs/svos-front/acceptance-matrix.sh (35 checks; usage in its header). 35/35 pass live at 1957 2026-10-05 from fv-ml1; against the previous config it reports 16 FAIL (positive control).
• Headers: client copies of Tailscale-User-*, X-Forwarded-*, Tailscale-Headers-Info, X-Real-IP, Forwarded and X-SVOS-Peer are deleted before auth. After auth, Tailscale-User-Login is SET to the authenticated user, and Authorization is deleted. That deletion is load-bearing: the kernel's human path (svoperatingsystem u1-principals INV-U1-06) refuses any request carrying Authorization. As a result the front door is human-only; a bearer client cannot come through it.
• Retired 2026-10-05: the WhoIs forward_auth gate (svos-whois-auth, 127.0.0.1:8405). Every mesh node has WhoIs login pfi, so it could not tell Prime's devices from the servers without restructuring the mesh, and Prime declined that. The unit, script, /etc/svos-front/ and the repo files are removed; git history has them.
• Config changes: validate WITHOUT putting the token on a command line, then reload: sudo systemd-run --wait --pipe --collect -p User=caddy -p Group=caddy -p EnvironmentFile=/etc/caddy/env -p "Environment=XDG_DATA_HOME=/var/lib/caddy XDG_CONFIG_HOME=/var/lib/caddy" /usr/local/bin/caddy validate --config <file> --adapter caddyfile, then systemctl reload caddy-svos. (env $(cat /etc/caddy/env) caddy … leaked the first token into sudo's argv log on 2026-10-05; it was revoked.)
• Tested 2026-10-05 ~1940–1946, from mesh peer fv-ml1, against a temporary header echo on 127.0.0.1:8400 (removed after):
  • no credentials → 401 Basic realm="SVOS"; wrong password → 401; username vuong (wrong case) → 401;
  • right credentials with forged Tailscale-User-Login/-Whatever, X-Forwarded-For/Proto/Zzz, Forwarded and X-Real-IP → 200. Upstream saw exactly one Tailscale-User-Login: Vuong, no Authorization, none of the forged headers, X-Forwarded-Proto: https, Host: svos.phasefinal.com, and path /health (the /api prefix stripped);
  • connector route with right credentials → 502 (auth passed; the connector was not running); cert verifies; LAN 10.100.50.43:443 → connection refused.

Original requirements (svos-dev)

App side, per svos-dev 2026-10-05: kernel on 127.0.0.1:8400 --trust-loopback (its Unix-socket mode needs a 0700 kernel-owned dir and creates a 0666 socket, so no shared-group socket in this build), interface connector on 127.0.0.1:8403. Static UI at /opt/svos/current/static. /api/* → kernel with the prefix stripped (/api/health is the kernel's); /health, /verbs/ping|pins|pin|unpin, /openapi.json → connector unchanged. ⚠ With --trust-loopback, every local process is inside the identity trust boundary, so run no untrusted workloads here. Agreed design (option A): Caddy on the tailnet address only, forward_auth to a loopback WhoIs responder fed the real socket peer ({http.request.remote.host}:{port}), fail closed, allow-list of logins, client identity/forwarding headers stripped, Origin passed through untouched.

App listeners stay private (loopback or a Unix socket). The front door has to give the browser HTTPS: the UI and the voice adapter need a secure context for the microphone. It also has to give the kernel's adapter a verified identity in Tailscale-User-Login, with any browser-supplied copy stripped. See the svos-dev thread of 2026-10-05.