vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
56668d8862118a52a2a70aadd5a7f207128ec700
esh-pfi-infrastructure/servers/nh3-svos/README.md
T

45 KiB
Raw Blame History

nh3-svos: host for the NEW SVOS operating system (VM 111 on nh3-pve-2, NH3)

Built 2026-10-05 at svos-dev's request, relaying Prime ("stand up a dedicated VM for it on nh3-pve-2"). It runs the new SVOS (repo corviduo/svoperatingsystem): a FastAPI/uvicorn kernel (SQLite WAL, synchronous FULL, one writer), an interface connector, an optional voice adapter and a static phone/desktop UI. It is a separate instance. The existing High Seat (svos.service on nh3-dev, :8770) and Miranda are NOT migrated by this host.

Ownership split: infra-ops owns the VM, OS, access, backups and ingress. svos-dev owns the application: install, configuration, enrollment and acceptance.
Address 10.100.50.43, VLAN 50 (nh3-servers), static via cloud-init; nh3-svos.nh3.internal
Guest VM 111 on nh3-pve-2, onboot 1, tag svos
Shape 4 vCPU x86-64-v3, 8 GiB, 100 GB on vmstore (local NVMe LVM-thin, discard on). nh3-pve-2 had 47 GB RAM and 946 GB vmstore free at build
OS Debian 13 genericcloud, Python 3.13 + venv, SQLite 3.46, systemd-timesyncd (synced), qemu-guest-agent
Identities vh 1000 (operator keys + lkraven@nh3-dev, so svos-dev's session can ssh in; NOPASSWD sudo from cloud-init, plus /etc/sudoers.d/91-vh-verifypw so sudo -v agrees), infra-ops 850, svc-svos 801 (nologin; the only writer of the data root)
Data root /var/lib/svos, svc-svos 0700, on the local root ext4. Never NFS, never shared
Releases /opt/svos root:svc-svos 2775, with releases/ for svos-dev's layout. The git checkout is not a runtime dependency
Docker none, by design
Backups nh3-pve-2 vzdump job, nightly 21:30, all 1 → pbs-ana namespace nh3-pve-2 (mirrored to pbs-nh3 at 06:30), snapshot mode with guest-agent fs-freeze
App-consistent hook svos-dev's /etc/qemu/fsfreeze-hook.d/50-svos stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ Debian's qemu-ga does NOT run fsfreeze hooks unless configured (no -F in the unit, no qemu-ga.conf by default). Here /etc/qemu/qemu-ga.conf points it at /usr/local/sbin/fsfreeze-dispatch (configs/qemu-ga/fsfreeze-dispatch), which unlike the stock /etc/qemu/fsfreeze-hook propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. Live drill 2026-10-05 1952–1953 (services running): quiesced backup vm/111/2026-10-06T02:52:11Z → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary 60-drill-refuse after 50-svos, since removed) vm/111/2026-10-06T02:53:03Z → the dispatcher unwound and restarted the services, and then ⚠ vzdump did NOT abort: it logged ERROR: unable to freeze guest fs, backed up UNFROZEN (crash-consistent, app live), and reported Backup job finished successfully, rc 0. Both images mapped from PBS and mounted read-only (ro,noload) on nh3-pve-2, never booted: svos.db integrity ok (each image still carries its ~0.9 MB WAL: keep -wal with the db when restoring), interface pins.sqlite3 integrity ok, config/key/config-generation hashes equal live, table counts equal live except events. Monitoring: scripts/check-backup-freshness.sh now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict DEGRADED, exit 4 (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script
Monitoring Beszel native agent 0.18.7 (playbooks/beszel-native.yaml, configs/beszel-agent/hosts/nh3-svos.env), hub system t6y646ifa3x7xxv, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05

Release f40bf6d / v0.0.25, U23 the tree in depth — UI-only publish (2026-10-19 0855; svos-dev thread 01M5ADWS1PZ5NWRXAVJVWGYTAE)

  • Prime pushed the tag v0.0.25 → f40bf6d78d06. The watcher saw it at 0855:01 and published (current → f40bf6d + inventory). Nothing restarted.
  • Kit bd343895…: kernel 0.0.25 (a version bump only) and UI 0.0.29, handle /assets/index-DN9_msyo.js + index-CNudli_R.css; d14dc94 is the previous handle. The served bundle carries f40bf6d. Health ok, 8 up. The processes map 2f29d0d (kernel, knowledge, voice) and d14dc94 (judge, interface).
  • Rollback: current → releases/d14dc94, then svos-front-inventory.

Release d14dc94 / v0.0.24, U22 card summaries (2026-10-19 0838–0841; Prime's go 0345 via svos-dev, thread 01M59P4Z1VJ4WHJB0N8N1P743K, delivered at 0836 after the NH3 WAN outage)

  • v0.0.23 (9a0eec8) was superseded before any push. Its staged dir /opt/svos/releases/9a0eec8 is unused.
  • Kit 4259f7f9…: kernel 0.0.24 (a version bump; no kernel code change since 2f29d0d), app-judge 0.0.2 (U22 explain), app-interface 0.0.8 (U22 summaries/summary, namespace interface-summaries), UI 0.0.28 (/assets/index-83rJZtc6.js). Everything else carried from 2f29d0d.
  • Steps:
    1. nh3-svos-d14dc94-v0.0.24.yaml: backup pre-d14dc94 (svos.db, briefs, pins, settings; integrity ok); judge.toml explain_seconds = 60 (backup judge.toml.pre-d14dc94; canonical configs/svos-apps/judge.toml); current → d14dc94; front door; restart ONLY svos-judge. Card 1 (card:8bd70e4a…, app-judge rev 2) was raised 0838:44 and approved 0839:12.
    2. …-interface.yaml (refuses until the judge's revision is pinned): restart ONLY svos-interface. Card 2 (card:5fc19cd9…, app-interface rev 6) was raised 0839:43 and approved 0839:54. summaries.sqlite3 was created.
    3. Grants were raised as Prime over loopback, as before: cards 3/4 were raised 0840:51 and approved 0841:12/0841:16. Grant 294 = prime → ns:interface-summaries read (not autonomous); 295 = write (autonomous).
  • ⚠ infra-hermes ran a manual fsfreeze/thaw at 0838:38–40, in the middle of step 1. It stopped and restarted every SVOS service. The thaw brought them up from 2f29d0d (before the switch), so the interface was still on 0.0.7 and no extra card was raised. The kernel, knowledge and voice now run 2f29d0d code (identical). Manual freezes on nh3-svos are ask-first from now on.
  • Health ok, 8 up, gen d750257258f9fb68. Rollback: the v0.0.22 kit (current → 2f29d0d, inventory, restart judge + interface: each raises its card again). The summaries store is additive.

Release 2f29d0d / v0.0.22, task result files listed by path — UI-only publish (2026-10-19 0125; svos-dev thread 01M59M4T3ZWS0WKQJWZ9RE86B4)

  • Prime pushed the tag v0.0.22 → 2f29d0d667d6. The watcher saw it at 0125:01 and published: current → 2f29d0d plus the front-door inventory. Nothing restarted.
  • Kit b4de9681…: kernel 0.0.22 (a version bump only) and UI 0.0.26. Handle /assets/index-D1Rgj1De.js + /assets/index-CNudli_R.css; 45d3a48 is the previous handle. The served bundle carries 2f29d0d. Health ok, gen d750257258f9fb68, 8 up.
  • Fix: the task result view reads the host manifest's file entries by path, so task f0e9cc59's PDF lists with Open/Save. Also two hardening fixes to the working look.
  • Rollback: current → releases/45d3a48, then svos-front-inventory.

Release 45d3a48 / v0.0.21, the /tasks draft survives a hidden tab — UI-only publish (2026-10-19 0104; svos-dev thread 01M59JY4V62Z43WZ0CWZGERK8M)

  • Prime pushed main + tag v0.0.21 → 45d3a48a8919 himself. A watcher on nh3-dev saw the tag at 0104:43, checked that it named the staged commit, and ran nh3-svos-45d3a48-v0.0.21.yaml, which switched current → 45d3a48 and ran svos-front-inventory. Nothing restarted and there was no card.
  • Kit sha256 1d5ea371…: the kernel 0.0.21 wheel (a version bump only) and UI 0.0.25. UI handle /assets/index-BjJxcKQV.js + /assets/index-CNudli_R.css; 81cc5a2 is the previous handle. The served bundle carries 45d3a48. Health ok, gen d750257258f9fb68, 8 up.
  • Fix: a draft no longer dies when the page's session publishes a not-ready state (a hidden tab, a visibility change). Prime's gen-large draft at 0115 by svos-dev's clock read "The draft was interrupted".
  • Rollback: current → releases/81cc5a2, then svos-front-inventory.

Release 81cc5a2 / v0.0.20, stronger look by default + the /tasks draft wait — UI-only publish (2026-10-19 0055; Prime's "deploy it" via svos-dev, threads 01M59JF8Z900KCHP5CJ98F90BW / 01M59JHRV294CVS0BCMHQR96XY)

  • Push: Prime pushed main (12f2eb2, one docs commit past the release) and the tag v0.0.20 → 81cc5a21dfab himself.
  • Kit: kernel 0.0.20 wheel (a version bump only) and UI 0.0.24 from a clean archive of the tag; everything else carried from 91e9f2d. Kit sha256 e81fadd8…. Playbooks: nh3-svos-81cc5a2-v0.0.20-stage.yaml, then …-v0.0.20.yaml.
  • Publish: current → 81cc5a2, then svos-front-inventory. UI handle /assets/index-Cm1ek0Xu.js + /assets/index-CNudli_R.css (the CSS is unchanged); 91e9f2d is the previous handle. Nothing restarted and there was no card. The processes still map 15c2d04 (kernel, knowledge, judge, voice) and 6d4cb1c (interface).
  • What it fixes: the /tasks draft. Each of Prime's four propose calls at 0050–0053 came back ok in 12–20 s, but the page's shared request helper (web/src/lib/notification-http.ts) aborted at 10 s and showed "Outcome not confirmed". 0.0.24 waits 125 s.
  • Verified from the served bundle: the build id is 81cc5a2, and it contains 125e3. The look comes from ?look=, else stronger (read at source in Yggdrasil.svelte). Seeing it render needs a browser.
  • Rollback: current → releases/91e9f2d, then svos-front-inventory.

Release 91e9f2d / v0.0.19, U21 working look — UI-only publish (2026-10-19 0037; Prime's "push it now" in-session; svos-dev thread 01M59H5W335E6W5JP67M6Y6SE7)

  • Push: origin main already named 91e9f2d by the time of the push; infra-ops pushed the tag v0.0.19 → 91e9f2d70c5e at 0037.
  • Kit: kernel 0.0.19 wheel (a version bump only) and web build UI 0.0.23 from a clean archive of the tag. Everything else was carried byte-identical from 6d4cb1c. Kit sha256 a7688247…. Playbooks: nh3-svos-91e9f2d-v0.0.19-stage.yaml (0032), then …-v0.0.19.yaml.
  • Publish: current → 91e9f2d (the front-door inventory generates from current/static), then svos-front-inventory. UI handle: /assets/index-sEHNjI-U.js + /assets/index-CNudli_R.css (the CSS is unchanged from 0.0.22); 6d4cb1c is the previous handle. Nothing restarted: no app or manifest change, no card. Config gen db56b0ed9b743fb7; health ok, 8 up.
  • ⚠ The pointer is ahead of every process: the kernel, knowledge, judge and voice still run from releases/15c2d04, and the interface from releases/6d4cb1c. Their code is identical to 91e9f2d's. Keep both dirs until those services restart.
  • Verified: the served index.html names the new bundle, and that bundle carries 91e9f2d and data-look (default today; ?look= takes a known look). The hero's live label needs a browser.
  • Rollback: current → releases/6d4cb1c, then svos-front-inventory.

Release 6d4cb1c / v0.0.18, U20 settings (2026-10-18 2351–2352; Prime's go 2350 via svos-dev, thread 01M59D7NTZNN75SRP0M4ZQF91A, and "push it" in-session)

  • Push: the go came while gitea still had only 15c2d04 (Prime's push never landed). On his "push it", infra-ops pushed svoperatingsystem main 15c2d04..6d4cb1c (fast-forward) and tag v0.0.18 → 6d4cb1c64de0 at 2351 from the nh3-dev clone.
  • Kit: kernel 0.0.18 (a version bump, no code change), app-interface 0.0.7 wheels and web build UI 0.0.22 from a clean archive of the tag (built from the local tag before the push, same commit). Everything else was carried byte-identical from 15c2d04. Kit sha256 1aec2aa3…. Playbooks: nh3-svos-6d4cb1c-v0.0.18-stage.yaml (2324), then …-v0.0.18.yaml. No config change (gen stays 29bc36bf074b8245).
  • Cutover: online sqlite backup backup/pre-6d4cb1c (svos.db, briefs.sqlite3, pins.sqlite3, all integrity ok), current → 6d4cb1c, front door on UI 0.0.22 (15c2d04 is the previous handle), and only svos-interface restarted.
  • ⚠ The kernel process was NOT restarted. It still runs 15c2d04's code and reports kernel 0.0.17 until its next restart. That is correct: 0.0.17 and 0.0.18 differ only in the version, and nothing in U20 reads it (svos-dev asked for this note). Knowledge, judge and voice likewise run identical code from the 15c2d04 dir; keep that dir until they restart.
  • Card: card:892d7886… (manifest, app-interface revision 5, 6759c436e0be) was raised 2351:50, spoken in ~6 s, and approved by Prime through Matrix at 2352:16. The interface was down (refused stale) from 2351:53 to 2352:23. All 8 apps were up again, and the new state/apps/interface/settings.sqlite3 existed.
  • Grants (Prime's "issue the grant", 2355): infra-ops sent both POST /verbs/kernel/grant requests as Prime over loopback, with the front door's identity headers (Tailscale-User-Login: Vuong, Host, X-Forwarded-Proto, Origin, Sec-Fetch-Site) and an Idempotency-Key, because the verb is human-only and needs a key. Two cards were raised at 2355:52, and Prime approved both at 2356:09 and 2356:12. They are runtime grants: id 207 prime → ns:interface-settings write, autonomous; id 208 … read, not autonomous. Remaining: the /settings and panel checks plus his listening test (svos-dev's list).
  • Rollback: current → releases/15c2d04, run svos-front-inventory, restart svos-interface. That reverts the manifest to 0.0.6, which raises another held card. settings.sqlite3 is additive and can stay.

Release 15c2d04 / v0.0.17, seat activity (2026-10-18 2254–2255; Prime's "deploy now" 2235 via svos-dev, thread 01M595YP8R62Y665C38MHPH6YX)

  • Kit: kernel 0.0.17, app-terminal 0.0.6 wheels and web build UI 0.0.21 from a clean archive of the tag. Everything else was carried byte-identical from b51aa5e (no source change; uv.lock changed only the kernel version). Kit sha256 f6e4ff1e…. release.json now says schema 14 (the carried value said 13; there is no migration in this release). Playbooks: nh3-svos-15c2d04-v0.0.17-stage.yaml, then nh3-svos-15c2d04-v0.0.17.yaml, then nh3-dev-svos-15c2d04-v0.0.17.yaml.
  • nh3-svos: offline backup backup/pre-15c2d04 (svos.db, integrity ok, user_version 14), instance.toml.pre-15c2d04, apps/voice.toml.pre-15c2d04. The six [[seats.runtimes]] got prompt/working/window (+ the U17 comment lines) copied verbatim from docs/operations/u13-edit-payload.md @15c2d04 by a script generated from that file, inserted under each program line. Config gen 29bc36bf074b8245 (was 13241a403080788c). Kernel READY in 0.79 s. voice.toml utterance_seconds 30 → 60 (UI 0.0.21's client admits 0.1..120 s). All 8 apps up by 2254:52.
    • ⚠ The first cutover run failed at the edit: svc-svos cannot read infra-ops's 0700 stage dir, so the script is now fed on stdin. The kernel was down from 2254:08 to 2254:22 in total, including the fix and the re-run; the voice app came back last, at about 2254:36.
  • nh3-dev: /opt/svos/releases/15c2d04 = 3726d73's apps-env with svos-terminal 0.0.6 (uv pip freeze differs in that line only), current → 15c2d04, only svos-terminal restarted (2255:05). postoffice, miranda and agent still run 3726d73's identical code. terminal.toml unchanged (activity_interval_seconds defaults to 5).
  • No card: terminal manifest revision 6 changes only version (0.0.5 → 0.0.6), so the kernel pinned it at 2255:07 without a card. The expected held-manifest card never existed.
  • Verified (events table): 13 seat/activity_changed at 2255:13, the first pass after the terminal restart. ops-infra was working (this session, mid-deploy: the positive control); the 12 other live seats were waiting. dev-ldp is live with no activity event, so it stays unknown. Its claude has been running 14 days, and with both marks declared, a screen that matches neither is unknown by design. The likeliest cause is an older harness screen (reasoned, not checked). The new kernel talked to the old 0.0.5 host for 43 s without an error. The only malformed reply pair came at 2255:06, while the terminal was restarting. GET /seats was not read: it needs a principal, and infra-ops borrowed none.
  • Rollback:
    • nh3-svos: stop the apps and the kernel, restore instance.toml.pre-15c2d04 (an old kernel refuses the runtime keys) and apps/voice.toml.pre-15c2d04, current → releases/b51aa5e, start, run svos-front-inventory. svos.db needs no restore (no migration); backup/pre-15c2d04 is there if wanted.
    • nh3-dev: current → releases/3726d73, restart svos-terminal (revision 5 again; likely pinned without a card, as above).

Release b51aa5e / v0.0.16, the judge app (2026-10-18 1633–1642; Prime: "go with your picks, cut over now"; svos-dev thread 01M57XAEZVQ39DKNYXRTMC29S5)

  • Kit: kernel 0.0.16, app-judge 0.0.1 and UI 0.0.20 from a clean archive of the tag. The rest was carried from 57d90d6. Kit sha256 25f78195…. Playbooks: nh3-svos-b51aa5e-v0.0.16-stage.yaml, then …-v0.0.16.yaml, then nh3-svos-tasks-judge.yaml.
  • app-judge:
    • Listens on 127.0.0.1:8409, unit svos-judge (configs/svos-apps/svos-judge.service), and is in the quiesce hook (50-svos.pre-judge kept).
    • Config config/apps/judge.toml (configs/svos-apps/judge.toml): drafting gen-large; route * → hermes/gen-large.
    • /etc/svos/judge.env (root 0600) holds two values:
      • SVOS_JUDGE_GATEWAY_KEY: LiteLLM alias svos-app-judge, scoped to exactly gen-large, gen, gen-reasoning, gen-small, sec and sec-reasoning; vault nh3-svos/svos-judge-gateway-key.
      • SVOS_APP_CREDENTIAL: enrolled 1633 via svos-kernel mint-secret + POST /auth/enroll; vault nh3-svos/svos-judge-credential.
    • Manifest revision 1 pinned, no card.
  • Config:
    • [[apps]] app-judge, and the hermes runtime gets models = [the same six], which is the /tasks select ∩ the judge key's catalogue (gen f5330a99).
    • After Prime's three owner grants (approved 1641:04–1641:10: app-judge → app:app-interface read, app-judge → app:app-terminal read, prime → app:app-judge read), [tasks].judge = "app-judge" (gen 13241a403080788c). All 8 apps are up.
  • Card audio VERIFIED with the v0.0.15 spoken form: the three grant cards spoke in 5.5, 4.5 and 5.4 s, all accepted. Before the fix: 12 of 12 failed at 15.0 s.
  • Rollback:
    • Remove [tasks].judge (instance.toml.pre-tasks-judge), stop and disable svos-judge.
    • For the full release: stop the apps and the kernel, restore backup/pre-b51aa5e/svos.db + instance.toml.pre-b51aa5e, current → releases/57d90d6, then start.

Release 57d90d6 / v0.0.15, the short spoken summons (2026-10-17 1100–1102; Prime's wording ruling via svos-dev, thread 01M5558E19Q3ZK4RRZXQG6A2Y9)

  • Kit: kernel 0.0.15 and app-voice 0.0.2 wheels from a clean archive of the tag. Everything else, the web build included, was carried byte-identical from 3726d73 (the UI bar still reads 3726d73; its sources are unchanged). Kit sha256 ffee056d…. Playbook: nh3-svos-57d90d6-v0.0.15.yaml.
  • Spoken form: a card summons now speaks "Pending card from : . Created . Approve or countermand from your desk." with no ids and no reference. The Matrix text is unchanged. The kernel's WAV cap is 12 MiB.
  • Config: voice.toml utterance_seconds 30 → 60 (voice 0.0.2 admits 0.1..120). ⚠ REVERTED to 30 on 2026-10-18 1646: the web client's capabilities() check (voice.ts) refuses utterance_seconds > 30, so the voice panel had read "Connection not confirmed" since 10-17 1101. The open succeeded, but the client threw on /capabilities. The short spoken summons fits in 30 s. [notifications.matrix] request_timeout_seconds 60 (this morning's stopgap) → 30. Config gen 40f469b75b5d0782. No migration (schema 14) and no manifest change, so no card. Kernel READY in 0.76 s; all 7 apps up by 1101:21.
  • Backup / rollback: backup/pre-57d90d6/svos.db (integrity ok, v14), plus instance.toml.pre-57d90d6 and apps/voice.toml.pre-57d90d6. To roll back: stop the apps and the kernel, restore both configs, then current → releases/3726d73, then start.
  • ⚠ The front-door inventory header names the release dir, so regenerate it after every switch, even when the web build is identical.

Release 3726d73 / v0.0.14, the task-seat success gate (2026-10-17 0908–0910; Prime's in-session go, svos-dev thread 01M5558E19Q3ZK4RRZXQG6A2Y9)

  • Kit: kernel 0.0.14, connector base 0.0.10, app-interface 0.0.6, app-terminal 0.0.5 wheels and web build UI 0.0.19, all from a clean archive of the tag. Everything else was carried byte-identical from 33a7077 (no source change; uv.lock changed only versions). Kit sha256 380e64f6…. Staged first (nh3-svos-3726d73-v0.0.14-stage.yaml), then three phases in svos-dev's order:
    • A (nh3-svos-3726d73-v0.0.14.yaml): offline backup to backup/pre-3726d73 (schema 13 svos.db plus the v1 briefs.sqlite3, both via the sqlite backup API with integrity ok, plus the config tar and instance.toml.pre-3726d73). Kernel READY in 0.73 s, with migration 0014, now schema 14.
    • B (nh3-dev-svos-3726d73-v0.0.14.yaml): nh3-dev current → 3726d73, plus [seat_host].scan_patterns pointing at hermes's tools/threat_patterns.py (backup terminal.toml.pre-3726d73). Only svos-terminal was restarted.
    • C (…-interface.yaml): interface start (briefs v1→v2), then the front-door inventory (33a7077 becomes previous).
  • Cards: terminal revision 5 (9822cb2f…, raised 0908:53) and interface revision 4 (898902b4…, raised 0909:04). Prime approved both, at 0909:20 and 0909:27. All 7 apps were up by 0909:33.
  • Hermes query-file route (Prime, same go): task_argv is now hermes chat --query-file {task} --oneshot -Q --ignore-rules -t {toolsets} --max-turns {max_turns} --run-budget {deadline_seconds} -m {model}, with task_toolsets = ["terminal", "file"] and task_max_turns = 200 (nh3-svos-hermes-query-file-route.yaml, backup instance.toml.pre-queryfile). Config gen is now c0e9beefe9bde65b.
  • Rollback:
    • nh3-svos: stop the apps and the kernel, then restore svos.db and state/apps/interface/briefs.sqlite3 from backup/pre-3726d73, then restore instance.toml.pre-3726d73, then current → releases/33a7077, then start.
    • nh3-dev: current → releases/33a7077, then restore terminal.toml.pre-3726d73, then restart svos-terminal.
    • Both apps' older revisions would raise cards again.
  • ⚠ Card audio has failed every time: 12 of 12 synthesis_failed since 10-10, each at exactly 15.0 s. That is the kernel's [notifications.matrix] request_timeout_seconds (default 15) running out while synthesis is still going. The kernel speaks the Matrix summons text, which includes three card ids spelled aloud, so it runs far past what 15 s covers (miranda-is ≈ 28 chars of text per second; 167 chars took 5.85 s median, n=3). Stopgap 2026-10-17 0917 (svos-dev's word): request_timeout_seconds = 60 (range 15..60; config gen d69e7ae87d14ed45; revert = instance.toml.pre-matrix-timeout). Long cards still fail at the voice app's 30 s audio cap (utterance_seconds, max 30, about 480 chars). The real fix is a shorter spoken summons, a kernel patch whose wording is Prime's call. Restore the 15 s default once it ships.

Release 452a592 / v0.0.11, U15 task seats slice 1 (2026-10-13 2131–2133; Prime's 2010 go via svos-dev, thread 01M4WAGAS2Z0P8J6GZRSCRPH2Q)

  • Order: tonight's 21:30 vzdump of VM 111 finished 21:31:30, then playbooks/svos-releases/nh3-svos-452a592-v0.0.11.yaml (the kit was built in-session: kernel 0.0.11, connector base 0.0.8 and app-terminal 0.0.4 wheels from a clean archive of the tag; everything else, including the web build of 7f95e54, carried byte-identical).
  • Backup before the migration: /var/lib/svos/backup/pre-452a592/ (the schema 12 svos.db via the sqlite backup API, integrity ok, plus the config/keys/apps tar) and config/instance.toml.pre-452a592.
  • Config: a [tasks] table with scratch_root = "/var/lib/svos/apps/terminal/tasks" (all ceilings at their defaults), plus a [[notifications.artifacts]] for that root. Its host is nh3-svos, not nh3-dev: the terminal's artifact_host stayed nh3-svos through U13, and its ResultReader names results by that host, so the mapping must match it, as the injects mapping does. No runtime has task_argv yet, so a task-spawn is refused runtime_not_task_capable until Prime names the command.
  • Result: kernel 0.0.11 READY in 1.17 s, migration 0013 run, /health schema 13, config_gen 5ade279d2fb2dd1c, apps up 7.
  • nh3-dev: /opt/svos/current → releases/452a592 (terminal 0.0.4, connector 0.0.8). /var/lib/svos/apps/terminal/tasks (0700) and terminal.toml [seat_host].tasks_root (backup .pre-452a592). Only svos-terminal was restarted. Its manifest revision 4 (0.0.4, seat-host) was pinned directly, with no card. The seats read unknown for 6 s, then live. svos-miranda, svos-postoffice and svos-agent still run adc862e's code until their next restart.
  • Rollback: stop the kernel, put svos.db back from backup/pre-452a592 (schema 12), restore instance.toml.pre-452a592, current → releases/7f95e54, then start. On nh3-dev: current → releases/adc862e, restore terminal.toml.pre-452a592, then restart svos-terminal.

Release afab123 / v0.0.9 + U13 phase B (2026-10-10 1646–1648, Prime-approved 1639; svos-dev thread 01M4M2Z2H3BQH375M2G30GQHDM)

  • /opt/svos/current → releases/afab123: kernel 0.0.9, schema 11 → 12 (migration 0012, the page verb's tables; start returned READY in 0.64 s). Built from a clean archive of afab123: kernel, connector 0.0.7, postoffice 0.0.2, terminal 0.0.3, agent 0.0.3 and miranda 0.0.2 wheels; interface 0.0.4, knowledge 0.0.1 and voice 0.0.1 wheels, static, vendor, hermes-plugin, the kernel unit and apps-requirements.txt carried byte-identical from b588510 (sources unchanged, verified). /health ok, schema 12, up 4, degraded 1 (the terminal, by design). Inventory regenerated; the front door's previous-asset handle is now b588510.
  • Pre-change backups: vzdump vm/111/2026-10-10T23:45:41Z on pbs-ana (freeze/thaw clean), and an OFFLINE copy taken with every service stopped: /var/lib/svos/backup/pre-afab123/ (svos.db integrity ok, user_version 11, plus config-keys-apps.tar). Rollback: stop all, remove the kernel drop-in, restore that svos.db (keep no stale -wal/-shm), switch current back to b588510, start.
  • ⚠ app-terminal and app-postoffice are HELD on b588510 by drop-ins svos-{terminal,postoffice}.service.d/10-hold-b588510.conf (configs/svos-apps/): their v0.0.9 manifests changed, and a new version would be held for approval and refuse the kernel's calls until then (svos-dev's sequencing). Both move to nh3-dev at the U13 edit: remove the drop-ins there, and do NOT delete releases/b588510 before that. interface, knowledge and voice run afab123.
  • B2, the kernel's app origin: drop-in svos-kernel.service.d/20-app-listen.conf (configs/svos-kernel/) adds --app-listen 127.0.0.1:8410; Caddy serves https://svos-kernel.phasefinal.com on 10.100.50.43 only (own LE cert, DNS-01), no Basic auth, bearer passed, identity headers stripped, source allowlisted to nh3-dev (10.100.10.50), paths method-pinned to svoperatingsystem dc241f8 u13-svos-edit.md item 1 (the apps' and Miranda's agent connector's calls: GET /auth/kernel-key /auth/whoami /registry/apps[/*] /principals/*/grants /seats /events /artifacts/resolve; POST /auth/enroll /auth/proofs /seats/enroll /registry/apps/*/manifest; /verbs/*), else 404. The three internal AdGuards answer the name with 10.100.50.43. Tested from nh3-dev: all 15 allowed method+path pairs → 401 from the kernel (JSON detail); /health, /stream, /cards, POST /auth/kernel-key, DELETE /registry/apps/x, /principals/a/b/grants → Caddy's empty 404; from nh3-svos and nh3-docker → 403. Backups /etc/caddy/Caddyfile.pre-kernel-origin, .pre-kernel-paths. ⚠ Validate from a caddy-readable path (/etc/caddy/Caddyfile.check): the first deploy validated /tmp/svos-stage-*/ (0700 infra-ops), got "permission denied", and a | tail hid it. The config was valid (Caddy's own reload accepted it), and the validator was then positive-controlled (a bogus directive → rc 1).
  • B5 + enrollment secrets (2026-10-10 1654): svos-kernel mint-secret (run as infra-ops without sudo, so no sudo-io copy) minted three tokens straight into the vault, read back by hash: nh3-svos/app-high-seat-enrollment-secret, nh3-svos/app-miranda-enrollment-secret, nh3-svos/miranda-persona-enrollment-secret. Only app-high-seat is declared so far ([[apps]] with enrollment_sha256, one reload, config_gen 22a05c20e17f93ce → 1bd948c754e571da, backup instance.toml.pre-high-seat); the other two are declared in the edit.
  • nh3-svos stays on afab123: svoperatingsystem dc241f8 changes only svos-miranda (0.0.3, the charter), which runs on nh3-dev, so a dc241f8 release here would change nothing that runs.
  • The app-side origins on nh3-dev (svos-seathost, svos-highseat) and the bridge env are in servers/nh3-dev/README.md.

Release b588510 (2026-10-09 0540, kernel 0.0.8: the SQLite reader-pool fix)

  • /opt/svos/current → releases/b588510. Only the kernel wheel changed (0.0.8). Every other artifact was carried byte-identical from ac42ace, the notify unit is unchanged, and there was no migration and no config change. Previous-asset handle: none, because the static build is identical to ac42ace's.
  • The fd outage of 2026-10-08 22:55–23:25 is fixed at the source. Root cause (svos-dev): Store._reader kept one read-only SQLite connection per anyio worker thread, and those threads die after 10 s idle, so a browser's polling bursts leaked a connection per dead worker. Kernel 0.0.7 hit EMFILE at the default soft limit of 1024 (509 svos.db + 506 -wal fds). The stopgap drop-in svos-kernel.service.d/10-nofile-stopgap.conf (LimitNOFILE=65536, 2325 10-08 → 0540 10-09) is removed; the limit is back to 1024.
  • Acceptance: /health ok, up 4, degraded 1. A 4-min headless-Chromium desk soak (211 /api requests) gave svos.db handles 9 → 12, with one step during the first burst and then flat for 2.5 min, and -wal flat at 9. 0.0.7 under a real browser had grown +22 in 2 min. ⚠ Harness differs: headless Chromium here, Prime's browser then.
  • Thaw-fix repeats: 0 app startup failures on the 10-08 21:30 nightly thaw and on the 10-09 05:39 pre-refresh vzdump.

Release ac42ace (2026-10-08 1515, svos-dev thread 01M4ES8Q110HNH8J4WWCGE6S4K)

  • /opt/svos/current → releases/ac42ace: kernel 0.0.7 (schema 11, no migration), connector base 0.0.6, interface UI 0.0.7 (top bar with the Pacific clock; the tree as the desk's stage), svos-agent 0.0.2 (staged). Every other app wheel is unchanged, so there was no re-enrollment and no config change. Built from a clean archive, the same way as a737a06; third-party pins are unchanged. Previous = a737a06 (the front door's previous-asset handle points at it). 6c6abbe is no longer referenced and is safe to delete (it cannot read schema 11).
  • Kernel readiness pair installed together: svos-kernel.service now has Type=notify, NotifyAccess=main and TimeoutStartSec=300 (copied from the release's deployment/nh3-svos/svos-kernel.service; the old unit is kept as .pre-notify). systemctl start svos-kernel now returns only after READY=1 (0.61 s, /health 200 at once). ⚠ Install a kernel ≥ 0.0.7 and this unit as a pair, never one alone: Type=notify on an older kernel times the start out at 300 s.
  • Freeze/thaw drill, with a control: the pre-refresh vzdump on the OLD kernel (15:14) made terminal and knowledge fail once at thaw (the known race). The post-refresh vzdump (vm/111/2026-10-08T22:16:12Z, 15:16) showed 0 app startup failures; all six units came back, and /health read up 4, degraded 1 within ~2 s. The one-poll "down" for the terminal right after the restart is its first health poll, not a fault.
  • Acceptance: /health ok, up 4, degraded 1. Chromium phone + desk over the LAN door: the top bar is present with clock 1517, the desk stage shows the tree (4 stage layers + hero), no overflow, 0 page errors; desk Connect voice shows "Voice connected".

Release a737a06 + apps (2026-10-07, svos-dev thread 01M4D1GNYBW7QVY7RF05R412V2, Prime's 2240 decision)

  • Release: /opt/svos/current → releases/a737a06 (previous 6c6abbe kept: it cannot read schema 11, and the front door's previous-asset handle still points at it). Built by infra-ops from a clean git archive: 9 wheels, the web/ 0.0.6 static build, kernel-requirements.txt (uv export) and apps-requirements.txt (uv pip compile against the kernel pins, hashed), vendor/bifrost-1.2.1 (gitea vh index, sha256-verified, because PyPI has an unrelated bifrost), and hermes-plugin/svos (staged for U13). Kernel 0.0.5 migrated schema 9 → 11 on first start.
  • Pre-change backups: vzdump vm/111/2026-10-08T06:02:14Z (freeze/thaw clean) and an online sqlite .backup plus config, keys and app state at /var/lib/svos/backup/pre-a737a06/ (integrity ok, user_version 9).
  • Apps:
    slug listener unit state
    app-interface 127.0.0.1:8403 svos-interface up
    app-knowledge 127.0.0.1:8405 svos-knowledge up; Albok token 01M4D22HHJ7AJHX30WPDE02DFC, vault nh3-svos/albok-token-knowledge, grants per A111 read:fleet/memory, read:fleet/stash, read:personal/notes (no wildcard, no restricted wing, no class grant)
    app-terminal 127.0.0.1:8404, base_url https://svos-terminal.phasefinal.com svos-terminal degraded by design: no zellij and no fleet session on this box
    app-postoffice 127.0.0.1:8406 svos-postoffice (installed, NOT enabled) declared + enrolled, stopped until Prime declares the bridge handle; handle = "svos" in its config is the example's placeholder
    app-voice 127.0.0.1:8402, front door /voice/* (same origin, BEFORE Basic auth; the lease bearer is the auth) svos-voice up since 2026-10-08 (Prime: voice on); engines in config/apps/voice.toml: STT Parakeet http://10.251.50.54:8300, TTS fleet gateway http://irv-ml1.nh3.internal:8198; owner grants app:app-voice write (autonomous) + read
    app-miranda, svos-agent — — packages in apps-env only (U13)
    Credentials /etc/svos/<app>.env (root 0600), vaulted as nh3-svos/svos-<app>-credential (terminal, knowledge, postoffice, voice; 2026-10-08); units in configs/svos-apps/; the quiesce hook lists every app unit.
  • Terminal origin: svos-terminal.phasefinal.com = headscale MagicDNS extra record → 100.64.0.9, its own Caddy site (LE via DNS-01). No Basic auth (Chromium cancels Basic prompts in cross-origin frames; the lease is the page's auth). /verbs/* and /health are refused unless they come from the box itself. The kernel's signed calls reach the app through Caddy, so nh3-svos resolves the name via /etc/hosts (100.64.0.9 svos-terminal.phasefinal.com svos.phasefinal.com): it has no MagicDNS (accept-dns=false). The terminal is up only while Caddy is.
  • Acceptance 2026-10-07 ~2315: 17/17 HTTPS controls through an SSH forward to the real listener (the 12 from 10-05, plus: health schema 11 with 2 up and 1 degraded; the terminal registered on its own origin; knowledge wings = exactly the 3 token wings; knowledge search hits; terminal attach refused 404 unknown seat). Chromium phone + desk: shell, inbox, stream, reload, /status, /graph, /feed, no overflow, no page errors; desk "Open Terminal here" → pane open, framed from https://svos-terminal.phasefinal.com/, body waiting (no lease). From mesh peer fv-ml1, the terminal origin returns: / 200 with frame-ancestors https://svos.phasefinal.com; lease routes 401; /verbs/* and /health 403; anything else 404.

Build (reproducible)

scripts/elway infra-ops@10.100.250.62 --playbook playbooks/nh3-pve-2-cloudinit-vm.yaml --var vmid=111 \
  --var name=nh3-svos --var ip=10.100.50.43/24 --var cores=4 --var memory=8192 --var disk=100G --var tags=svos \
  --var "description=..."
scripts/elway vh@10.100.50.43        --playbook playbooks/bootstrap-infra-ops-user.yaml
scripts/elway infra-ops@10.100.50.43 --playbook playbooks/nh3-svos-host.yaml

Front door: mesh-only https://svos.phasefinal.com, HTTP Basic auth (Prime, 2026-10-05)

  • Mesh: tailscaled joined headscale as nh3-svos, 100.64.0.9, --accept-dns=false --accept-routes=false.
  • Name: svos.phasefinal.com (and svos-terminal.phasefinal.com) has no public record. Mesh clients get 100.64.0.9 from the headscale MagicDNS extra records (/var/lib/headscale/extra-records.json on nh3-headscale, hot-reloaded). Since 2026-10-08 (Prime: "serve it on the LAN too and point the dns at the internal LAN address") the three internal AdGuards (ana 10.250.50.70:8053, esh 10.0.50.45, nh3 10.100.50.40) carry hand-made rewrites for both names → 10.100.50.43, outside dns-sync.py's .internal authority (like *.nh3.phasefinal.com). Verified 10-08 from LAN hosts at all three sites (nh3-dev, nh3-docker, esh-docker-vm, ana-docker): / 401, /sw.js 200, terminal / 200, terminal /health 403. nh3-svos itself still resolves both names to 100.64.0.9 via /etc/hosts, so the kernel's signed calls to the terminal keep arriving from 100.64.0.9.
  • TLS: Caddy 2.11.4 (the fleet caddy-cf build with the Cloudflare DNS module) at /usr/local/bin/caddy, unit caddy-svos.service, config /etc/caddy/Caddyfile (canonical configs/svos-front/Caddyfile). It listens on 100.64.0.9:443 and, since 2026-10-08, 10.100.50.43:443 (both site blocks), with no :80 and nothing on the internet. The LE cert for svos.phasefinal.com comes via DNS-01, token nh3-svos-caddy-dns01 (phasefinal.com only), in /etc/caddy/env (root:caddy 0640), vaulted nh3-svos/cloudflare-caddy-dns01-token.
  • Identity: HTTP Basic auth, one account Vuong (Prime: "Basic auth is fine, saved in browser is fine. Access is via mesh because we don't host externally." SVOS addresses him as Prime; the login is Vuong). The argon2id hash is in /etc/caddy/svos-basic-auth.accounts (root:caddy 0640, not in git), imported into basic_auth argon2id SVOS. The plaintext is vaulted at nh3-svos/basic-auth-vuong. To rotate: pipe the new password to caddy hash-password --algorithm argon2id on stdin (never --plaintext), rewrite the accounts file, secret put, then systemctl reload caddy-svos.
  • Credential-free offline shell (svos-dev, 2026-10-05): the service worker fetches its hash-pinned INVENTORY with credentials:'omit', so exactly those files plus /sw.js are served BEFORE Basic auth (GET/HEAD, exact files, no fallback). Public build bytes only; /, /status, /api/*, the connector and everything else stay gated. The path list is generated into /etc/caddy/svos-public-inventory by /usr/local/sbin/svos-front-inventory (configs/svos-front/): it hash-checks every file against INVENTORY, validates, then reloads. ⚠ Run it as root after every SVOS release, or the next build's hashed asset names stay gated and worker install fails. --check changes nothing.
  • Previous release's assets across a rollout: the same tool writes /etc/caddy/svos-public-previous: when /opt/svos/current has moved, the release named in the old inventory header becomes "previous", and its hashed /assets/* files that the new build lacks are served credential-free from that release's own static dir (hash-checked). One previous release at most; comments only when there is none (true at 2026-10-05, one release). Replica-tested: old asset 200 with the old INVENTORY hash, new asset with the new hash, POST to the old asset 401, unknown asset 401/404; tampered files and an unsafe path make the tool refuse.
  • Hosting bounds (svos-dev, per the web contract): request bodies capped at exactly 1 MiB on every route (413 over); headers within 5 s, whole body within 15 s (Caddy read_header / read_body); no write, response or idle deadline, so /api/stream (SSE) stays open. Live 2026-10-05 2014–2021 from fv-ml1: 1 MiB+1 → 413 on /api/verbs and /verbs/pin; exactly 1 MiB passes Caddy (kernel then answers); unfinished headers closed at 5.0 s; stalled body cut at 15.0 s; a signed app-interface ping returned pong before and DURING a stalled body; an SSE stream stayed open 41–45 s (client hung up) alongside all of it. ⚠ A stalled body over HTTP/1.1 can come back as Caddy's empty 200 OK (Content-Length 0) instead of 504: replica 3/3 buffered, 2/3 unbuffered; live 2 of 3. Over h2, which browsers use, it is 502 (3/3 replica, 2/2 live). The app never gets a complete request either way; an empty 200 is not success. handle_errors 499 and request_buffers do not change it (tested).
  • Static routes (gated), per svoperatingsystem web/README.md Hosting contract: / and /status → index.html; /assets/*, /icons/*, /favicon.svg, /manifest.webmanifest, /index.html, /sw.js → exact file, missing → 404; other HTML document requests (Sec-Fetch-Dest: document or Accept: text/html) → index.html; anything else → 404. Unversioned files get Cache-Control: no-cache.
  • Acceptance matrix: configs/svos-front/acceptance-matrix.sh (35 checks; usage in its header). 35/35 pass live at 1957 2026-10-05 from fv-ml1; against the previous config it reports 16 FAIL (positive control).
  • Headers: client copies of Tailscale-User-*, X-Forwarded-*, Tailscale-Headers-Info, X-Real-IP, Forwarded and X-SVOS-Peer are deleted before auth. After auth, Tailscale-User-Login is SET to the authenticated user, and Authorization is deleted. That deletion is load-bearing: the kernel's human path (svoperatingsystem u1-principals INV-U1-06) refuses any request carrying Authorization. As a result the front door is human-only; a bearer client cannot come through it.
  • Retired 2026-10-05: the WhoIs forward_auth gate (svos-whois-auth, 127.0.0.1:8405). Every mesh node has WhoIs login pfi, so it could not tell Prime's devices from the servers without restructuring the mesh, and Prime declined that. The unit, script, /etc/svos-front/ and the repo files are removed; git history has them.
  • Config changes: validate WITHOUT putting the token on a command line, then reload: sudo systemd-run --wait --pipe --collect -p User=caddy -p Group=caddy -p EnvironmentFile=/etc/caddy/env -p "Environment=XDG_DATA_HOME=/var/lib/caddy XDG_CONFIG_HOME=/var/lib/caddy" /usr/local/bin/caddy validate --config <file> --adapter caddyfile, then systemctl reload caddy-svos. (env $(cat /etc/caddy/env) caddy … leaked the first token into sudo's argv log on 2026-10-05; it was revoked.)
  • Tested 2026-10-05 ~1940–1946, from mesh peer fv-ml1, against a temporary header echo on 127.0.0.1:8400 (removed after):
    • no credentials → 401 Basic realm="SVOS"; wrong password → 401; username vuong (wrong case) → 401;
    • right credentials with forged Tailscale-User-Login/-Whatever, X-Forwarded-For/Proto/Zzz, Forwarded and X-Real-IP → 200. Upstream saw exactly one Tailscale-User-Login: Vuong, no Authorization, none of the forged headers, X-Forwarded-Proto: https, Host: svos.phasefinal.com, and path /health (the /api prefix stripped);
    • connector route with right credentials → 502 (auth passed; the connector was not running); cert verifies; LAN 10.100.50.43:443 → connection refused.

Original requirements (svos-dev)

App side, per svos-dev 2026-10-05: kernel on 127.0.0.1:8400 --trust-loopback (its Unix-socket mode needs a 0700 kernel-owned dir and creates a 0666 socket, so no shared-group socket in this build), interface connector on 127.0.0.1:8403. Static UI at /opt/svos/current/static. /api/* → kernel with the prefix stripped (/api/health is the kernel's); /health, /verbs/ping|pins|pin|unpin, /openapi.json → connector unchanged. ⚠ With --trust-loopback, every local process is inside the identity trust boundary, so run no untrusted workloads here. Agreed design (option A): Caddy on the tailnet address only, forward_auth to a loopback WhoIs responder fed the real socket peer ({http.request.remote.host}:{port}), fail closed, allow-list of logins, client identity/forwarding headers stripped, Origin passed through untouched.

App listeners stay private (loopback or a Unix socket). The front door has to give the browser HTTPS: the UI and the voice adapter need a secure context for the microphone. It also has to give the kernel's adapter a verified identity in Tailscale-User-Login, with any browser-supplied copy stripped. See the svos-dev thread of 2026-10-05.

Reference in New Issue View Git Blame Copy Permalink