ops(svos): U13 phase B — nh3-svos on v0.0.9 (schema 12), kernel app origin, seat-host and high-seat origins, bridge env on nh3-dev

Prime approved svos-dev's U13 pre-edit ask (2026-10-10 1639).
- B1: release afab123 on nh3-svos (kernel 0.0.9, migration 11->12) after a
  vzdump and an offline svos.db copy; app-terminal and app-postoffice held on
  b588510 by drop-ins until the edit.
- B2: --app-listen 127.0.0.1:8410 (drop-in) fronted by Caddy as
  svos-kernel.phasefinal.com on the LAN, allowlisted to nh3-dev; the path list
  adds /auth/kernel-key, which every connector client needs. DNS rewrite on
  the three internal AdGuards.
- B3/B4: svos-seathost (-> :8404) and svos-highseat (-> :8773) on nh3-dev's
  fleet Caddy, allowlisted to nh3-svos; /opt/svos apps env on nh3-dev with
  svos-postoffice[push] (althing-core 3.7.0 from althing 7a2a1c1) and
  svos-terminal. The fleet Caddyfile is now tracked in configs/.

configs/fleet-tls-caddy/Caddyfile

@@ -0,0 +1,55 @@
+{
+	email lkraven@lkraven.com
+}
+
+# Fleet internal TLS — caddy is the WILDCARD CERT AUTHORITY for *.nh3.phasefinal.com
+# (publicly-trusted LE via Cloudflare DNS-01, renewed automatically). Consumers:
+#   * self-terminate with a copy of this cert (talk: ~/talk/certs, kept fresh by
+#     fleet-tls-cert-check.timer), OR
+#   * front here via a reverse_proxy handle block on @<tool> host <tool>.nh3.phasefinal.com
+*.nh3.phasefinal.com {
+	tls {
+		dns cloudflare {env.CF_API_TOKEN}
+		resolvers 1.1.1.1 8.8.8.8
+	}
+	# Scriberr (fv-ml1:8080, plain HTTP). Fronted so the browser recorder gets a SecureContext:
+	# getUserMedia is refused on http:// origins ("Failed to initialize recorder"). 2026-10-01.
+	# Scriberr's ALLOWED_ORIGINS must list this name (fv-ml1 /opt/docker/compose/scriberr/.env).
+	@scriberr host scriberr.nh3.phasefinal.com
+	handle @scriberr {
+		reverse_proxy 10.251.50.54:8080 {
+			flush_interval -1
+		}
+	}
+	# SVOS U13 (2026-10-10, Prime-approved; svoperatingsystem docs/operations/u13-svos-edit.md "Before the edit" 2,
+	# svos docs/u13-cutover-runbook.md B3/B4). Two app origins on nh3-dev's LOOPBACK, reachable only from nh3-svos
+	# (10.100.50.43), whose kernel signs every call it makes to them. Until each app runs (the U13 edit) they answer 502.
+	# - svos-seathost: app-terminal, which moves to nh3-dev at the edit (svos-terminal, listener 127.0.0.1:8404).
+	#   The kernel's instance.toml names this as app-terminal's base_url. No buffering (the app has long-lived routes).
+	# - svos-highseat: app-high-seat, the High Seat's signed-only listener (127.0.0.1:8773, enabled at the edit's E1).
+	@svos_seathost host svos-seathost.nh3.phasefinal.com
+	handle @svos_seathost {
+		@not_nh3_svos not remote_ip 10.100.50.43
+		handle @not_nh3_svos {
+			respond 403
+		}
+		handle {
+			reverse_proxy 127.0.0.1:8404 {
+				flush_interval -1
+			}
+		}
+	}
+	@svos_highseat host svos-highseat.nh3.phasefinal.com
+	handle @svos_highseat {
+		@not_nh3_svos not remote_ip 10.100.50.43
+		handle @not_nh3_svos {
+			respond 403
+		}
+		handle {
+			reverse_proxy 127.0.0.1:8773
+		}
+	}
+	handle {
+		respond "nh3 fleet TLS authority — no tool fronted here (talk self-terminates on :8092)" 404
+	}
+}

configs/svos-apps/svos-postoffice.service.d/10-hold-b588510.conf

@@ -0,0 +1,8 @@
+# /etc/systemd/system/svos-postoffice.service.d/10-hold-b588510.conf on nh3-svos (U13 B1, 2026-10-10).
+# TEMPORARY: keeps app-postoffice on release b588510's code while the kernel runs v0.0.9 (afab123). Its v0.0.9 manifest
+# changed, and a new version would re-register, be held for the owner's approval, and refuse the kernel's calls as
+# wrong-revision until then (svos-dev's sequencing). The app moves to nh3-dev at the U13 edit; REMOVE this drop-in
+# there, and keep /opt/svos/releases/b588510 on disk until it is gone.
+[Service]
+ExecStart=
+ExecStart=/opt/svos/releases/b588510/apps-env/bin/svos-postoffice

configs/svos-apps/svos-terminal.service.d/10-hold-b588510.conf

@@ -0,0 +1,8 @@
+# /etc/systemd/system/svos-terminal.service.d/10-hold-b588510.conf on nh3-svos (U13 B1, 2026-10-10).
+# TEMPORARY: keeps app-terminal on release b588510's code while the kernel runs v0.0.9 (afab123). Its v0.0.9 manifest
+# changed, and a new version would re-register, be held for the owner's approval, and refuse the kernel's calls as
+# wrong-revision until then (svos-dev's sequencing). The app moves to nh3-dev at the U13 edit; REMOVE this drop-in
+# there, and keep /opt/svos/releases/b588510 on disk until it is gone.
+[Service]
+ExecStart=
+ExecStart=/opt/svos/releases/b588510/apps-env/bin/svos-terminal

configs/svos-front/Caddyfile

@@ -212,3 +212,47 @@ https://svos-terminal.phasefinal.com:443 {
 		}
 	}
 }
+
+# The kernel's app origin (U13 B2, 2026-10-10, Prime-approved; svoperatingsystem U9a and docs/operations/u13-svos-edit.md
+# "Before the edit" 1). Apps that run OFF this box reach the kernel here: from the U13 edit, app-terminal,
+# app-postoffice (the bridge) and app-high-seat, all on nh3-dev. LAN only, because nh3-dev is not on the mesh; the
+# internal AdGuards answer svos-kernel.phasefinal.com -> 10.100.50.43 and no MagicDNS record exists.
+# - Upstream is the kernel's SECOND listener, `serve --app-listen 127.0.0.1:8410` (configs/svos-kernel/20-app-listen.conf):
+#   a request there is never a human and loopback trust never applies, so every call authenticates with its own bearer.
+#   NO Basic auth, and Authorization passes through untouched.
+# - Source allowlisted to nh3-dev (10.100.10.50); anything else gets 403 from Caddy.
+# - Paths: svos-dev's app surface (/auth/enroll, /auth/proofs, /registry/*, /verbs/*, /seats/enroll) plus
+#   /auth/kernel-key, which every svos_connector client fetches to trust the kernel's signatures (client.kernel_key; it
+#   is also the joint runbook's B2 check: refused by the KERNEL with 401 without a credential). Anything else 404.
+# - Client identity/forwarding headers are deleted as on the other sites; the same 1 MiB body cap; nothing streams.
+https://svos-kernel.phasefinal.com:443 {
+	bind 10.100.50.43
+	tls {
+		dns cloudflare {env.CF_API_TOKEN}
+		resolvers 1.1.1.1 8.8.8.8
+	}
+	route {
+		request_header -Tailscale-User-*
+		request_header -X-Forwarded-*
+		request_header -Tailscale-Headers-Info
+		request_header -X-Real-IP
+		request_header -Forwarded
+		request_header -X-SVOS-Peer
+
+		request_body {
+			max_size 1MiB
+		}
+
+		@not_nh3_dev not remote_ip 10.100.10.50
+		handle @not_nh3_dev {
+			respond 403
+		}
+		@app_surface path /auth/enroll /auth/proofs /auth/kernel-key /registry/* /verbs/* /seats/enroll
+		handle @app_surface {
+			reverse_proxy 127.0.0.1:8410
+		}
+		handle {
+			respond 404
+		}
+	}
+}

configs/svos-kernel/20-app-listen.conf

@@ -0,0 +1,9 @@
+# /etc/systemd/system/svos-kernel.service.d/20-app-listen.conf on nh3-svos (U13 B2, 2026-10-10, Prime-approved).
+# Adds the kernel's second listener for apps OFF the box (kernel >= 0.0.9, svoperatingsystem U9a). A request on
+# 127.0.0.1:8410 is never a human and loopback trust never applies there; Caddy fronts it as svos-kernel.phasefinal.com
+# on the LAN, allowlisted to nh3-dev (configs/svos-front/Caddyfile). The primary listener is unchanged. The ExecStart
+# below is the release unit's (deployment/nh3-svos/svos-kernel.service) plus --app-listen; keep the two in step when
+# that unit changes. serve refuses --app-listen on the primary listener's port.
+[Service]
+ExecStart=
+ExecStart=/opt/svos/current/kernel-env/bin/svos-kernel serve --data-root /var/lib/svos --host 127.0.0.1 --port 8400 --trust-loopback --app-listen 127.0.0.1:8410

persistent-memory.md

@@ -118,7 +118,7 @@ _As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections ca
 ### 2026-10-09: live now (as of 2026-10-09 ~1955 PT)
 
 - **Nothing is mid-action.** No claims held (`scripts/ops-log claims`). eshpfi is pushed through this snapshot's commit. ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- <paths>` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose (auto-regenerated).
-- **Approved, NOT built (do not start before Prime says so after this snapshot):** SVOS app-traffic ingress for U13. Details, release (`afab123`, v0.0.9, schema 12) and gotchas are in the 2026-10-09 Recent-decisions entry; svos-dev's edit doc is `docs/operations/u13-svos-edit.md`; highseat-dev owns the schedule.
+- **U13 phase B (infra) DONE 2026-10-10 1646–1650 on Prime's go:** nh3-svos on v0.0.9 (`afab123`, schema 12) with app-terminal + app-postoffice HELD on b588510 by drop-ins (remove at the edit; keep `releases/b588510` until then); kernel app origin `svos-kernel.phasefinal.com` (LAN, nh3-dev only); `svos-seathost` + `svos-highseat` origins on nh3-dev (nh3-svos only); bridge env `/opt/svos/current/apps-env` on nh3-dev. Next is the atomic edit (highseat-dev's runbook `svos/docs/u13-cutover-runbook.md`, Prime's go; E2 is S→I). → `servers/nh3-svos/README.md`, `servers/nh3-dev/README.md`
 - **Owed by me (no input needed):**
   - Destroy `zroot/ROOT/debian@pre-patch-20261008` on fv-ml1 ~10-15 if nothing regressed.
   - nh3-svos housekeeping: releases `6c6abbe` and `a737a06` are no longer referenced by the front door and can be deleted (`ac42ace` holds the same static as the current `b588510`).
@@ -186,6 +186,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
 
 ## Recent decisions
 
+- `[2026-10-10]` **U13 phase B built (Prime 1639: "implement as asked")**: B1 kernel 0.0.9 + schema 12 (vzdump `vm/111/2026-10-10T23:45:41Z` + offline `pre-afab123` copy first), B2 `svos-kernel.phasefinal.com` → `--app-listen 127.0.0.1:8410` (allowlist nh3-dev; `/auth/kernel-key` ADDED to svos-dev's path list because every connector client fetches it), B3 seat-host origin + bridge env on nh3-dev, B4 app-high-seat origin. Kernel answers 401 `no principal` through the origin; other sources 403.
 - `[2026-10-10]` **esh-nas-pve `nvme/ROOT@pre-pve9` destroyed 0750 (607M)** after a week clean on PVE 9.2.21 / 7.0.14-20 (up 6 d, no err-level journal in 7 d, pools healthy, all guests up; only `openipmi.service` failed, the known item). Left: `nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z` (now 1.43G, a pre-PVE-9 rollback point that is useless as one) and `nvme/subvol-105-disk-0@pre-vaapi-20260911`.
 - `[2026-10-09]` **vh gets every albok read group (Prime 2244: "vh gets albok-everything")**: albok-read, albok-personal and albok-feedback (the restricted wing's default-deny ends for vh only), plus his four Macs' keys on vh (`playbooks/nh3-albok-vh-viewer.yaml`). No net new root path: those Macs already reach root there via lkraven@nh3-dev and the infra-ops key. Later restricted groups (1513+) are NOT included.
 - `[2026-10-09]` **nh3-dev: orphaned svos pytest killed 2242** (14.4 GB RSS, +2.3 GB/h, swap 100%, Beszel 90% alert; SIGTERM ignored, SIGKILL; used 25.4 → 11.3 GB). highseat-dev's retry-loop test with an instant fake sleep; they added pytest-timeout 300 s and a 50-attempt bound.

servers/nh3-dev/README.md

@@ -176,6 +176,23 @@ partition steps skip themselves now; growpart + resize2fs do the work, online).
   (their code; the model endpoint + key live in server env only — unit is `0600`,
   not committed).
 
+## SVOS U13 phase B on this box (2026-10-10, Prime-approved; infra-ops)
+
+The new SVOS kernel runs on nh3-svos; at the U13 edit three of its apps run HERE. Prepared, nothing running yet:
+
+- **Origins on the fleet Caddy** (`fleet-tls-caddy`, canonical `configs/fleet-tls-caddy/Caddyfile`, backup
+  `~/.config/caddy-fleet/Caddyfile.bak-20261010-u13`), both reachable **only from nh3-svos** (10.100.50.43; else 403)
+  and both 502 until their app listens:
+  - `https://svos-seathost.nh3.phasefinal.com` → `127.0.0.1:8404`, app-terminal's listener (unbuffered).
+  - `https://svos-highseat.nh3.phasefinal.com` → `127.0.0.1:8773`, app-high-seat (the High Seat's signed-only listener).
+- **Apps env `/opt/svos/current/apps-env`** (→ `releases/afab123`, lkraven-owned): python 3.13.2 copied to
+  `/opt/svos/python/` (so it does not depend on `~/.local/share/uv`), the release's hashed `apps-requirements.txt`, then
+  svos-connector 0.0.7, **svos-postoffice 0.0.2 with its `push` extra satisfied by althing-core 3.7.0** installed
+  (non-editable) from a clean archive of althing `7a2a1c1`, and svos-terminal 0.0.3. ⚠ althing-core's own deps
+  (python-ulid 4.0.1, pyyaml 6.0.3) are resolved, not hash-pinned. No units yet: the bridge and terminal units, their
+  config and their credentials are written at the edit. The kernel is reached at `https://svos-kernel.phasefinal.com`
+  (LAN, allowlisted to this box). No input firewall here, so a LAN listener for the bridge needs no rule.
+
 ## Box-wide Playwright / Chromium (2026-06-04)
 
 Available to **every user/project** on this box — no per-home `playwright install`:

servers/nh3-svos/README.md

@@ -23,6 +23,32 @@ configuration, enrollment and acceptance.
 | App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events`. **Monitoring:** `scripts/check-backup-freshness.sh` now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict **DEGRADED, exit 4** (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script |
 | Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 |
 
+## Release afab123 / v0.0.9 + U13 phase B (2026-10-10 1646–1648, Prime-approved 1639; svos-dev thread `01M4M2Z2H3BQH375M2G30GQHDM`)
+
+- **`/opt/svos/current` → `releases/afab123`: kernel 0.0.9, schema 11 → 12** (migration 0012, the page verb's tables;
+  start returned READY in 0.64 s). Built from a clean archive of afab123: kernel, connector 0.0.7, postoffice 0.0.2,
+  terminal 0.0.3, agent 0.0.3 and miranda 0.0.2 wheels; interface 0.0.4, knowledge 0.0.1 and voice 0.0.1 wheels,
+  static, vendor, hermes-plugin, the kernel unit and `apps-requirements.txt` carried byte-identical from b588510
+  (sources unchanged, verified). `/health` ok, schema 12, up 4, degraded 1 (the terminal, by design). Inventory
+  regenerated; the front door's previous-asset handle is now b588510.
+- **Pre-change backups:** vzdump `vm/111/2026-10-10T23:45:41Z` on pbs-ana (freeze/thaw clean), and an OFFLINE copy
+  taken with every service stopped: `/var/lib/svos/backup/pre-afab123/` (`svos.db` integrity ok, user_version 11, plus
+  `config-keys-apps.tar`). **Rollback:** stop all, remove the kernel drop-in, restore that `svos.db` (keep no stale
+  `-wal`/`-shm`), switch `current` back to b588510, start.
+- ⚠ **app-terminal and app-postoffice are HELD on b588510** by drop-ins `svos-{terminal,postoffice}.service.d/10-hold-b588510.conf`
+  (`configs/svos-apps/`): their v0.0.9 manifests changed, and a new version would be held for approval and refuse the
+  kernel's calls until then (svos-dev's sequencing). Both move to nh3-dev at the U13 edit: **remove the drop-ins
+  there, and do NOT delete `releases/b588510` before that.** interface, knowledge and voice run afab123.
+- **B2, the kernel's app origin:** drop-in `svos-kernel.service.d/20-app-listen.conf` (`configs/svos-kernel/`) adds
+  `--app-listen 127.0.0.1:8410`; Caddy serves `https://svos-kernel.phasefinal.com` on **10.100.50.43 only** (own LE
+  cert, DNS-01), no Basic auth, bearer passed, identity headers stripped, **source allowlisted to nh3-dev** (10.100.10.50),
+  paths `/auth/enroll`, `/auth/proofs`, `/auth/kernel-key`, `/registry/*`, `/verbs/*`, `/seats/enroll`, else 404.
+  `/auth/kernel-key` is ADDED to svos-dev's list: every `svos_connector` client fetches it to trust the kernel's
+  signatures, and it is the runbook's B2 check. The three internal AdGuards answer the name with 10.100.50.43.
+  Tested: from nh3-dev `/auth/kernel-key` and `/registry/apps` → **401 `no principal` from the kernel**, `/health` and
+  `/api/health` → 404; from nh3-svos and nh3-docker → 403. Caddyfile backup `/etc/caddy/Caddyfile.pre-kernel-origin`.
+- The app-side origins on nh3-dev (`svos-seathost`, `svos-highseat`) and the bridge env are in `servers/nh3-dev/README.md`.
+
 ## Release b588510 (2026-10-09 0540, kernel 0.0.8: the SQLite reader-pool fix)
 
 - `/opt/svos/current` → `releases/b588510`. Only the kernel wheel changed (0.0.8). Every other artifact was carried