From feef8330caed94af2577ddf1833aa83f58c61ed8 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sat, 10 Oct 2026 16:50:18 -0700 Subject: [PATCH] =?UTF-8?q?ops(svos):=20U13=20phase=20B=20=E2=80=94=20nh3-?= =?UTF-8?q?svos=20on=20v0.0.9=20(schema=2012),=20kernel=20app=20origin,=20?= =?UTF-8?q?seat-host=20and=20high-seat=20origins,=20bridge=20env=20on=20nh?= =?UTF-8?q?3-dev?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prime approved svos-dev's U13 pre-edit ask (2026-10-10 1639). - B1: release afab123 on nh3-svos (kernel 0.0.9, migration 11->12) after a vzdump and an offline svos.db copy; app-terminal and app-postoffice held on b588510 by drop-ins until the edit. - B2: --app-listen 127.0.0.1:8410 (drop-in) fronted by Caddy as svos-kernel.phasefinal.com on the LAN, allowlisted to nh3-dev; the path list adds /auth/kernel-key, which every connector client needs. DNS rewrite on the three internal AdGuards. - B3/B4: svos-seathost (-> :8404) and svos-highseat (-> :8773) on nh3-dev's fleet Caddy, allowlisted to nh3-svos; /opt/svos apps env on nh3-dev with svos-postoffice[push] (althing-core 3.7.0 from althing 7a2a1c1) and svos-terminal. The fleet Caddyfile is now tracked in configs/. --- configs/fleet-tls-caddy/Caddyfile | 55 +++++++++++++++++++ .../10-hold-b588510.conf | 8 +++ .../10-hold-b588510.conf | 8 +++ configs/svos-front/Caddyfile | 44 +++++++++++++++ configs/svos-kernel/20-app-listen.conf | 9 +++ persistent-memory.md | 3 +- servers/nh3-dev/README.md | 17 ++++++ servers/nh3-svos/README.md | 26 +++++++++ 8 files changed, 169 insertions(+), 1 deletion(-) create mode 100644 configs/fleet-tls-caddy/Caddyfile create mode 100644 configs/svos-apps/svos-postoffice.service.d/10-hold-b588510.conf create mode 100644 configs/svos-apps/svos-terminal.service.d/10-hold-b588510.conf create mode 100644 configs/svos-kernel/20-app-listen.conf diff --git a/configs/fleet-tls-caddy/Caddyfile b/configs/fleet-tls-caddy/Caddyfile new file mode 100644 index 0000000..0142042 --- /dev/null +++ b/configs/fleet-tls-caddy/Caddyfile @@ -0,0 +1,55 @@ +{ + email lkraven@lkraven.com +} + +# Fleet internal TLS — caddy is the WILDCARD CERT AUTHORITY for *.nh3.phasefinal.com +# (publicly-trusted LE via Cloudflare DNS-01, renewed automatically). Consumers: +# * self-terminate with a copy of this cert (talk: ~/talk/certs, kept fresh by +# fleet-tls-cert-check.timer), OR +# * front here via a reverse_proxy handle block on @ host .nh3.phasefinal.com +*.nh3.phasefinal.com { + tls { + dns cloudflare {env.CF_API_TOKEN} + resolvers 1.1.1.1 8.8.8.8 + } + # Scriberr (fv-ml1:8080, plain HTTP). Fronted so the browser recorder gets a SecureContext: + # getUserMedia is refused on http:// origins ("Failed to initialize recorder"). 2026-10-01. + # Scriberr's ALLOWED_ORIGINS must list this name (fv-ml1 /opt/docker/compose/scriberr/.env). + @scriberr host scriberr.nh3.phasefinal.com + handle @scriberr { + reverse_proxy 10.251.50.54:8080 { + flush_interval -1 + } + } + # SVOS U13 (2026-10-10, Prime-approved; svoperatingsystem docs/operations/u13-svos-edit.md "Before the edit" 2, + # svos docs/u13-cutover-runbook.md B3/B4). Two app origins on nh3-dev's LOOPBACK, reachable only from nh3-svos + # (10.100.50.43), whose kernel signs every call it makes to them. Until each app runs (the U13 edit) they answer 502. + # - svos-seathost: app-terminal, which moves to nh3-dev at the edit (svos-terminal, listener 127.0.0.1:8404). + # The kernel's instance.toml names this as app-terminal's base_url. No buffering (the app has long-lived routes). + # - svos-highseat: app-high-seat, the High Seat's signed-only listener (127.0.0.1:8773, enabled at the edit's E1). + @svos_seathost host svos-seathost.nh3.phasefinal.com + handle @svos_seathost { + @not_nh3_svos not remote_ip 10.100.50.43 + handle @not_nh3_svos { + respond 403 + } + handle { + reverse_proxy 127.0.0.1:8404 { + flush_interval -1 + } + } + } + @svos_highseat host svos-highseat.nh3.phasefinal.com + handle @svos_highseat { + @not_nh3_svos not remote_ip 10.100.50.43 + handle @not_nh3_svos { + respond 403 + } + handle { + reverse_proxy 127.0.0.1:8773 + } + } + handle { + respond "nh3 fleet TLS authority — no tool fronted here (talk self-terminates on :8092)" 404 + } +} diff --git a/configs/svos-apps/svos-postoffice.service.d/10-hold-b588510.conf b/configs/svos-apps/svos-postoffice.service.d/10-hold-b588510.conf new file mode 100644 index 0000000..9907485 --- /dev/null +++ b/configs/svos-apps/svos-postoffice.service.d/10-hold-b588510.conf @@ -0,0 +1,8 @@ +# /etc/systemd/system/svos-postoffice.service.d/10-hold-b588510.conf on nh3-svos (U13 B1, 2026-10-10). +# TEMPORARY: keeps app-postoffice on release b588510's code while the kernel runs v0.0.9 (afab123). Its v0.0.9 manifest +# changed, and a new version would re-register, be held for the owner's approval, and refuse the kernel's calls as +# wrong-revision until then (svos-dev's sequencing). The app moves to nh3-dev at the U13 edit; REMOVE this drop-in +# there, and keep /opt/svos/releases/b588510 on disk until it is gone. +[Service] +ExecStart= +ExecStart=/opt/svos/releases/b588510/apps-env/bin/svos-postoffice diff --git a/configs/svos-apps/svos-terminal.service.d/10-hold-b588510.conf b/configs/svos-apps/svos-terminal.service.d/10-hold-b588510.conf new file mode 100644 index 0000000..00695c3 --- /dev/null +++ b/configs/svos-apps/svos-terminal.service.d/10-hold-b588510.conf @@ -0,0 +1,8 @@ +# /etc/systemd/system/svos-terminal.service.d/10-hold-b588510.conf on nh3-svos (U13 B1, 2026-10-10). +# TEMPORARY: keeps app-terminal on release b588510's code while the kernel runs v0.0.9 (afab123). Its v0.0.9 manifest +# changed, and a new version would re-register, be held for the owner's approval, and refuse the kernel's calls as +# wrong-revision until then (svos-dev's sequencing). The app moves to nh3-dev at the U13 edit; REMOVE this drop-in +# there, and keep /opt/svos/releases/b588510 on disk until it is gone. +[Service] +ExecStart= +ExecStart=/opt/svos/releases/b588510/apps-env/bin/svos-terminal diff --git a/configs/svos-front/Caddyfile b/configs/svos-front/Caddyfile index 4b0af11..be05327 100644 --- a/configs/svos-front/Caddyfile +++ b/configs/svos-front/Caddyfile @@ -212,3 +212,47 @@ https://svos-terminal.phasefinal.com:443 { } } } + +# The kernel's app origin (U13 B2, 2026-10-10, Prime-approved; svoperatingsystem U9a and docs/operations/u13-svos-edit.md +# "Before the edit" 1). Apps that run OFF this box reach the kernel here: from the U13 edit, app-terminal, +# app-postoffice (the bridge) and app-high-seat, all on nh3-dev. LAN only, because nh3-dev is not on the mesh; the +# internal AdGuards answer svos-kernel.phasefinal.com -> 10.100.50.43 and no MagicDNS record exists. +# - Upstream is the kernel's SECOND listener, `serve --app-listen 127.0.0.1:8410` (configs/svos-kernel/20-app-listen.conf): +# a request there is never a human and loopback trust never applies, so every call authenticates with its own bearer. +# NO Basic auth, and Authorization passes through untouched. +# - Source allowlisted to nh3-dev (10.100.10.50); anything else gets 403 from Caddy. +# - Paths: svos-dev's app surface (/auth/enroll, /auth/proofs, /registry/*, /verbs/*, /seats/enroll) plus +# /auth/kernel-key, which every svos_connector client fetches to trust the kernel's signatures (client.kernel_key; it +# is also the joint runbook's B2 check: refused by the KERNEL with 401 without a credential). Anything else 404. +# - Client identity/forwarding headers are deleted as on the other sites; the same 1 MiB body cap; nothing streams. +https://svos-kernel.phasefinal.com:443 { + bind 10.100.50.43 + tls { + dns cloudflare {env.CF_API_TOKEN} + resolvers 1.1.1.1 8.8.8.8 + } + route { + request_header -Tailscale-User-* + request_header -X-Forwarded-* + request_header -Tailscale-Headers-Info + request_header -X-Real-IP + request_header -Forwarded + request_header -X-SVOS-Peer + + request_body { + max_size 1MiB + } + + @not_nh3_dev not remote_ip 10.100.10.50 + handle @not_nh3_dev { + respond 403 + } + @app_surface path /auth/enroll /auth/proofs /auth/kernel-key /registry/* /verbs/* /seats/enroll + handle @app_surface { + reverse_proxy 127.0.0.1:8410 + } + handle { + respond 404 + } + } +} diff --git a/configs/svos-kernel/20-app-listen.conf b/configs/svos-kernel/20-app-listen.conf new file mode 100644 index 0000000..af2d4b8 --- /dev/null +++ b/configs/svos-kernel/20-app-listen.conf @@ -0,0 +1,9 @@ +# /etc/systemd/system/svos-kernel.service.d/20-app-listen.conf on nh3-svos (U13 B2, 2026-10-10, Prime-approved). +# Adds the kernel's second listener for apps OFF the box (kernel >= 0.0.9, svoperatingsystem U9a). A request on +# 127.0.0.1:8410 is never a human and loopback trust never applies there; Caddy fronts it as svos-kernel.phasefinal.com +# on the LAN, allowlisted to nh3-dev (configs/svos-front/Caddyfile). The primary listener is unchanged. The ExecStart +# below is the release unit's (deployment/nh3-svos/svos-kernel.service) plus --app-listen; keep the two in step when +# that unit changes. serve refuses --app-listen on the primary listener's port. +[Service] +ExecStart= +ExecStart=/opt/svos/current/kernel-env/bin/svos-kernel serve --data-root /var/lib/svos --host 127.0.0.1 --port 8400 --trust-loopback --app-listen 127.0.0.1:8410 diff --git a/persistent-memory.md b/persistent-memory.md index ebb1800..9bd0cde 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -118,7 +118,7 @@ _As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections ca ### 2026-10-09: live now (as of 2026-10-09 ~1955 PT) - **Nothing is mid-action.** No claims held (`scripts/ops-log claims`). eshpfi is pushed through this snapshot's commit. ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- ` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose (auto-regenerated). -- **Approved, NOT built (do not start before Prime says so after this snapshot):** SVOS app-traffic ingress for U13. Details, release (`afab123`, v0.0.9, schema 12) and gotchas are in the 2026-10-09 Recent-decisions entry; svos-dev's edit doc is `docs/operations/u13-svos-edit.md`; highseat-dev owns the schedule. +- **U13 phase B (infra) DONE 2026-10-10 1646–1650 on Prime's go:** nh3-svos on v0.0.9 (`afab123`, schema 12) with app-terminal + app-postoffice HELD on b588510 by drop-ins (remove at the edit; keep `releases/b588510` until then); kernel app origin `svos-kernel.phasefinal.com` (LAN, nh3-dev only); `svos-seathost` + `svos-highseat` origins on nh3-dev (nh3-svos only); bridge env `/opt/svos/current/apps-env` on nh3-dev. Next is the atomic edit (highseat-dev's runbook `svos/docs/u13-cutover-runbook.md`, Prime's go; E2 is S→I). → `servers/nh3-svos/README.md`, `servers/nh3-dev/README.md` - **Owed by me (no input needed):** - Destroy `zroot/ROOT/debian@pre-patch-20261008` on fv-ml1 ~10-15 if nothing regressed. - nh3-svos housekeeping: releases `6c6abbe` and `a737a06` are no longer referenced by the front door and can be deleted (`ac42ace` holds the same static as the current `b588510`). @@ -186,6 +186,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache- ## Recent decisions +- `[2026-10-10]` **U13 phase B built (Prime 1639: "implement as asked")**: B1 kernel 0.0.9 + schema 12 (vzdump `vm/111/2026-10-10T23:45:41Z` + offline `pre-afab123` copy first), B2 `svos-kernel.phasefinal.com` → `--app-listen 127.0.0.1:8410` (allowlist nh3-dev; `/auth/kernel-key` ADDED to svos-dev's path list because every connector client fetches it), B3 seat-host origin + bridge env on nh3-dev, B4 app-high-seat origin. Kernel answers 401 `no principal` through the origin; other sources 403. - `[2026-10-10]` **esh-nas-pve `nvme/ROOT@pre-pve9` destroyed 0750 (607M)** after a week clean on PVE 9.2.21 / 7.0.14-20 (up 6 d, no err-level journal in 7 d, pools healthy, all guests up; only `openipmi.service` failed, the known item). Left: `nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z` (now 1.43G, a pre-PVE-9 rollback point that is useless as one) and `nvme/subvol-105-disk-0@pre-vaapi-20260911`. - `[2026-10-09]` **vh gets every albok read group (Prime 2244: "vh gets albok-everything")**: albok-read, albok-personal and albok-feedback (the restricted wing's default-deny ends for vh only), plus his four Macs' keys on vh (`playbooks/nh3-albok-vh-viewer.yaml`). No net new root path: those Macs already reach root there via lkraven@nh3-dev and the infra-ops key. Later restricted groups (1513+) are NOT included. - `[2026-10-09]` **nh3-dev: orphaned svos pytest killed 2242** (14.4 GB RSS, +2.3 GB/h, swap 100%, Beszel 90% alert; SIGTERM ignored, SIGKILL; used 25.4 → 11.3 GB). highseat-dev's retry-loop test with an instant fake sleep; they added pytest-timeout 300 s and a 50-attempt bound. diff --git a/servers/nh3-dev/README.md b/servers/nh3-dev/README.md index 1f59eb9..6a8d7fb 100644 --- a/servers/nh3-dev/README.md +++ b/servers/nh3-dev/README.md @@ -176,6 +176,23 @@ partition steps skip themselves now; growpart + resize2fs do the work, online). (their code; the model endpoint + key live in server env only — unit is `0600`, not committed). +## SVOS U13 phase B on this box (2026-10-10, Prime-approved; infra-ops) + +The new SVOS kernel runs on nh3-svos; at the U13 edit three of its apps run HERE. Prepared, nothing running yet: + +- **Origins on the fleet Caddy** (`fleet-tls-caddy`, canonical `configs/fleet-tls-caddy/Caddyfile`, backup + `~/.config/caddy-fleet/Caddyfile.bak-20261010-u13`), both reachable **only from nh3-svos** (10.100.50.43; else 403) + and both 502 until their app listens: + - `https://svos-seathost.nh3.phasefinal.com` → `127.0.0.1:8404`, app-terminal's listener (unbuffered). + - `https://svos-highseat.nh3.phasefinal.com` → `127.0.0.1:8773`, app-high-seat (the High Seat's signed-only listener). +- **Apps env `/opt/svos/current/apps-env`** (→ `releases/afab123`, lkraven-owned): python 3.13.2 copied to + `/opt/svos/python/` (so it does not depend on `~/.local/share/uv`), the release's hashed `apps-requirements.txt`, then + svos-connector 0.0.7, **svos-postoffice 0.0.2 with its `push` extra satisfied by althing-core 3.7.0** installed + (non-editable) from a clean archive of althing `7a2a1c1`, and svos-terminal 0.0.3. ⚠ althing-core's own deps + (python-ulid 4.0.1, pyyaml 6.0.3) are resolved, not hash-pinned. No units yet: the bridge and terminal units, their + config and their credentials are written at the edit. The kernel is reached at `https://svos-kernel.phasefinal.com` + (LAN, allowlisted to this box). No input firewall here, so a LAN listener for the bridge needs no rule. + ## Box-wide Playwright / Chromium (2026-06-04) Available to **every user/project** on this box — no per-home `playwright install`: diff --git a/servers/nh3-svos/README.md b/servers/nh3-svos/README.md index 1f683d8..9bde2d4 100644 --- a/servers/nh3-svos/README.md +++ b/servers/nh3-svos/README.md @@ -23,6 +23,32 @@ configuration, enrollment and acceptance. | App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events`. **Monitoring:** `scripts/check-backup-freshness.sh` now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict **DEGRADED, exit 4** (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script | | Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 | +## Release afab123 / v0.0.9 + U13 phase B (2026-10-10 1646–1648, Prime-approved 1639; svos-dev thread `01M4M2Z2H3BQH375M2G30GQHDM`) + +- **`/opt/svos/current` → `releases/afab123`: kernel 0.0.9, schema 11 → 12** (migration 0012, the page verb's tables; + start returned READY in 0.64 s). Built from a clean archive of afab123: kernel, connector 0.0.7, postoffice 0.0.2, + terminal 0.0.3, agent 0.0.3 and miranda 0.0.2 wheels; interface 0.0.4, knowledge 0.0.1 and voice 0.0.1 wheels, + static, vendor, hermes-plugin, the kernel unit and `apps-requirements.txt` carried byte-identical from b588510 + (sources unchanged, verified). `/health` ok, schema 12, up 4, degraded 1 (the terminal, by design). Inventory + regenerated; the front door's previous-asset handle is now b588510. +- **Pre-change backups:** vzdump `vm/111/2026-10-10T23:45:41Z` on pbs-ana (freeze/thaw clean), and an OFFLINE copy + taken with every service stopped: `/var/lib/svos/backup/pre-afab123/` (`svos.db` integrity ok, user_version 11, plus + `config-keys-apps.tar`). **Rollback:** stop all, remove the kernel drop-in, restore that `svos.db` (keep no stale + `-wal`/`-shm`), switch `current` back to b588510, start. +- ⚠ **app-terminal and app-postoffice are HELD on b588510** by drop-ins `svos-{terminal,postoffice}.service.d/10-hold-b588510.conf` + (`configs/svos-apps/`): their v0.0.9 manifests changed, and a new version would be held for approval and refuse the + kernel's calls until then (svos-dev's sequencing). Both move to nh3-dev at the U13 edit: **remove the drop-ins + there, and do NOT delete `releases/b588510` before that.** interface, knowledge and voice run afab123. +- **B2, the kernel's app origin:** drop-in `svos-kernel.service.d/20-app-listen.conf` (`configs/svos-kernel/`) adds + `--app-listen 127.0.0.1:8410`; Caddy serves `https://svos-kernel.phasefinal.com` on **10.100.50.43 only** (own LE + cert, DNS-01), no Basic auth, bearer passed, identity headers stripped, **source allowlisted to nh3-dev** (10.100.10.50), + paths `/auth/enroll`, `/auth/proofs`, `/auth/kernel-key`, `/registry/*`, `/verbs/*`, `/seats/enroll`, else 404. + `/auth/kernel-key` is ADDED to svos-dev's list: every `svos_connector` client fetches it to trust the kernel's + signatures, and it is the runbook's B2 check. The three internal AdGuards answer the name with 10.100.50.43. + Tested: from nh3-dev `/auth/kernel-key` and `/registry/apps` → **401 `no principal` from the kernel**, `/health` and + `/api/health` → 404; from nh3-svos and nh3-docker → 403. Caddyfile backup `/etc/caddy/Caddyfile.pre-kernel-origin`. +- The app-side origins on nh3-dev (`svos-seathost`, `svos-highseat`) and the bridge env are in `servers/nh3-dev/README.md`. + ## Release b588510 (2026-10-09 0540, kernel 0.0.8: the SQLite reader-pool fix) - `/opt/svos/current` → `releases/b588510`. Only the kernel wheel changed (0.0.8). Every other artifact was carried