ops(svos): U13 phase B — nh3-svos on v0.0.9 (schema 12), kernel app origin, seat-host and high-seat origins, bridge env on nh3-dev
Prime approved svos-dev's U13 pre-edit ask (2026-10-10 1639). - B1: release afab123 on nh3-svos (kernel 0.0.9, migration 11->12) after a vzdump and an offline svos.db copy; app-terminal and app-postoffice held on b588510 by drop-ins until the edit. - B2: --app-listen 127.0.0.1:8410 (drop-in) fronted by Caddy as svos-kernel.phasefinal.com on the LAN, allowlisted to nh3-dev; the path list adds /auth/kernel-key, which every connector client needs. DNS rewrite on the three internal AdGuards. - B3/B4: svos-seathost (-> :8404) and svos-highseat (-> :8773) on nh3-dev's fleet Caddy, allowlisted to nh3-svos; /opt/svos apps env on nh3-dev with svos-postoffice[push] (althing-core 3.7.0 from althing 7a2a1c1) and svos-terminal. The fleet Caddyfile is now tracked in configs/.
This commit is contained in:
@@ -176,6 +176,23 @@ partition steps skip themselves now; growpart + resize2fs do the work, online).
|
||||
(their code; the model endpoint + key live in server env only — unit is `0600`,
|
||||
not committed).
|
||||
|
||||
## SVOS U13 phase B on this box (2026-10-10, Prime-approved; infra-ops)
|
||||
|
||||
The new SVOS kernel runs on nh3-svos; at the U13 edit three of its apps run HERE. Prepared, nothing running yet:
|
||||
|
||||
- **Origins on the fleet Caddy** (`fleet-tls-caddy`, canonical `configs/fleet-tls-caddy/Caddyfile`, backup
|
||||
`~/.config/caddy-fleet/Caddyfile.bak-20261010-u13`), both reachable **only from nh3-svos** (10.100.50.43; else 403)
|
||||
and both 502 until their app listens:
|
||||
- `https://svos-seathost.nh3.phasefinal.com` → `127.0.0.1:8404`, app-terminal's listener (unbuffered).
|
||||
- `https://svos-highseat.nh3.phasefinal.com` → `127.0.0.1:8773`, app-high-seat (the High Seat's signed-only listener).
|
||||
- **Apps env `/opt/svos/current/apps-env`** (→ `releases/afab123`, lkraven-owned): python 3.13.2 copied to
|
||||
`/opt/svos/python/` (so it does not depend on `~/.local/share/uv`), the release's hashed `apps-requirements.txt`, then
|
||||
svos-connector 0.0.7, **svos-postoffice 0.0.2 with its `push` extra satisfied by althing-core 3.7.0** installed
|
||||
(non-editable) from a clean archive of althing `7a2a1c1`, and svos-terminal 0.0.3. ⚠ althing-core's own deps
|
||||
(python-ulid 4.0.1, pyyaml 6.0.3) are resolved, not hash-pinned. No units yet: the bridge and terminal units, their
|
||||
config and their credentials are written at the edit. The kernel is reached at `https://svos-kernel.phasefinal.com`
|
||||
(LAN, allowlisted to this box). No input firewall here, so a LAN listener for the bridge needs no rule.
|
||||
|
||||
## Box-wide Playwright / Chromium (2026-06-04)
|
||||
|
||||
Available to **every user/project** on this box — no per-home `playwright install`:
|
||||
|
||||
@@ -23,6 +23,32 @@ configuration, enrollment and acceptance.
|
||||
| App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events`. **Monitoring:** `scripts/check-backup-freshness.sh` now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict **DEGRADED, exit 4** (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script |
|
||||
| Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 |
|
||||
|
||||
## Release afab123 / v0.0.9 + U13 phase B (2026-10-10 1646–1648, Prime-approved 1639; svos-dev thread `01M4M2Z2H3BQH375M2G30GQHDM`)
|
||||
|
||||
- **`/opt/svos/current` → `releases/afab123`: kernel 0.0.9, schema 11 → 12** (migration 0012, the page verb's tables;
|
||||
start returned READY in 0.64 s). Built from a clean archive of afab123: kernel, connector 0.0.7, postoffice 0.0.2,
|
||||
terminal 0.0.3, agent 0.0.3 and miranda 0.0.2 wheels; interface 0.0.4, knowledge 0.0.1 and voice 0.0.1 wheels,
|
||||
static, vendor, hermes-plugin, the kernel unit and `apps-requirements.txt` carried byte-identical from b588510
|
||||
(sources unchanged, verified). `/health` ok, schema 12, up 4, degraded 1 (the terminal, by design). Inventory
|
||||
regenerated; the front door's previous-asset handle is now b588510.
|
||||
- **Pre-change backups:** vzdump `vm/111/2026-10-10T23:45:41Z` on pbs-ana (freeze/thaw clean), and an OFFLINE copy
|
||||
taken with every service stopped: `/var/lib/svos/backup/pre-afab123/` (`svos.db` integrity ok, user_version 11, plus
|
||||
`config-keys-apps.tar`). **Rollback:** stop all, remove the kernel drop-in, restore that `svos.db` (keep no stale
|
||||
`-wal`/`-shm`), switch `current` back to b588510, start.
|
||||
- ⚠ **app-terminal and app-postoffice are HELD on b588510** by drop-ins `svos-{terminal,postoffice}.service.d/10-hold-b588510.conf`
|
||||
(`configs/svos-apps/`): their v0.0.9 manifests changed, and a new version would be held for approval and refuse the
|
||||
kernel's calls until then (svos-dev's sequencing). Both move to nh3-dev at the U13 edit: **remove the drop-ins
|
||||
there, and do NOT delete `releases/b588510` before that.** interface, knowledge and voice run afab123.
|
||||
- **B2, the kernel's app origin:** drop-in `svos-kernel.service.d/20-app-listen.conf` (`configs/svos-kernel/`) adds
|
||||
`--app-listen 127.0.0.1:8410`; Caddy serves `https://svos-kernel.phasefinal.com` on **10.100.50.43 only** (own LE
|
||||
cert, DNS-01), no Basic auth, bearer passed, identity headers stripped, **source allowlisted to nh3-dev** (10.100.10.50),
|
||||
paths `/auth/enroll`, `/auth/proofs`, `/auth/kernel-key`, `/registry/*`, `/verbs/*`, `/seats/enroll`, else 404.
|
||||
`/auth/kernel-key` is ADDED to svos-dev's list: every `svos_connector` client fetches it to trust the kernel's
|
||||
signatures, and it is the runbook's B2 check. The three internal AdGuards answer the name with 10.100.50.43.
|
||||
Tested: from nh3-dev `/auth/kernel-key` and `/registry/apps` → **401 `no principal` from the kernel**, `/health` and
|
||||
`/api/health` → 404; from nh3-svos and nh3-docker → 403. Caddyfile backup `/etc/caddy/Caddyfile.pre-kernel-origin`.
|
||||
- The app-side origins on nh3-dev (`svos-seathost`, `svos-highseat`) and the bridge env are in `servers/nh3-dev/README.md`.
|
||||
|
||||
## Release b588510 (2026-10-09 0540, kernel 0.0.8: the SQLite reader-pool fix)
|
||||
|
||||
- `/opt/svos/current` → `releases/b588510`. Only the kernel wheel changed (0.0.8). Every other artifact was carried
|
||||
|
||||
Reference in New Issue
Block a user