vh
5e8307ed7a
Built in-session from a clean archive of the tag (kernel, connector base and terminal wheels; everything else carried from 7f95e54). Ran after the 21:30 vzdump of VM 111, with an offline backup of the schema 12 database first. check-config accepted the new [tasks] table and the tasks artifact mapping (host nh3-svos, the terminal's artifact_host). Kernel READY in 1.17 s, schema 13, apps up 7. nh3-dev's current moved to 452a592; svos-terminal 0.0.4 registered manifest revision 4, which was pinned without a card.
223 lines
25 KiB
Markdown
223 lines
25 KiB
Markdown
# nh3-svos: host for the NEW SVOS operating system (VM 111 on nh3-pve-2, NH3)
|
||
|
||
Built 2026-10-05 at svos-dev's request, relaying Prime ("stand up a dedicated VM for it on nh3-pve-2"). It runs the
|
||
new SVOS (repo `corviduo/svoperatingsystem`): a FastAPI/uvicorn kernel (SQLite WAL, synchronous FULL, one writer),
|
||
an interface connector, an optional voice adapter and a static phone/desktop UI.
|
||
**It is a separate instance.** The existing High Seat (`svos.service` on nh3-dev, `:8770`) and Miranda are NOT
|
||
migrated by this host.
|
||
|
||
**Ownership split:** infra-ops owns the VM, OS, access, backups and ingress. svos-dev owns the application: install,
|
||
configuration, enrollment and acceptance.
|
||
|
||
| | |
|
||
|---|---|
|
||
| Address | `10.100.50.43`, VLAN 50 (nh3-servers), static via cloud-init; `nh3-svos.nh3.internal` |
|
||
| Guest | VM **111** on **nh3-pve-2**, `onboot 1`, tag `svos` |
|
||
| Shape | 4 vCPU `x86-64-v3`, 8 GiB, 100 GB on `vmstore` (local NVMe LVM-thin, discard on). nh3-pve-2 had 47 GB RAM and 946 GB vmstore free at build |
|
||
| OS | Debian 13 genericcloud, Python 3.13 + venv, SQLite 3.46, systemd-timesyncd (synced), qemu-guest-agent |
|
||
| Identities | `vh` 1000 (operator keys + lkraven@nh3-dev, so svos-dev's session can ssh in; NOPASSWD sudo from cloud-init, plus `/etc/sudoers.d/91-vh-verifypw` so `sudo -v` agrees), `infra-ops` 850, **`svc-svos` 801** (nologin; the only writer of the data root) |
|
||
| Data root | `/var/lib/svos`, svc-svos 0700, on the local root ext4. Never NFS, never shared |
|
||
| Releases | `/opt/svos` root:svc-svos 2775, with `releases/` for svos-dev's layout. The git checkout is not a runtime dependency |
|
||
| Docker | none, by design |
|
||
| Backups | nh3-pve-2 vzdump job, nightly **21:30**, `all 1` → pbs-ana namespace `nh3-pve-2` (mirrored to pbs-nh3 at 06:30), snapshot mode with guest-agent fs-freeze |
|
||
| App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events`. **Monitoring:** `scripts/check-backup-freshness.sh` now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict **DEGRADED, exit 4** (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script |
|
||
| Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 |
|
||
|
||
## Release 452a592 / v0.0.11, U15 task seats slice 1 (2026-10-13 2131–2133; Prime's 2010 go via svos-dev, thread `01M4WAGAS2Z0P8J6GZRSCRPH2Q`)
|
||
|
||
- **Order:** tonight's 21:30 vzdump of VM 111 finished 21:31:30, then `playbooks/svos-releases/nh3-svos-452a592-v0.0.11.yaml`
|
||
(the kit was built in-session: kernel 0.0.11, connector base 0.0.8 and app-terminal 0.0.4 wheels from a clean archive of
|
||
the tag; everything else, including the web build of 7f95e54, carried byte-identical).
|
||
- **Backup before the migration:** `/var/lib/svos/backup/pre-452a592/` (the schema 12 `svos.db` via the sqlite backup API,
|
||
integrity ok, plus the config/keys/apps tar) and `config/instance.toml.pre-452a592`.
|
||
- **Config:** a `[tasks]` table with `scratch_root = "/var/lib/svos/apps/terminal/tasks"` (all ceilings at their defaults),
|
||
plus a `[[notifications.artifacts]]` for that root. Its **host is `nh3-svos`, not `nh3-dev`**: the terminal's
|
||
`artifact_host` stayed `nh3-svos` through U13, and its ResultReader names results by that host, so the mapping must
|
||
match it, as the injects mapping does. No runtime has `task_argv` yet, so a task-spawn is refused
|
||
`runtime_not_task_capable` until Prime names the command.
|
||
- **Result:** kernel 0.0.11 READY in 1.17 s, migration 0013 run, `/health` schema 13, config_gen `5ade279d2fb2dd1c`, apps up 7.
|
||
- **nh3-dev:** `/opt/svos/current → releases/452a592` (terminal 0.0.4, connector 0.0.8). `/var/lib/svos/apps/terminal/tasks`
|
||
(0700) and `terminal.toml [seat_host].tasks_root` (backup `.pre-452a592`). Only svos-terminal was restarted. Its manifest
|
||
revision 4 (0.0.4, `seat-host`) was **pinned directly, with no card**. The seats read unknown for 6 s, then live.
|
||
svos-miranda, svos-postoffice and svos-agent still run adc862e's code until their next restart.
|
||
- **Rollback:** stop the kernel, put `svos.db` back from `backup/pre-452a592` (schema 12), restore `instance.toml.pre-452a592`,
|
||
`current → releases/7f95e54`, then start. On nh3-dev: `current → releases/adc862e`, restore `terminal.toml.pre-452a592`,
|
||
then restart svos-terminal.
|
||
|
||
## Release afab123 / v0.0.9 + U13 phase B (2026-10-10 1646–1648, Prime-approved 1639; svos-dev thread `01M4M2Z2H3BQH375M2G30GQHDM`)
|
||
|
||
- **`/opt/svos/current` → `releases/afab123`: kernel 0.0.9, schema 11 → 12** (migration 0012, the page verb's tables;
|
||
start returned READY in 0.64 s). Built from a clean archive of afab123: kernel, connector 0.0.7, postoffice 0.0.2,
|
||
terminal 0.0.3, agent 0.0.3 and miranda 0.0.2 wheels; interface 0.0.4, knowledge 0.0.1 and voice 0.0.1 wheels,
|
||
static, vendor, hermes-plugin, the kernel unit and `apps-requirements.txt` carried byte-identical from b588510
|
||
(sources unchanged, verified). `/health` ok, schema 12, up 4, degraded 1 (the terminal, by design). Inventory
|
||
regenerated; the front door's previous-asset handle is now b588510.
|
||
- **Pre-change backups:** vzdump `vm/111/2026-10-10T23:45:41Z` on pbs-ana (freeze/thaw clean), and an OFFLINE copy
|
||
taken with every service stopped: `/var/lib/svos/backup/pre-afab123/` (`svos.db` integrity ok, user_version 11, plus
|
||
`config-keys-apps.tar`). **Rollback:** stop all, remove the kernel drop-in, restore that `svos.db` (keep no stale
|
||
`-wal`/`-shm`), switch `current` back to b588510, start.
|
||
- ⚠ **app-terminal and app-postoffice are HELD on b588510** by drop-ins `svos-{terminal,postoffice}.service.d/10-hold-b588510.conf`
|
||
(`configs/svos-apps/`): their v0.0.9 manifests changed, and a new version would be held for approval and refuse the
|
||
kernel's calls until then (svos-dev's sequencing). Both move to nh3-dev at the U13 edit: **remove the drop-ins
|
||
there, and do NOT delete `releases/b588510` before that.** interface, knowledge and voice run afab123.
|
||
- **B2, the kernel's app origin:** drop-in `svos-kernel.service.d/20-app-listen.conf` (`configs/svos-kernel/`) adds
|
||
`--app-listen 127.0.0.1:8410`; Caddy serves `https://svos-kernel.phasefinal.com` on **10.100.50.43 only** (own LE
|
||
cert, DNS-01), no Basic auth, bearer passed, identity headers stripped, **source allowlisted to nh3-dev** (10.100.10.50),
|
||
paths **method-pinned to svoperatingsystem dc241f8 `u13-svos-edit.md` item 1** (the apps' and Miranda's agent
|
||
connector's calls: GET `/auth/kernel-key` `/auth/whoami` `/registry/apps[/*]` `/principals/*/grants` `/seats` `/events`
|
||
`/artifacts/resolve`; POST `/auth/enroll` `/auth/proofs` `/seats/enroll` `/registry/apps/*/manifest`; `/verbs/*`), else
|
||
404. The three internal AdGuards answer the name with 10.100.50.43. Tested from nh3-dev: all 15 allowed method+path
|
||
pairs → **401 from the kernel** (JSON `detail`); `/health`, `/stream`, `/cards`, POST `/auth/kernel-key`, DELETE
|
||
`/registry/apps/x`, `/principals/a/b/grants` → Caddy's empty 404; from nh3-svos and nh3-docker → 403. Backups
|
||
`/etc/caddy/Caddyfile.pre-kernel-origin`, `.pre-kernel-paths`. ⚠ **Validate from a caddy-readable path**
|
||
(`/etc/caddy/Caddyfile.check`): the first deploy validated `/tmp/svos-stage-*/` (0700 infra-ops), got "permission
|
||
denied", and a `| tail` hid it. The config was valid (Caddy's own reload accepted it), and the validator was then
|
||
positive-controlled (a bogus directive → rc 1).
|
||
- **B5 + enrollment secrets (2026-10-10 1654):** `svos-kernel mint-secret` (run as infra-ops without sudo, so no
|
||
sudo-io copy) minted three tokens straight into the vault, read back by hash: `nh3-svos/app-high-seat-enrollment-secret`,
|
||
`nh3-svos/app-miranda-enrollment-secret`, `nh3-svos/miranda-persona-enrollment-secret`. Only app-high-seat is
|
||
declared so far (`[[apps]]` with `enrollment_sha256`, one reload, config_gen `22a05c20e17f93ce` → `1bd948c754e571da`,
|
||
backup `instance.toml.pre-high-seat`); the other two are declared in the edit.
|
||
- nh3-svos stays on afab123: svoperatingsystem `dc241f8` changes only svos-miranda (0.0.3, the charter), which runs on
|
||
nh3-dev, so a dc241f8 release here would change nothing that runs.
|
||
- The app-side origins on nh3-dev (`svos-seathost`, `svos-highseat`) and the bridge env are in `servers/nh3-dev/README.md`.
|
||
|
||
## Release b588510 (2026-10-09 0540, kernel 0.0.8: the SQLite reader-pool fix)
|
||
|
||
- `/opt/svos/current` → `releases/b588510`. Only the kernel wheel changed (0.0.8). Every other artifact was carried
|
||
byte-identical from ac42ace, the notify unit is unchanged, and there was no migration and no config change.
|
||
Previous-asset handle: none, because the static build is identical to ac42ace's.
|
||
- **The fd outage of 2026-10-08 22:55–23:25 is fixed at the source.** Root cause (svos-dev): `Store._reader` kept
|
||
one read-only SQLite connection per anyio worker thread, and those threads die after 10 s idle, so a browser's
|
||
polling bursts leaked a connection per dead worker. Kernel 0.0.7 hit EMFILE at the default soft limit of 1024
|
||
(509 `svos.db` + 506 `-wal` fds). The stopgap drop-in `svos-kernel.service.d/10-nofile-stopgap.conf`
|
||
(LimitNOFILE=65536, 2325 10-08 → 0540 10-09) is **removed**; the limit is back to 1024.
|
||
- Acceptance: `/health` ok, up 4, degraded 1. A 4-min headless-Chromium desk soak (211 `/api` requests) gave
|
||
`svos.db` handles 9 → 12, with one step during the first burst and then flat for 2.5 min, and `-wal` flat at 9.
|
||
0.0.7 under a real browser had grown +22 in 2 min. ⚠ Harness differs: headless Chromium here, Prime's browser then.
|
||
- Thaw-fix repeats: 0 app startup failures on the 10-08 21:30 nightly thaw and on the 10-09 05:39 pre-refresh vzdump.
|
||
|
||
## Release ac42ace (2026-10-08 1515, svos-dev thread `01M4ES8Q110HNH8J4WWCGE6S4K`)
|
||
|
||
- `/opt/svos/current` → `releases/ac42ace`: kernel **0.0.7** (schema 11, no migration), connector base 0.0.6,
|
||
interface UI **0.0.7** (top bar with the Pacific clock; the tree as the desk's stage), svos-agent 0.0.2 (staged).
|
||
Every other app wheel is unchanged, so there was no re-enrollment and no config change. Built from a clean archive,
|
||
the same way as a737a06; third-party pins are unchanged. **Previous = `a737a06`** (the front door's previous-asset
|
||
handle points at it). `6c6abbe` is no longer referenced and is safe to delete (it cannot read schema 11).
|
||
- **Kernel readiness pair installed together:** `svos-kernel.service` now has `Type=notify`, `NotifyAccess=main` and
|
||
`TimeoutStartSec=300` (copied from the release's `deployment/nh3-svos/svos-kernel.service`; the old unit is kept as
|
||
`.pre-notify`). `systemctl start svos-kernel` now returns only after `READY=1` (0.61 s, `/health` 200 at once). ⚠
|
||
Install a kernel ≥ 0.0.7 and this unit as a pair, never one alone: `Type=notify` on an older kernel times the start
|
||
out at 300 s.
|
||
- **Freeze/thaw drill, with a control:** the pre-refresh vzdump on the OLD kernel (15:14) made terminal and knowledge
|
||
fail once at thaw (the known race). The post-refresh vzdump (`vm/111/2026-10-08T22:16:12Z`, 15:16) showed **0**
|
||
app startup failures; all six units came back, and `/health` read up 4, degraded 1 within ~2 s. The one-poll
|
||
"down" for the terminal right after the restart is its first health poll, not a fault.
|
||
- Acceptance: `/health` ok, up 4, degraded 1. Chromium phone + desk over the LAN door: the top bar is present with
|
||
clock `1517`, the desk stage shows the tree (4 stage layers + hero), no overflow, 0 page errors; desk Connect voice
|
||
shows "Voice connected".
|
||
|
||
## Release a737a06 + apps (2026-10-07, svos-dev thread `01M4D1GNYBW7QVY7RF05R412V2`, Prime's 2240 decision)
|
||
|
||
- **Release:** `/opt/svos/current` → `releases/a737a06` (previous `6c6abbe` kept: it cannot read schema 11, and the
|
||
front door's previous-asset handle still points at it). Built by infra-ops from a clean `git archive`: 9 wheels,
|
||
the `web/` 0.0.6 static build, `kernel-requirements.txt` (uv export) and `apps-requirements.txt` (uv pip compile
|
||
against the kernel pins, hashed), `vendor/bifrost-1.2.1` (gitea vh index, sha256-verified, because PyPI has an
|
||
unrelated `bifrost`), and `hermes-plugin/svos` (staged for U13). Kernel 0.0.5 migrated schema 9 → 11 on first start.
|
||
- **Pre-change backups:** vzdump `vm/111/2026-10-08T06:02:14Z` (freeze/thaw clean) and an online sqlite `.backup`
|
||
plus config, keys and app state at `/var/lib/svos/backup/pre-a737a06/` (integrity ok, user_version 9).
|
||
- **Apps:**
|
||
| slug | listener | unit | state |
|
||
|---|---|---|---|
|
||
| app-interface | 127.0.0.1:8403 | svos-interface | up |
|
||
| app-knowledge | 127.0.0.1:8405 | svos-knowledge | up; Albok token `01M4D22HHJ7AJHX30WPDE02DFC`, vault `nh3-svos/albok-token-knowledge`, grants per A111 `read:fleet/memory`, `read:fleet/stash`, `read:personal/notes` (no wildcard, no restricted wing, no class grant) |
|
||
| app-terminal | 127.0.0.1:8404, base_url **https://svos-terminal.phasefinal.com** | svos-terminal | **degraded by design**: no zellij and no fleet session on this box |
|
||
| app-postoffice | 127.0.0.1:8406 | svos-postoffice (installed, NOT enabled) | declared + enrolled, stopped until Prime declares the bridge handle; `handle = "svos"` in its config is the example's placeholder |
|
||
| app-voice | 127.0.0.1:8402, front door `/voice/*` (same origin, BEFORE Basic auth; the lease bearer is the auth) | svos-voice | up since 2026-10-08 (Prime: voice on); engines in `config/apps/voice.toml`: STT Parakeet `http://10.251.50.54:8300`, TTS fleet gateway `http://irv-ml1.nh3.internal:8198`; owner grants `app:app-voice` write (autonomous) + read |
|
||
| app-miranda, svos-agent | — | — | packages in apps-env only (U13) |
|
||
Credentials `/etc/svos/<app>.env` (root 0600), vaulted as `nh3-svos/svos-<app>-credential` (terminal, knowledge, postoffice, voice; 2026-10-08); units in `configs/svos-apps/`; the quiesce hook lists every app unit.
|
||
- **Terminal origin:** `svos-terminal.phasefinal.com` = headscale MagicDNS extra record → 100.64.0.9, its own Caddy
|
||
site (LE via DNS-01). **No Basic auth** (Chromium cancels Basic prompts in cross-origin frames; the lease is the
|
||
page's auth). `/verbs/*` and `/health` are refused unless they come from the box itself. The kernel's signed calls
|
||
reach the app through Caddy, so nh3-svos resolves the name via `/etc/hosts` (`100.64.0.9 svos-terminal.phasefinal.com
|
||
svos.phasefinal.com`): it has no MagicDNS (`accept-dns=false`). The terminal is `up` only while Caddy is.
|
||
- **Acceptance 2026-10-07 ~2315:** 17/17 HTTPS controls through an SSH forward to the real listener (the 12 from 10-05,
|
||
plus: health schema 11 with 2 up and 1 degraded; the terminal registered on its own origin; knowledge `wings`
|
||
= exactly the 3 token wings; knowledge `search` hits; terminal `attach` refused `404 unknown seat`). Chromium
|
||
phone + desk: shell, inbox, stream, reload, `/status`, `/graph`, `/feed`, no overflow, no page errors; desk
|
||
"Open Terminal here" → pane open, framed from `https://svos-terminal.phasefinal.com/`, body `waiting` (no lease).
|
||
From mesh peer fv-ml1, the terminal origin returns: `/` 200 with `frame-ancestors https://svos.phasefinal.com`;
|
||
lease routes 401; `/verbs/*` and `/health` 403; anything else 404.
|
||
|
||
## Build (reproducible)
|
||
|
||
```bash
|
||
scripts/elway infra-ops@10.100.250.62 --playbook playbooks/nh3-pve-2-cloudinit-vm.yaml --var vmid=111 \
|
||
--var name=nh3-svos --var ip=10.100.50.43/24 --var cores=4 --var memory=8192 --var disk=100G --var tags=svos \
|
||
--var "description=..."
|
||
scripts/elway vh@10.100.50.43 --playbook playbooks/bootstrap-infra-ops-user.yaml
|
||
scripts/elway infra-ops@10.100.50.43 --playbook playbooks/nh3-svos-host.yaml
|
||
```
|
||
|
||
## Front door: mesh-only `https://svos.phasefinal.com`, HTTP Basic auth (Prime, 2026-10-05)
|
||
|
||
- **Mesh:** tailscaled joined headscale as `nh3-svos`, **100.64.0.9**, `--accept-dns=false --accept-routes=false`.
|
||
- **Name:** `svos.phasefinal.com` (and `svos-terminal.phasefinal.com`) has no public record. Mesh clients get
|
||
**100.64.0.9** from the headscale MagicDNS extra records (`/var/lib/headscale/extra-records.json` on nh3-headscale,
|
||
hot-reloaded). **Since 2026-10-08 (Prime: "serve it on the LAN too and point the dns at the internal LAN address")**
|
||
the three internal AdGuards (ana `10.250.50.70:8053`, esh `10.0.50.45`, nh3 `10.100.50.40`) carry hand-made
|
||
rewrites for both names → **`10.100.50.43`**, outside `dns-sync.py`'s `.internal` authority (like
|
||
`*.nh3.phasefinal.com`). Verified 10-08 from LAN hosts at all three sites (nh3-dev, nh3-docker, esh-docker-vm,
|
||
ana-docker): `/` 401, `/sw.js` 200, terminal `/` 200, terminal `/health` 403. nh3-svos itself still resolves both
|
||
names to 100.64.0.9 via `/etc/hosts`, so the kernel's signed calls to the terminal keep arriving from 100.64.0.9.
|
||
- **TLS:** Caddy 2.11.4 (the fleet `caddy-cf` build with the Cloudflare DNS module) at `/usr/local/bin/caddy`, unit
|
||
`caddy-svos.service`, config `/etc/caddy/Caddyfile` (canonical `configs/svos-front/Caddyfile`). It listens on
|
||
**100.64.0.9:443 and, since 2026-10-08, 10.100.50.43:443** (both site blocks), with no `:80` and nothing on the internet. The LE cert for `svos.phasefinal.com` comes via DNS-01, token `nh3-svos-caddy-dns01`
|
||
(phasefinal.com only), in `/etc/caddy/env` (root:caddy 0640), vaulted `nh3-svos/cloudflare-caddy-dns01-token`.
|
||
- **Identity: HTTP Basic auth, one account `Vuong`** (Prime: "Basic auth is fine, saved in browser is fine. Access is
|
||
via mesh because we don't host externally." SVOS addresses him as Prime; the login is `Vuong`). The argon2id hash is in
|
||
**`/etc/caddy/svos-basic-auth.accounts`** (root:caddy 0640, not in git), imported into `basic_auth argon2id SVOS`.
|
||
The plaintext is vaulted at **`nh3-svos/basic-auth-vuong`**. To rotate: pipe the new password to
|
||
`caddy hash-password --algorithm argon2id` on stdin (never `--plaintext`), rewrite the accounts file, `secret put`,
|
||
then `systemctl reload caddy-svos`.
|
||
- **Credential-free offline shell (svos-dev, 2026-10-05):** the service worker fetches its hash-pinned INVENTORY with `credentials:'omit'`, so exactly those files plus `/sw.js` are served BEFORE Basic auth (GET/HEAD, exact files, no fallback). Public build bytes only; `/`, `/status`, `/api/*`, the connector and everything else stay gated. The path list is **generated** into `/etc/caddy/svos-public-inventory` by **`/usr/local/sbin/svos-front-inventory`** (`configs/svos-front/`): it hash-checks every file against INVENTORY, validates, then reloads. ⚠ **Run it as root after every SVOS release**, or the next build's hashed asset names stay gated and worker install fails. `--check` changes nothing.
|
||
- **Previous release's assets across a rollout:** the same tool writes `/etc/caddy/svos-public-previous`: when `/opt/svos/current` has moved, the release named in the old inventory header becomes "previous", and its hashed `/assets/*` files that the new build lacks are served credential-free from **that release's own static dir** (hash-checked). One previous release at most; comments only when there is none (true at 2026-10-05, one release). Replica-tested: old asset 200 with the old INVENTORY hash, new asset with the new hash, POST to the old asset 401, unknown asset 401/404; tampered files and an unsafe path make the tool refuse.
|
||
- **Hosting bounds (svos-dev, per the web contract):** request bodies capped at exactly **1 MiB** on every route (413 over); **headers within 5 s, whole body within 15 s** (Caddy `read_header` / `read_body`); **no** write, response or idle deadline, so `/api/stream` (SSE) stays open. Live 2026-10-05 2014–2021 from fv-ml1: 1 MiB+1 → 413 on `/api/verbs` and `/verbs/pin`; exactly 1 MiB passes Caddy (kernel then answers); unfinished headers closed at 5.0 s; stalled body cut at 15.0 s; a signed `app-interface` ping returned pong before and DURING a stalled body; an SSE stream stayed open 41–45 s (client hung up) alongside all of it. ⚠ **A stalled body over HTTP/1.1 can come back as Caddy's empty `200 OK` (Content-Length 0)** instead of 504: replica 3/3 buffered, 2/3 unbuffered; live 2 of 3. Over h2, which browsers use, it is 502 (3/3 replica, 2/2 live). The app never gets a complete request either way; an empty 200 is not success. `handle_errors 499` and `request_buffers` do not change it (tested).
|
||
- **Static routes (gated), per svoperatingsystem `web/README.md` Hosting contract:** `/` and `/status` → index.html; `/assets/*`, `/icons/*`, `/favicon.svg`, `/manifest.webmanifest`, `/index.html`, `/sw.js` → exact file, missing → 404; other HTML document requests (`Sec-Fetch-Dest: document` or `Accept: text/html`) → index.html; anything else → 404. Unversioned files get `Cache-Control: no-cache`.
|
||
- **Acceptance matrix:** `configs/svos-front/acceptance-matrix.sh` (35 checks; usage in its header). 35/35 pass live at 1957 2026-10-05 from fv-ml1; against the previous config it reports 16 FAIL (positive control).
|
||
- **Headers:** client copies of `Tailscale-User-*`, `X-Forwarded-*`, `Tailscale-Headers-Info`, `X-Real-IP`, `Forwarded`
|
||
and `X-SVOS-Peer` are deleted before auth. After auth, `Tailscale-User-Login` is SET to the authenticated user, and
|
||
**`Authorization` is deleted**. That deletion is load-bearing: the kernel's human path (svoperatingsystem
|
||
`u1-principals` INV-U1-06) refuses any request carrying `Authorization`. As a result the front door is human-only; a
|
||
bearer client cannot come through it.
|
||
- **Retired 2026-10-05:** the WhoIs `forward_auth` gate (`svos-whois-auth`, 127.0.0.1:8405). Every mesh node has WhoIs
|
||
login `pfi`, so it could not tell Prime's devices from the servers without restructuring the mesh, and Prime declined
|
||
that. The unit, script, `/etc/svos-front/` and the repo files are removed; git history has them.
|
||
- **Config changes:** validate WITHOUT putting the token on a command line, then reload:
|
||
`sudo systemd-run --wait --pipe --collect -p User=caddy -p Group=caddy -p EnvironmentFile=/etc/caddy/env
|
||
-p "Environment=XDG_DATA_HOME=/var/lib/caddy XDG_CONFIG_HOME=/var/lib/caddy" /usr/local/bin/caddy validate --config
|
||
<file> --adapter caddyfile`, then `systemctl reload caddy-svos`. (`env $(cat /etc/caddy/env) caddy …` leaked the first
|
||
token into sudo's argv log on 2026-10-05; it was revoked.)
|
||
- **Tested 2026-10-05 ~1940–1946, from mesh peer fv-ml1, against a temporary header echo on 127.0.0.1:8400 (removed after):**
|
||
- no credentials → 401 `Basic realm="SVOS"`; wrong password → 401; username `vuong` (wrong case) → 401;
|
||
- right credentials with forged `Tailscale-User-Login`/`-Whatever`, `X-Forwarded-For/Proto/Zzz`, `Forwarded` and
|
||
`X-Real-IP` → 200. Upstream saw exactly one `Tailscale-User-Login: Vuong`, no `Authorization`, none of the forged
|
||
headers, `X-Forwarded-Proto: https`, `Host: svos.phasefinal.com`, and path `/health` (the `/api` prefix stripped);
|
||
- connector route with right credentials → 502 (auth passed; the connector was not running); cert verifies; LAN
|
||
10.100.50.43:443 → connection refused.
|
||
|
||
### Original requirements (svos-dev)
|
||
|
||
App side, per svos-dev 2026-10-05: kernel on `127.0.0.1:8400 --trust-loopback` (its Unix-socket mode needs a 0700
|
||
kernel-owned dir and creates a 0666 socket, so no shared-group socket in this build), interface connector on
|
||
`127.0.0.1:8403`. Static UI at `/opt/svos/current/static`. `/api/*` → kernel with the prefix stripped (`/api/health` is
|
||
the kernel's); `/health`, `/verbs/ping|pins|pin|unpin`, `/openapi.json` → connector unchanged. ⚠ With
|
||
`--trust-loopback`, every local process is inside the identity trust boundary, so run no untrusted workloads here.
|
||
Agreed design (option A): Caddy on the tailnet address only, `forward_auth` to a loopback WhoIs responder fed the real
|
||
socket peer (`{http.request.remote.host}:{port}`), fail closed, allow-list of logins, client identity/forwarding headers
|
||
stripped, Origin passed through untouched.
|
||
|
||
App listeners stay private (loopback or a Unix socket). The front door has to give the browser **HTTPS**: the UI and the
|
||
voice adapter need a secure context for the microphone. It also has to give the kernel's adapter a **verified identity**
|
||
in `Tailscale-User-Login`, with any browser-supplied copy stripped. See the svos-dev thread of 2026-10-05.
|