feat(nh3-svos): credential-free SVOS offline shell before Basic auth; contract static routes; backup drill results

svos-dev: the service worker fetches its hash-pinned INVENTORY with credentials omitted. Exactly those files plus /sw.js are now served before basic_auth (GET/HEAD, exact files). The list is generated from the release's sw.js by svos-front-inventory, which hash-checks each file and validates before reload. Gated static routes follow the web/README hosting contract: missing assets 404 instead of index.html, other document requests get index.html, everything else 404, unversioned files no-cache. acceptance-matrix.sh: 35/35 live; 16 FAIL against the previous config (positive control). git added to the nh3-svos playbook (kernel's remote check). Live backup drill: quiesced backup ~1-2 s outage; a refused freeze does NOT abort vzdump (unfrozen backup, job reported successful); both images restore-mounted read-only and verified.
2026-10-05 19:58:19 -07:00
parent 7c3eee877a
commit e369b1c769
6 changed files with 186 additions and 7 deletions
@@ -20,7 +20,7 @@ configuration, enrollment and acceptance.
 | Releases | `/opt/svos` root:svc-svos 2775, with `releases/` for svos-dev's layout. The git checkout is not a runtime dependency |
 | Docker | none, by design |
 | Backups | nh3-pve-2 vzdump job, nightly **21:30**, `all 1` → pbs-ana namespace `nh3-pve-2` (mirrored to pbs-nh3 at 06:30), snapshot mode with guest-agent fs-freeze |
-| App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. Tested 2026-10-05 via `qm guest cmd 111 fsfreeze-*`, both success and failure paths. The live-service backup drill is still pending |
+| App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events` |
 | Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 |

 ## Build (reproducible)
@@ -48,6 +48,9 @@ scripts/elway infra-ops@10.100.50.43 --playbook playbooks/nh3-svos-host.yaml
  The plaintext is vaulted at **`nh3-svos/basic-auth-vuong`**. To rotate: pipe the new password to
  `caddy hash-password --algorithm argon2id` on stdin (never `--plaintext`), rewrite the accounts file, `secret put`,
  then `systemctl reload caddy-svos`.
+- **Credential-free offline shell (svos-dev, 2026-10-05):** the service worker fetches its hash-pinned INVENTORY with `credentials:'omit'`, so exactly those files plus `/sw.js` are served BEFORE Basic auth (GET/HEAD, exact files, no fallback). Public build bytes only; `/`, `/status`, `/api/*`, the connector and everything else stay gated. The path list is **generated** into `/etc/caddy/svos-public-inventory` by **`/usr/local/sbin/svos-front-inventory`** (`configs/svos-front/`): it hash-checks every file against INVENTORY, validates, then reloads. ⚠ **Run it as root after every SVOS release**, or the next build's hashed asset names stay gated and worker install fails. `--check` changes nothing.
+- **Static routes (gated), per svoperatingsystem `web/README.md` Hosting contract:** `/` and `/status` → index.html; `/assets/*`, `/icons/*`, `/favicon.svg`, `/manifest.webmanifest`, `/index.html`, `/sw.js` → exact file, missing → 404; other HTML document requests (`Sec-Fetch-Dest: document` or `Accept: text/html`) → index.html; anything else → 404. Unversioned files get `Cache-Control: no-cache`.
+- **Acceptance matrix:** `configs/svos-front/acceptance-matrix.sh` (35 checks; usage in its header). 35/35 pass live at 1957 2026-10-05 from fv-ml1; against the previous config it reports 16 FAIL (positive control).
 - **Headers:** client copies of `Tailscale-User-*`, `X-Forwarded-*`, `Tailscale-Headers-Info`, `X-Real-IP`, `Forwarded`
  and `X-SVOS-Peer` are deleted before auth. After auth, `Tailscale-User-Login` is SET to the authenticated user, and
  **`Authorization` is deleted**. That deletion is load-bearing: the kernel's human path (svoperatingsystem
@@ -68,7 +71,7 @@ scripts/elway infra-ops@10.100.50.43 --playbook playbooks/nh3-svos-host.yaml
    headers, `X-Forwarded-Proto: https`, `Host: svos.phasefinal.com`, and path `/health` (the `/api` prefix stripped);
  - connector route with right credentials → 502 (auth passed; the connector was not running); cert verifies; LAN
    10.100.50.43:443 → connection refused.
- **Still to run:** the live-service vzdump + restore-mount drill, once svos-dev runs the services.
+- **Open (svos-dev's numbers needed):** the hosting contract also asks the boundary to bound request body size and read duration for the connector; not set yet.

 ### Original requirements (svos-dev)