feat(svos-front,backups): SVOS hosting bounds and previous-release assets; backup check flags fs-freeze failures and covers nh3-pve-2

Front door: request bodies capped at 1 MiB, headers within 5 s, body within 15 s, no response deadlines (SSE stays open). Validated live: 413 over the cap, stalled body cut at 15 s, signed ping and an SSE stream unaffected. svos-front-inventory also serves the previous release's hashed assets credential-free from that release's static dir, hash-checked. Backup check: reads the logs of vzdump tasks PVE records as OK and reports ERROR lines; an fs-freeze/thaw error is an application-consistency failure, new verdict DEGRADED (exit 4), wired into the alert wrapper. nh3-pve-2 added to the node and namespace lists (its job results were never read). The svos drill task is acknowledged by UPID; BACKUP_IGNORE_ACKS=1 counts it (positive control: exit 4).

servers/nh3-svos/README.md

@@ -20,7 +20,7 @@ configuration, enrollment and acceptance.
 | Releases | `/opt/svos` root:svc-svos 2775, with `releases/` for svos-dev's layout. The git checkout is not a runtime dependency |
 | Docker | none, by design |
 | Backups | nh3-pve-2 vzdump job, nightly **21:30**, `all 1` → pbs-ana namespace `nh3-pve-2` (mirrored to pbs-nh3 at 06:30), snapshot mode with guest-agent fs-freeze |
-| App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events` |
+| App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events`. **Monitoring:** `scripts/check-backup-freshness.sh` now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict **DEGRADED, exit 4** (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script |
 | Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 |
 
 ## Build (reproducible)
@@ -49,6 +49,8 @@ scripts/elway infra-ops@10.100.50.43 --playbook playbooks/nh3-svos-host.yaml
   `caddy hash-password --algorithm argon2id` on stdin (never `--plaintext`), rewrite the accounts file, `secret put`,
   then `systemctl reload caddy-svos`.
 - **Credential-free offline shell (svos-dev, 2026-10-05):** the service worker fetches its hash-pinned INVENTORY with `credentials:'omit'`, so exactly those files plus `/sw.js` are served BEFORE Basic auth (GET/HEAD, exact files, no fallback). Public build bytes only; `/`, `/status`, `/api/*`, the connector and everything else stay gated. The path list is **generated** into `/etc/caddy/svos-public-inventory` by **`/usr/local/sbin/svos-front-inventory`** (`configs/svos-front/`): it hash-checks every file against INVENTORY, validates, then reloads. ⚠ **Run it as root after every SVOS release**, or the next build's hashed asset names stay gated and worker install fails. `--check` changes nothing.
+- **Previous release's assets across a rollout:** the same tool writes `/etc/caddy/svos-public-previous`: when `/opt/svos/current` has moved, the release named in the old inventory header becomes "previous", and its hashed `/assets/*` files that the new build lacks are served credential-free from **that release's own static dir** (hash-checked). One previous release at most; comments only when there is none (true at 2026-10-05, one release). Replica-tested: old asset 200 with the old INVENTORY hash, new asset with the new hash, POST to the old asset 401, unknown asset 401/404; tampered files and an unsafe path make the tool refuse.
+- **Hosting bounds (svos-dev, per the web contract):** request bodies capped at exactly **1 MiB** on every route (413 over); **headers within 5 s, whole body within 15 s** (Caddy `read_header` / `read_body`); **no** write, response or idle deadline, so `/api/stream` (SSE) stays open. Live 2026-10-05 2014–2021 from fv-ml1: 1 MiB+1 → 413 on `/api/verbs` and `/verbs/pin`; exactly 1 MiB passes Caddy (kernel then answers); unfinished headers closed at 5.0 s; stalled body cut at 15.0 s; a signed `app-interface` ping returned pong before and DURING a stalled body; an SSE stream stayed open 41–45 s (client hung up) alongside all of it. ⚠ **A stalled body over HTTP/1.1 can come back as Caddy's empty `200 OK` (Content-Length 0)** instead of 504: replica 3/3 buffered, 2/3 unbuffered; live 2 of 3. Over h2, which browsers use, it is 502 (3/3 replica, 2/2 live). The app never gets a complete request either way; an empty 200 is not success. `handle_errors 499` and `request_buffers` do not change it (tested).
 - **Static routes (gated), per svoperatingsystem `web/README.md` Hosting contract:** `/` and `/status` → index.html; `/assets/*`, `/icons/*`, `/favicon.svg`, `/manifest.webmanifest`, `/index.html`, `/sw.js` → exact file, missing → 404; other HTML document requests (`Sec-Fetch-Dest: document` or `Accept: text/html`) → index.html; anything else → 404. Unversioned files get `Cache-Control: no-cache`.
 - **Acceptance matrix:** `configs/svos-front/acceptance-matrix.sh` (35 checks; usage in its header). 35/35 pass live at 1957 2026-10-05 from fv-ml1; against the previous config it reports 16 FAIL (positive control).
 - **Headers:** client copies of `Tailscale-User-*`, `X-Forwarded-*`, `Tailscale-Headers-Info`, `X-Real-IP`, `Forwarded`
@@ -71,7 +73,6 @@ scripts/elway infra-ops@10.100.50.43 --playbook playbooks/nh3-svos-host.yaml
     headers, `X-Forwarded-Proto: https`, `Host: svos.phasefinal.com`, and path `/health` (the `/api` prefix stripped);
   - connector route with right credentials → 502 (auth passed; the connector was not running); cert verifies; LAN
     10.100.50.43:443 → connection refused.
-- **Open (svos-dev's numbers needed):** the hosting contract also asks the boundary to bound request body size and read duration for the connector; not set yet.
 
 ### Original requirements (svos-dev)