feat(svos-front,backups): SVOS hosting bounds and previous-release assets; backup check flags fs-freeze failures and covers nh3-pve-2

Front door: request bodies capped at 1 MiB, headers within 5 s, body within 15 s, no response deadlines (SSE stays open). Validated live: 413 over the cap, stalled body cut at 15 s, signed ping and an SSE stream unaffected. svos-front-inventory also serves the previous release's hashed assets credential-free from that release's static dir, hash-checked. Backup check: reads the logs of vzdump tasks PVE records as OK and reports ERROR lines; an fs-freeze/thaw error is an application-consistency failure, new verdict DEGRADED (exit 4), wired into the alert wrapper. nh3-pve-2 added to the node and namespace lists (its job results were never read). The svos drill task is acknowledged by UPID; BACKUP_IGNORE_ACKS=1 counts it (positive control: exit 4).

configs/svos-front/Caddyfile

@@ -15,6 +15,16 @@
 	email lkraven@lkraven.com
 	auto_https disable_redirects
 	admin localhost:2019
+	# Hosting bounds (svos-dev 2026-10-05, svoperatingsystem web/README.md Hosting contract): request headers within 5 s,
+	# the whole request body within 15 s. NO write/response deadline: /api/stream (SSE) must stay open. Measured on a
+	# replica 2026-10-05 before deploy: SSE 25 s and a 20 s slow response survive on h1 and h2; a stalled body is cut at
+	# 15 s (h1 504, h2 502); unfinished headers are closed at 5 s.
+	servers {
+		timeouts {
+			read_header 5s
+			read_body 15s
+		}
+	}
 }
 
 https://svos.phasefinal.com:443 {
@@ -33,6 +43,12 @@ https://svos.phasefinal.com:443 {
 		request_header -Forwarded
 		request_header -X-SVOS-Peer
 
+		# Request bodies capped at exactly 1 MiB (1,048,576 B, = the kernel dispatcher's own cap) on every route;
+		# only /api/* and the connector read bodies at all. Over the cap -> 413. Responses are not limited.
+		request_body {
+			max_size 1MiB
+		}
+
 		# Credential-free offline shell (svos-dev 2026-10-05): the service worker fetches its hash-pinned INVENTORY
 		# with credentials:'omit', so exactly those files plus /sw.js are served BEFORE Basic auth, GET/HEAD only,
 		# exact files (no fallback). Public build bytes only; no API or connector path. The path list is GENERATED
@@ -47,6 +63,10 @@ https://svos.phasefinal.com:443 {
 			root * /opt/svos/current/static
 			file_server
 		}
+		# Previous release's hashed assets across a rollout (svos-dev 2026-10-05): open workers restore evicted inventory
+		# credential-free. Generated by the same tool: a matcher + handle rooted in THAT release's static dir, or
+		# comments only when there is no previous release.
+		import /etc/caddy/svos-public-previous
 
 		basic_auth argon2id SVOS {
 			import /etc/caddy/svos-basic-auth.accounts

configs/svos-front/svos-front-inventory

@@ -1,17 +1,26 @@
 #!/usr/bin/env python3
-"""svos-front-inventory: regenerate the SVOS front door's credential-free shell paths from the live release.
+"""svos-front-inventory: regenerate the SVOS front door's credential-free shell paths from the live release(s).
 
 eshpfi configs/svos-front/, installed at /usr/local/sbin/svos-front-inventory on nh3-svos. Run as root after every
 SVOS release (after /opt/svos/current moves), or the service worker's install breaks: its INVENTORY names hashed
 asset files, and only those exact paths (plus /sw.js) are served before Basic auth (svos-dev, 2026-10-05).
 
-What it does:
+Two generated files, both imported by /etc/caddy/Caddyfile, both root:caddy 0640:
-  1. Reads INVENTORY from /opt/svos/current/static/sw.js (a JSON object of path -> sha256).
+  /etc/caddy/svos-public-inventory  `path ...` of the CURRENT release's INVENTORY + /sw.js
-  2. Refuses unless every path is a plain absolute file path and every file exists with exactly that sha256.
+  /etc/caddy/svos-public-previous   a matcher + handle serving the PREVIOUS release's hashed /assets/* files that
-  3. Writes /etc/caddy/svos-public-inventory (one Caddyfile `path` line, root:caddy 0640) atomically.
+                                    the current build no longer has, from that release's own static dir. Already-
-  4. Validates the whole Caddyfile as the caddy user with the unit's EnvironmentFile (no secret on any argv),
+                                    open workers restore evicted inventory with credentials:'omit', so the old
-     restores the previous inventory file if validation fails, and otherwise reloads caddy-svos.
+                                    hashed assets must stay reachable credential-free across a rollout. Empty
---check: steps 1-2 only, plus a diff against the installed file; changes nothing.
+                                    (comments only) when there is no previous release.
+Which release is "previous" is explicit state, not a guess: it is the release named in the header of the installed
+svos-public-inventory when current has moved on from it; otherwise the previous file's own header is kept. One
+previous release at most (the web contract's one current/previous build record).
+
+Every path must be a plain absolute file path and every file must exist with exactly its INVENTORY sha256, in its
+own release, or nothing is written. After writing, the whole Caddyfile is validated as the caddy user with the
+unit's EnvironmentFile (no secret on any argv); on failure both previous files are restored and caddy is not
+reloaded. --check: verify and diff only, change nothing.
+Testing: SVOS_FRONT_TEST_ROOT=<dir> re-roots /opt/svos and /etc/caddy under <dir> and skips validate/reload/chown.
 """
 import hashlib
 import json
@@ -22,9 +31,13 @@ import subprocess
 import sys
 from pathlib import Path
 
-STATIC = Path("/opt/svos/current/static")
+TEST_ROOT = os.environ.get("SVOS_FRONT_TEST_ROOT")
-OUT = Path("/etc/caddy/svos-public-inventory")
+BASE = Path(TEST_ROOT) if TEST_ROOT else Path("/")
+CURRENT = BASE / "opt/svos/current"
+OUT = BASE / "etc/caddy/svos-public-inventory"
+PREV_OUT = BASE / "etc/caddy/svos-public-previous"
 SAFE = re.compile(r"^/[A-Za-z0-9._-]+(/[A-Za-z0-9._-]+)*$")
+HEADER = re.compile(r"^# GENERATED by \S+ from (\S+)/static/sw\.js", re.M)
 VALIDATE = ["systemd-run", "--quiet", "--wait", "--pipe", "--collect", "-p", "User=caddy", "-p", "Group=caddy",
             "-p", "EnvironmentFile=/etc/caddy/env",
             "-p", "Environment=XDG_DATA_HOME=/var/lib/caddy XDG_CONFIG_HOME=/var/lib/caddy",
@@ -35,56 +48,103 @@ def die(msg):
     sys.exit(f"svos-front-inventory: {msg}")
 
 
-def inventory():
+def inventory(release):
-    src = (STATIC / "sw.js").read_text(encoding="utf-8")
+    """path -> sha256 from <release>/static/sw.js, unchecked."""
+    src = (Path(release) / "static/sw.js").read_text(encoding="utf-8")
     m = re.search(r"^const INVENTORY = (\{.*?\});$", src, re.M)
     if not m:
-        die("no `const INVENTORY = {...};` line in sw.js")
+        die(f"no `const INVENTORY = {{...}};` line in {release}/static/sw.js")
     inv = json.loads(m.group(1))
     if not inv:
-        die("INVENTORY is empty")
+        die(f"INVENTORY is empty in {release}")
-    for path, digest in inv.items():
+    return inv
+
+
+def verify(release, inv, paths):
+    static = Path(release) / "static"
+    for path in paths:
         if not SAFE.match(path) or ".." in path.split("/"):
-            die(f"unsafe inventory path {path!r}")
+            die(f"unsafe inventory path {path!r} in {release}")
-        f = STATIC / path.lstrip("/")
+        f = static / path.lstrip("/")
         if not f.is_file() or f.is_symlink():
-            die(f"{path} is not a regular file in {STATIC}")
+            die(f"{path} is not a regular file in {static}")
         actual = hashlib.sha256(f.read_bytes()).hexdigest()
-        if actual != digest:
+        if actual != inv[path]:
-            die(f"{path} sha256 {actual} != INVENTORY {digest}")
+            die(f"{path} in {release}: sha256 {actual} != INVENTORY {inv[path]}")
-    return sorted(inv)
 
 
-def render(paths):
+def header_release(text):
-    release = os.path.realpath(STATIC.parent)
+    m = HEADER.search(text)
-    return (f"# GENERATED by /usr/local/sbin/svos-front-inventory from {release}/static/sw.js; do not edit.\n"
+    return m.group(1) if m else None
-            f"# Exact credential-free shell paths (hash-checked), plus /sw.js.\n"
+
-            f"path {' '.join(paths + ['/sw.js'])}\n")
+
+def shown(release):
+    """The release path as Caddy and humans see it on the host (without the test re-root)."""
+    return "/" + os.path.relpath(release, BASE) if TEST_ROOT else release
+
+
+def real(shown_path):
+    return str(BASE / shown_path.lstrip("/")) if TEST_ROOT else shown_path
 
 
 def main():
-    new = render(inventory())
+    check = "--check" in sys.argv[1:]
-    old = OUT.read_text(encoding="utf-8") if OUT.exists() else ""
+    cur = os.path.realpath(CURRENT)
-    if "--check" in sys.argv[1:]:
+    cur_inv = inventory(cur)
-        print("inventory ok;", "installed file is current" if old == new else "installed file DIFFERS:\n" + new)
+    verify(cur, cur_inv, sorted(cur_inv))
+    new_cur = (f"# GENERATED by /usr/local/sbin/svos-front-inventory from {shown(cur)}/static/sw.js; do not edit.\n"
+               f"# Exact credential-free shell paths (hash-checked), plus /sw.js.\n"
+               f"path {' '.join(sorted(cur_inv) + ['/sw.js'])}\n")
+
+    old_cur = OUT.read_text(encoding="utf-8") if OUT.exists() else ""
+    old_prev = PREV_OUT.read_text(encoding="utf-8") if PREV_OUT.exists() else ""
+    candidates = [header_release(old_cur), header_release(old_prev)]
+    prev = next((r for r in candidates if r and real(r) != cur and os.path.isdir(real(r))), None)
+
+    if prev:
+        prev_inv = inventory(real(prev))
+        keep = sorted(p for p in prev_inv if p.startswith("/assets/") and p not in cur_inv)
+        verify(real(prev), prev_inv, keep)
+    if prev and keep:
+        new_prev = (f"# GENERATED by /usr/local/sbin/svos-front-inventory from {prev}/static/sw.js; do not edit.\n"
+                    f"# Previous release's hashed assets the current build no longer has (hash-checked), served\n"
+                    f"# credential-free from that release's own static dir.\n"
+                    f"@public_previous {{\n\tmethod GET HEAD\n\tpath {' '.join(keep)}\n}}\n"
+                    f"handle @public_previous {{\n\troot * {prev}/static\n\tfile_server\n}}\n")
+    else:
+        new_prev = ("# GENERATED by /usr/local/sbin/svos-front-inventory; do not edit.\n"
+                    "# No previous release with assets to keep: nothing is served from here.\n")
+
+    if check:
+        for name, old, new in (("current", old_cur, new_cur), ("previous", old_prev, new_prev)):
+            print(f"{name}: " + ("installed file is current" if old == new else "installed file DIFFERS:\n" + new))
         return
-    if old == new:
+    if old_cur == new_cur and old_prev == new_prev:
         print("unchanged; nothing to do")
         return
-    tmp = OUT.with_name(OUT.name + ".new")
+
-    tmp.write_text(new, encoding="utf-8")
+    saved = {}
+    for path, text in ((OUT, new_cur), (PREV_OUT, new_prev)):
+        if path.exists():
+            saved[path] = path.read_text(encoding="utf-8")
+        tmp = path.with_name(path.name + ".new")
+        tmp.write_text(text, encoding="utf-8")
+        if not TEST_ROOT:
             shutil.chown(tmp, "root", "caddy")
         os.chmod(tmp, 0o640)
-    backup = OUT.with_name(OUT.name + ".prev")
+        os.replace(tmp, path)
-    if OUT.exists():
+    if TEST_ROOT:
-        shutil.copy2(OUT, backup)
+        print("TEST: files written, validate/reload skipped")
-    os.replace(tmp, OUT)
+        return
     if subprocess.run(VALIDATE).returncode != 0:
-        if backup.exists():
+        for path in (OUT, PREV_OUT):
-            os.replace(backup, OUT)
+            if path in saved:
-        die("caddy validate FAILED; previous inventory restored, caddy not reloaded")
+                path.write_text(saved[path], encoding="utf-8")
+            else:
+                path.unlink()
+        die("caddy validate FAILED; previous files restored, caddy not reloaded")
     subprocess.run(["systemctl", "reload", "caddy-svos"], check=True)
-    print("inventory updated and caddy-svos reloaded:\n" + new)
+    print("updated and caddy-svos reloaded:\n" + new_cur + new_prev)
 
 
 if __name__ == "__main__":

persistent-memory.md

@@ -301,7 +301,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
 
 ## Recent decisions
 
-- `[2026-10-05]` **nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only `https://svos.phasefinal.com` with HTTP Basic auth, login `Vuong` (Prime, direct)** — WhoIs gate retired (all mesh nodes are login `pfi`; Prime declined a mesh restructure). Password vaulted `nh3-svos/basic-auth-vuong`. qemu-ga freeze hooks were never run on Debian; now `configs/qemu-ga/fsfreeze-dispatch` (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"). Offline shell assets served before auth; rerun `svos-front-inventory` after every SVOS release. → `servers/nh3-svos/README.md`.
+- `[2026-10-05]` **nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only `https://svos.phasefinal.com` with HTTP Basic auth, login `Vuong` (Prime, direct)** — WhoIs gate retired (all mesh nodes are login `pfi`; Prime declined a mesh restructure). Password vaulted `nh3-svos/basic-auth-vuong`. qemu-ga freeze hooks were never run on Debian; now `configs/qemu-ga/fsfreeze-dispatch` (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"); the backup check now flags that as DEGRADED (exit 4) and covers nh3-pve-2, which it had never read. Offline shell (+ previous release's assets) served before auth; rerun `svos-front-inventory` after every SVOS release. Hosting bounds 1 MiB / 15 s body / 5 s headers. → `servers/nh3-svos/README.md`.
 - `[2026-10-05]` **fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE)** — run 1 aborted on its own guard (live rsync walked into an overlay `merged` view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (`-x`, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → `servers/fv-ml1/README.md`
 - `[2026-10-05]` **ana-docker: Docker CE repo bullseye → bookworm, 29.8.2 (Prime)** — with `live-restore: true` the engine upgrade kept containers up, but 6 that bind-mount `docker.sock` (crowdsec, traefik, dozzle, beszel-agent, dockge, gitea-runner) went stale and needed restarts. → `servers/ana-docker/README.md`
 - `[2026-10-05]` **RMM wildcard cert now auto-renews (Prime)** — `*.phasefinal.com` on pfi-tacticalrmm was certbot `manual` (could never renew; was 16 days from expiry). Now dns-cloudflare with a phasefinal.com-only token (vault `pfi-tacticalrmm/cloudflare-certbot-dns01-token`); reissued to 2027-01-03, dry-run OK. → `servers/pfi-tacticalrmm/README.md`

scripts/backup-freshness-alert.sh

@@ -32,6 +32,8 @@
 #   1  backups STALE/DOWN, and the alert was delivered
 #   2  the ALERT PATH ITSELF FAILED (investigate this before the backups)
 #   3  ERRORED-JOBS — every body fresh, a job errored recently; alert delivered
+#   4  DEGRADED — no job errored, but an OK task logged an ERROR (fs-freeze refused =
+#      crash-consistent image only); alert delivered
 #
 # 3 is deliberately not 1. A nightly job that errored is worth a look; a stale
 # backup body is worth a page. Reporting both as "STALE" over a fleet whose
@@ -83,6 +85,9 @@ printf '%s\n' "$out"
 if [ "$rc" -eq 3 ]; then
   subject="🟡 Backup jobs errored ($(date '+%Y-%m-%d')) — all bodies fresh"
   lead="A vzdump job errored recently. Every backup body on the fleet is FRESH; nothing is stale."
+elif [ "$rc" -eq 4 ]; then
+  subject="🟠 Backup DEGRADED ($(date '+%Y-%m-%d')) — an OK job logged an ERROR"
+  lead="Every backup body is FRESH and no job failed, but a backup logged an ERROR inside a task PVE calls OK. An fs-freeze failure there means that image is crash-consistent only, not application-consistent."
 else
   subject="🔴 Backup freshness ALERT ($(date '+%Y-%m-%d'))"
   lead="Automated daily backup-freshness check found STALE or DOWN backup layer(s) on the PFI fleet."

scripts/check-backup-freshness.sh

@@ -8,6 +8,9 @@
 #   0  all backups fresh
 #   1  STALE — a backup body is past the threshold, or an endpoint is down
 #   3  ERRORED-JOBS — every body is fresh, but a vzdump job errored recently
+#   4  DEGRADED — no job errored, but a backup in the window logged an ERROR inside a
+#      task PVE still calls OK; above all an fs-freeze refusal, which means the image
+#      is crash-consistent, NOT application-consistent (see the layer below)
 #
 # 1 and 3 are deliberately different. Collapsing them prints "STALE" over a
 # fleet whose every backup is current, which is how an alarm earns being
@@ -55,6 +58,8 @@ declare -A NS_NODE=(
   [ana-pve]=infra-ops@10.250.250.31
   [esh-pve]=infra-ops@10.0.250.35
   [nh3-pve]=infra-ops@10.100.250.60
+  [nh3-pve-2]=infra-ops@10.100.250.62   # standalone (not clustered): own jobs.cfg, own namespace. Added 2026-10-05;
+                                       # until then its job results were never read at all
 )
 declare -A NS_JOBS=()      # ns → one "enabled|all|vmids|excludes" line per job
 declare -A NS_KNOWN=()     # ns → 1 once coverage was successfully read
@@ -80,9 +85,17 @@ PVE_NODES=(
   "esh-pve       infra-ops@10.0.250.35"
   "esh-nas-pve   infra-ops@10.0.50.55"
   "nh3-pve       infra-ops@10.100.250.60"
+  "nh3-pve-2     infra-ops@10.100.250.62"
 )
 JOB_WINDOW_H="${BACKUP_JOB_WINDOW_HOURS:-36}"
 jobfail=()
+degraded=()
+acked=()
+# Acknowledged OK-but-ERROR tasks: a known, deliberate cause (a drill), printed every run in their own section and
+# never counted. Key = the full UPID, value = why. Keep the paper trail; the entries age out with the window.
+declare -A ACKED_TASKS=(
+  [UPID:nh3-pve-2:0004FFBA:00719F66:6AC4628F:vzdump:111:root@pam:]="2026-10-05 svos backup drill B: freeze refused ON PURPOSE by a temporary hook (servers/nh3-svos/README.md)"
+)
 
 load_jobs() { # $1 = namespace
   local ns="$1" node="${NS_NODE[$1]:-}" raw
@@ -257,6 +270,57 @@ for entry in "${PVE_NODES[@]}"; do
   done <<<"$bad"
 done
 
+# --- Layer: did the jobs that SUCCEEDED actually quiesce? --------------------
+#
+# ⚠ A task PVE records as OK can still have failed the thing that matters. Measured 2026-10-05 (nh3-pve-2, VM 111,
+# svos backup drill): when the guest REFUSES fs-freeze, vzdump logs
+#   ERROR: unable to freeze guest fs - child process has failed to execute fsfreeze hook: exit status 1
+# then backs up UNFROZEN and ends "Backup job finished successfully", task status OK, exit 0. The image is
+# crash-consistent, not application-consistent, and the job-success layer above cannot see it. So: read the task
+# LOG of every OK vzdump task in the window, and report any ERROR line. Freeze/thaw errors are named as
+# application-consistency failures; anything else is reported as an ERROR inside an OK task.
+# The remote side stays a plain grep per file (see load_jobs on why embedded awk is avoided). A task log that
+# cannot be read is an error, never a silent pass.
+for entry in "${PVE_NODES[@]}"; do
+  set -- $entry; node_label="$1"; node_addr="$2"
+  idx=$($SSH "$node_addr" "sudo -n cat /var/log/pve/tasks/index" 2>/dev/null)
+  [ -n "$idx" ] || continue   # already reported UNREADABLE by the layer above
+  okups=$(awk -v now="$now" -v win="$((JOB_WINDOW_H * 3600))" '
+    { n = split($1, f, ":"); if (n < 7 || f[6] != "vzdump") next
+      if (now - strtonum("0x" f[5]) > win) next
+      st = $0; sub(/^[^ ]+ [^ ]+ /, "", st); if (st == "OK") print $1 }' <<<"$idx")
+  [ -n "$okups" ] || continue
+  script=""
+  while IFS= read -r u; do
+    [ -n "$u" ] || continue
+    sh=$(cut -d: -f5 <<<"$u"); d=${sh: -1}
+    script+="f='/var/log/pve/tasks/$d/$u'; if [ -r \"\$f\" ]; then grep -E '^(INFO: Starting Backup of VM|ERROR:)' \"\$f\" | sed 's|^|$u\t|'; else printf 'MISSING\t%s\n' '$u'; fi
+"
+  done <<<"$okups"
+  hits=$($SSH "$node_addr" "sudo -n bash -s" <<<"$script" 2>/dev/null) || { errors+=("vzdump task LOGS on $node_label unreadable — freeze failures there are invisible"); continue; }
+  # attribute each ERROR to the guest whose "Starting Backup of VM" line came last in that task
+  while IFS=$'\t' read -r u line; do
+    [ -n "$u" ] || continue
+    if [ "$u" = "MISSING" ]; then errors+=("vzdump task log missing on $node_label: $line"); continue; fi
+    case "$line" in
+      "INFO: Starting Backup of VM "*) cur=${line#INFO: Starting Backup of VM }; cur=${cur%% *}; continue ;;
+    esac
+    gid=${cur:-$(cut -d: -f7 <<<"$u")}
+    ts=$(date -d "@$((16#$(cut -d: -f5 <<<"$u")))" '+%m-%d %H:%M' 2>/dev/null)
+    if grep -qiE 'freeze|thaw' <<<"$line"; then
+      msg="$node_label guest $gid ($ts): APPLICATION-CONSISTENCY FAILURE, image is crash-consistent only :: ${line#ERROR: }"
+    else
+      msg="$node_label guest $gid ($ts): ERROR inside a task PVE calls OK :: ${line#ERROR: }"
+    fi
+    if [ -n "${ACKED_TASKS[$u]:-}" ] && [ -z "${BACKUP_IGNORE_ACKS:-}" ]; then
+      acked+=("$msg  [ack: ${ACKED_TASKS[$u]}]")
+    else
+      degraded+=("$msg")
+    fi
+  done <<<"$hits"
+  cur=""
+done
+
 # --- restic CONTENT assertion (operator ruling 2026-09-22) -------------------
 #
 # The checks above read a snapshot directory's MTIME on the NAS. That proves a
@@ -347,6 +411,14 @@ if [ "${#jobfail[@]}" -gt 0 ]; then
   echo; echo "BACKUP JOBS THAT ERRORED in the last ${JOB_WINDOW_H}h (${#jobfail[@]}):"
   printf '  ❌ %s\n' "${jobfail[@]}"
 fi
+if [ "${#degraded[@]}" -gt 0 ]; then
+  echo; echo "DEGRADED — task OK, but the log has an ERROR (last ${JOB_WINDOW_H}h, ${#degraded[@]}):"
+  printf '  🟠 %s\n' "${degraded[@]}"
+fi
+if [ "${#acked[@]}" -gt 0 ]; then
+  echo; echo "ACKNOWLEDGED (known cause, not counted; BACKUP_IGNORE_ACKS=1 counts them):"
+  printf '  ☑  %s\n' "${acked[@]}"
+fi
 if [ "${#stale[@]}" -gt 0 ] || [ "${#errors[@]}" -gt 0 ]; then
   echo; echo "STALE / PROBLEMS (${#stale[@]}+${#errors[@]}):"
   [ "${#stale[@]}"  -gt 0 ] && printf '  🔴 %s\n' "${stale[@]}"
@@ -375,5 +447,11 @@ if [ "${#jobfail[@]}" -gt 0 ]; then
   echo "        Errors older than the window age out on their own."
   exit 3
 fi
+if [ "${#degraded[@]}" -gt 0 ]; then
+  echo; echo "RESULT: DEGRADED — every body is FRESH and no job errored, but ${#degraded[@]} backup(s) logged an"
+  echo "        ERROR inside an OK task. An fs-freeze failure means that image is crash-consistent ONLY:"
+  echo "        do not count it as application-consistent. Worth a look, not a page."
+  exit 4
+fi
 echo; echo "RESULT: all backups fresh"
 exit 0

servers/nh3-svos/README.md

@@ -20,7 +20,7 @@ configuration, enrollment and acceptance.
 | Releases | `/opt/svos` root:svc-svos 2775, with `releases/` for svos-dev's layout. The git checkout is not a runtime dependency |
 | Docker | none, by design |
 | Backups | nh3-pve-2 vzdump job, nightly **21:30**, `all 1` → pbs-ana namespace `nh3-pve-2` (mirrored to pbs-nh3 at 06:30), snapshot mode with guest-agent fs-freeze |
-| App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events` |
+| App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events`. **Monitoring:** `scripts/check-backup-freshness.sh` now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict **DEGRADED, exit 4** (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script |
 | Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 |
 
 ## Build (reproducible)
@@ -49,6 +49,8 @@ scripts/elway infra-ops@10.100.50.43 --playbook playbooks/nh3-svos-host.yaml
   `caddy hash-password --algorithm argon2id` on stdin (never `--plaintext`), rewrite the accounts file, `secret put`,
   then `systemctl reload caddy-svos`.
 - **Credential-free offline shell (svos-dev, 2026-10-05):** the service worker fetches its hash-pinned INVENTORY with `credentials:'omit'`, so exactly those files plus `/sw.js` are served BEFORE Basic auth (GET/HEAD, exact files, no fallback). Public build bytes only; `/`, `/status`, `/api/*`, the connector and everything else stay gated. The path list is **generated** into `/etc/caddy/svos-public-inventory` by **`/usr/local/sbin/svos-front-inventory`** (`configs/svos-front/`): it hash-checks every file against INVENTORY, validates, then reloads. ⚠ **Run it as root after every SVOS release**, or the next build's hashed asset names stay gated and worker install fails. `--check` changes nothing.
+- **Previous release's assets across a rollout:** the same tool writes `/etc/caddy/svos-public-previous`: when `/opt/svos/current` has moved, the release named in the old inventory header becomes "previous", and its hashed `/assets/*` files that the new build lacks are served credential-free from **that release's own static dir** (hash-checked). One previous release at most; comments only when there is none (true at 2026-10-05, one release). Replica-tested: old asset 200 with the old INVENTORY hash, new asset with the new hash, POST to the old asset 401, unknown asset 401/404; tampered files and an unsafe path make the tool refuse.
+- **Hosting bounds (svos-dev, per the web contract):** request bodies capped at exactly **1 MiB** on every route (413 over); **headers within 5 s, whole body within 15 s** (Caddy `read_header` / `read_body`); **no** write, response or idle deadline, so `/api/stream` (SSE) stays open. Live 2026-10-05 2014–2021 from fv-ml1: 1 MiB+1 → 413 on `/api/verbs` and `/verbs/pin`; exactly 1 MiB passes Caddy (kernel then answers); unfinished headers closed at 5.0 s; stalled body cut at 15.0 s; a signed `app-interface` ping returned pong before and DURING a stalled body; an SSE stream stayed open 41–45 s (client hung up) alongside all of it. ⚠ **A stalled body over HTTP/1.1 can come back as Caddy's empty `200 OK` (Content-Length 0)** instead of 504: replica 3/3 buffered, 2/3 unbuffered; live 2 of 3. Over h2, which browsers use, it is 502 (3/3 replica, 2/2 live). The app never gets a complete request either way; an empty 200 is not success. `handle_errors 499` and `request_buffers` do not change it (tested).
 - **Static routes (gated), per svoperatingsystem `web/README.md` Hosting contract:** `/` and `/status` → index.html; `/assets/*`, `/icons/*`, `/favicon.svg`, `/manifest.webmanifest`, `/index.html`, `/sw.js` → exact file, missing → 404; other HTML document requests (`Sec-Fetch-Dest: document` or `Accept: text/html`) → index.html; anything else → 404. Unversioned files get `Cache-Control: no-cache`.
 - **Acceptance matrix:** `configs/svos-front/acceptance-matrix.sh` (35 checks; usage in its header). 35/35 pass live at 1957 2026-10-05 from fv-ml1; against the previous config it reports 16 FAIL (positive control).
 - **Headers:** client copies of `Tailscale-User-*`, `X-Forwarded-*`, `Tailscale-Headers-Info`, `X-Real-IP`, `Forwarded`
@@ -71,7 +73,6 @@ scripts/elway infra-ops@10.100.50.43 --playbook playbooks/nh3-svos-host.yaml
     headers, `X-Forwarded-Proto: https`, `Host: svos.phasefinal.com`, and path `/health` (the `/api` prefix stripped);
   - connector route with right credentials → 502 (auth passed; the connector was not running); cert verifies; LAN
     10.100.50.43:443 → connection refused.
-- **Open (svos-dev's numbers needed):** the hosting contract also asks the boundary to bound request body size and read duration for the connector; not set yet.
 
 ### Original requirements (svos-dev)