From 1210b4a0d524517fe0bb5255aa76962ed77415d2 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 5 Oct 2026 20:22:16 -0700 Subject: [PATCH] feat(svos-front,backups): SVOS hosting bounds and previous-release assets; backup check flags fs-freeze failures and covers nh3-pve-2 Front door: request bodies capped at 1 MiB, headers within 5 s, body within 15 s, no response deadlines (SSE stays open). Validated live: 413 over the cap, stalled body cut at 15 s, signed ping and an SSE stream unaffected. svos-front-inventory also serves the previous release's hashed assets credential-free from that release's static dir, hash-checked. Backup check: reads the logs of vzdump tasks PVE records as OK and reports ERROR lines; an fs-freeze/thaw error is an application-consistency failure, new verdict DEGRADED (exit 4), wired into the alert wrapper. nh3-pve-2 added to the node and namespace lists (its job results were never read). The svos drill task is acknowledged by UPID; BACKUP_IGNORE_ACKS=1 counts it (positive control: exit 4). --- configs/svos-front/Caddyfile | 20 ++++ configs/svos-front/svos-front-inventory | 146 +++++++++++++++++------- persistent-memory.md | 2 +- scripts/backup-freshness-alert.sh | 5 + scripts/check-backup-freshness.sh | 78 +++++++++++++ servers/nh3-svos/README.md | 5 +- 6 files changed, 210 insertions(+), 46 deletions(-) diff --git a/configs/svos-front/Caddyfile b/configs/svos-front/Caddyfile index 9542a3e..9b312e7 100644 --- a/configs/svos-front/Caddyfile +++ b/configs/svos-front/Caddyfile @@ -15,6 +15,16 @@ email lkraven@lkraven.com auto_https disable_redirects admin localhost:2019 + # Hosting bounds (svos-dev 2026-10-05, svoperatingsystem web/README.md Hosting contract): request headers within 5 s, + # the whole request body within 15 s. NO write/response deadline: /api/stream (SSE) must stay open. Measured on a + # replica 2026-10-05 before deploy: SSE 25 s and a 20 s slow response survive on h1 and h2; a stalled body is cut at + # 15 s (h1 504, h2 502); unfinished headers are closed at 5 s. + servers { + timeouts { + read_header 5s + read_body 15s + } + } } https://svos.phasefinal.com:443 { @@ -33,6 +43,12 @@ https://svos.phasefinal.com:443 { request_header -Forwarded request_header -X-SVOS-Peer + # Request bodies capped at exactly 1 MiB (1,048,576 B, = the kernel dispatcher's own cap) on every route; + # only /api/* and the connector read bodies at all. Over the cap -> 413. Responses are not limited. + request_body { + max_size 1MiB + } + # Credential-free offline shell (svos-dev 2026-10-05): the service worker fetches its hash-pinned INVENTORY # with credentials:'omit', so exactly those files plus /sw.js are served BEFORE Basic auth, GET/HEAD only, # exact files (no fallback). Public build bytes only; no API or connector path. The path list is GENERATED @@ -47,6 +63,10 @@ https://svos.phasefinal.com:443 { root * /opt/svos/current/static file_server } + # Previous release's hashed assets across a rollout (svos-dev 2026-10-05): open workers restore evicted inventory + # credential-free. Generated by the same tool: a matcher + handle rooted in THAT release's static dir, or + # comments only when there is no previous release. + import /etc/caddy/svos-public-previous basic_auth argon2id SVOS { import /etc/caddy/svos-basic-auth.accounts diff --git a/configs/svos-front/svos-front-inventory b/configs/svos-front/svos-front-inventory index f7ff08f..44e71f4 100755 --- a/configs/svos-front/svos-front-inventory +++ b/configs/svos-front/svos-front-inventory @@ -1,17 +1,26 @@ #!/usr/bin/env python3 -"""svos-front-inventory: regenerate the SVOS front door's credential-free shell paths from the live release. +"""svos-front-inventory: regenerate the SVOS front door's credential-free shell paths from the live release(s). eshpfi configs/svos-front/, installed at /usr/local/sbin/svos-front-inventory on nh3-svos. Run as root after every SVOS release (after /opt/svos/current moves), or the service worker's install breaks: its INVENTORY names hashed asset files, and only those exact paths (plus /sw.js) are served before Basic auth (svos-dev, 2026-10-05). -What it does: - 1. Reads INVENTORY from /opt/svos/current/static/sw.js (a JSON object of path -> sha256). - 2. Refuses unless every path is a plain absolute file path and every file exists with exactly that sha256. - 3. Writes /etc/caddy/svos-public-inventory (one Caddyfile `path` line, root:caddy 0640) atomically. - 4. Validates the whole Caddyfile as the caddy user with the unit's EnvironmentFile (no secret on any argv), - restores the previous inventory file if validation fails, and otherwise reloads caddy-svos. ---check: steps 1-2 only, plus a diff against the installed file; changes nothing. +Two generated files, both imported by /etc/caddy/Caddyfile, both root:caddy 0640: + /etc/caddy/svos-public-inventory `path ...` of the CURRENT release's INVENTORY + /sw.js + /etc/caddy/svos-public-previous a matcher + handle serving the PREVIOUS release's hashed /assets/* files that + the current build no longer has, from that release's own static dir. Already- + open workers restore evicted inventory with credentials:'omit', so the old + hashed assets must stay reachable credential-free across a rollout. Empty + (comments only) when there is no previous release. +Which release is "previous" is explicit state, not a guess: it is the release named in the header of the installed +svos-public-inventory when current has moved on from it; otherwise the previous file's own header is kept. One +previous release at most (the web contract's one current/previous build record). + +Every path must be a plain absolute file path and every file must exist with exactly its INVENTORY sha256, in its +own release, or nothing is written. After writing, the whole Caddyfile is validated as the caddy user with the +unit's EnvironmentFile (no secret on any argv); on failure both previous files are restored and caddy is not +reloaded. --check: verify and diff only, change nothing. +Testing: SVOS_FRONT_TEST_ROOT= re-roots /opt/svos and /etc/caddy under and skips validate/reload/chown. """ import hashlib import json @@ -22,9 +31,13 @@ import subprocess import sys from pathlib import Path -STATIC = Path("/opt/svos/current/static") -OUT = Path("/etc/caddy/svos-public-inventory") +TEST_ROOT = os.environ.get("SVOS_FRONT_TEST_ROOT") +BASE = Path(TEST_ROOT) if TEST_ROOT else Path("/") +CURRENT = BASE / "opt/svos/current" +OUT = BASE / "etc/caddy/svos-public-inventory" +PREV_OUT = BASE / "etc/caddy/svos-public-previous" SAFE = re.compile(r"^/[A-Za-z0-9._-]+(/[A-Za-z0-9._-]+)*$") +HEADER = re.compile(r"^# GENERATED by \S+ from (\S+)/static/sw\.js", re.M) VALIDATE = ["systemd-run", "--quiet", "--wait", "--pipe", "--collect", "-p", "User=caddy", "-p", "Group=caddy", "-p", "EnvironmentFile=/etc/caddy/env", "-p", "Environment=XDG_DATA_HOME=/var/lib/caddy XDG_CONFIG_HOME=/var/lib/caddy", @@ -35,56 +48,103 @@ def die(msg): sys.exit(f"svos-front-inventory: {msg}") -def inventory(): - src = (STATIC / "sw.js").read_text(encoding="utf-8") +def inventory(release): + """path -> sha256 from /static/sw.js, unchecked.""" + src = (Path(release) / "static/sw.js").read_text(encoding="utf-8") m = re.search(r"^const INVENTORY = (\{.*?\});$", src, re.M) if not m: - die("no `const INVENTORY = {...};` line in sw.js") + die(f"no `const INVENTORY = {{...}};` line in {release}/static/sw.js") inv = json.loads(m.group(1)) if not inv: - die("INVENTORY is empty") - for path, digest in inv.items(): + die(f"INVENTORY is empty in {release}") + return inv + + +def verify(release, inv, paths): + static = Path(release) / "static" + for path in paths: if not SAFE.match(path) or ".." in path.split("/"): - die(f"unsafe inventory path {path!r}") - f = STATIC / path.lstrip("/") + die(f"unsafe inventory path {path!r} in {release}") + f = static / path.lstrip("/") if not f.is_file() or f.is_symlink(): - die(f"{path} is not a regular file in {STATIC}") + die(f"{path} is not a regular file in {static}") actual = hashlib.sha256(f.read_bytes()).hexdigest() - if actual != digest: - die(f"{path} sha256 {actual} != INVENTORY {digest}") - return sorted(inv) + if actual != inv[path]: + die(f"{path} in {release}: sha256 {actual} != INVENTORY {inv[path]}") -def render(paths): - release = os.path.realpath(STATIC.parent) - return (f"# GENERATED by /usr/local/sbin/svos-front-inventory from {release}/static/sw.js; do not edit.\n" - f"# Exact credential-free shell paths (hash-checked), plus /sw.js.\n" - f"path {' '.join(paths + ['/sw.js'])}\n") +def header_release(text): + m = HEADER.search(text) + return m.group(1) if m else None + + +def shown(release): + """The release path as Caddy and humans see it on the host (without the test re-root).""" + return "/" + os.path.relpath(release, BASE) if TEST_ROOT else release + + +def real(shown_path): + return str(BASE / shown_path.lstrip("/")) if TEST_ROOT else shown_path def main(): - new = render(inventory()) - old = OUT.read_text(encoding="utf-8") if OUT.exists() else "" - if "--check" in sys.argv[1:]: - print("inventory ok;", "installed file is current" if old == new else "installed file DIFFERS:\n" + new) + check = "--check" in sys.argv[1:] + cur = os.path.realpath(CURRENT) + cur_inv = inventory(cur) + verify(cur, cur_inv, sorted(cur_inv)) + new_cur = (f"# GENERATED by /usr/local/sbin/svos-front-inventory from {shown(cur)}/static/sw.js; do not edit.\n" + f"# Exact credential-free shell paths (hash-checked), plus /sw.js.\n" + f"path {' '.join(sorted(cur_inv) + ['/sw.js'])}\n") + + old_cur = OUT.read_text(encoding="utf-8") if OUT.exists() else "" + old_prev = PREV_OUT.read_text(encoding="utf-8") if PREV_OUT.exists() else "" + candidates = [header_release(old_cur), header_release(old_prev)] + prev = next((r for r in candidates if r and real(r) != cur and os.path.isdir(real(r))), None) + + if prev: + prev_inv = inventory(real(prev)) + keep = sorted(p for p in prev_inv if p.startswith("/assets/") and p not in cur_inv) + verify(real(prev), prev_inv, keep) + if prev and keep: + new_prev = (f"# GENERATED by /usr/local/sbin/svos-front-inventory from {prev}/static/sw.js; do not edit.\n" + f"# Previous release's hashed assets the current build no longer has (hash-checked), served\n" + f"# credential-free from that release's own static dir.\n" + f"@public_previous {{\n\tmethod GET HEAD\n\tpath {' '.join(keep)}\n}}\n" + f"handle @public_previous {{\n\troot * {prev}/static\n\tfile_server\n}}\n") + else: + new_prev = ("# GENERATED by /usr/local/sbin/svos-front-inventory; do not edit.\n" + "# No previous release with assets to keep: nothing is served from here.\n") + + if check: + for name, old, new in (("current", old_cur, new_cur), ("previous", old_prev, new_prev)): + print(f"{name}: " + ("installed file is current" if old == new else "installed file DIFFERS:\n" + new)) return - if old == new: + if old_cur == new_cur and old_prev == new_prev: print("unchanged; nothing to do") return - tmp = OUT.with_name(OUT.name + ".new") - tmp.write_text(new, encoding="utf-8") - shutil.chown(tmp, "root", "caddy") - os.chmod(tmp, 0o640) - backup = OUT.with_name(OUT.name + ".prev") - if OUT.exists(): - shutil.copy2(OUT, backup) - os.replace(tmp, OUT) + + saved = {} + for path, text in ((OUT, new_cur), (PREV_OUT, new_prev)): + if path.exists(): + saved[path] = path.read_text(encoding="utf-8") + tmp = path.with_name(path.name + ".new") + tmp.write_text(text, encoding="utf-8") + if not TEST_ROOT: + shutil.chown(tmp, "root", "caddy") + os.chmod(tmp, 0o640) + os.replace(tmp, path) + if TEST_ROOT: + print("TEST: files written, validate/reload skipped") + return if subprocess.run(VALIDATE).returncode != 0: - if backup.exists(): - os.replace(backup, OUT) - die("caddy validate FAILED; previous inventory restored, caddy not reloaded") + for path in (OUT, PREV_OUT): + if path in saved: + path.write_text(saved[path], encoding="utf-8") + else: + path.unlink() + die("caddy validate FAILED; previous files restored, caddy not reloaded") subprocess.run(["systemctl", "reload", "caddy-svos"], check=True) - print("inventory updated and caddy-svos reloaded:\n" + new) + print("updated and caddy-svos reloaded:\n" + new_cur + new_prev) if __name__ == "__main__": diff --git a/persistent-memory.md b/persistent-memory.md index 8cb119b..38d79dc 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -301,7 +301,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca ## Recent decisions -- `[2026-10-05]` **nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only `https://svos.phasefinal.com` with HTTP Basic auth, login `Vuong` (Prime, direct)** — WhoIs gate retired (all mesh nodes are login `pfi`; Prime declined a mesh restructure). Password vaulted `nh3-svos/basic-auth-vuong`. qemu-ga freeze hooks were never run on Debian; now `configs/qemu-ga/fsfreeze-dispatch` (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"). Offline shell assets served before auth; rerun `svos-front-inventory` after every SVOS release. → `servers/nh3-svos/README.md`. +- `[2026-10-05]` **nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only `https://svos.phasefinal.com` with HTTP Basic auth, login `Vuong` (Prime, direct)** — WhoIs gate retired (all mesh nodes are login `pfi`; Prime declined a mesh restructure). Password vaulted `nh3-svos/basic-auth-vuong`. qemu-ga freeze hooks were never run on Debian; now `configs/qemu-ga/fsfreeze-dispatch` (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"); the backup check now flags that as DEGRADED (exit 4) and covers nh3-pve-2, which it had never read. Offline shell (+ previous release's assets) served before auth; rerun `svos-front-inventory` after every SVOS release. Hosting bounds 1 MiB / 15 s body / 5 s headers. → `servers/nh3-svos/README.md`. - `[2026-10-05]` **fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE)** — run 1 aborted on its own guard (live rsync walked into an overlay `merged` view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (`-x`, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → `servers/fv-ml1/README.md` - `[2026-10-05]` **ana-docker: Docker CE repo bullseye → bookworm, 29.8.2 (Prime)** — with `live-restore: true` the engine upgrade kept containers up, but 6 that bind-mount `docker.sock` (crowdsec, traefik, dozzle, beszel-agent, dockge, gitea-runner) went stale and needed restarts. → `servers/ana-docker/README.md` - `[2026-10-05]` **RMM wildcard cert now auto-renews (Prime)** — `*.phasefinal.com` on pfi-tacticalrmm was certbot `manual` (could never renew; was 16 days from expiry). Now dns-cloudflare with a phasefinal.com-only token (vault `pfi-tacticalrmm/cloudflare-certbot-dns01-token`); reissued to 2027-01-03, dry-run OK. → `servers/pfi-tacticalrmm/README.md` diff --git a/scripts/backup-freshness-alert.sh b/scripts/backup-freshness-alert.sh index c394517..4b32784 100755 --- a/scripts/backup-freshness-alert.sh +++ b/scripts/backup-freshness-alert.sh @@ -32,6 +32,8 @@ # 1 backups STALE/DOWN, and the alert was delivered # 2 the ALERT PATH ITSELF FAILED (investigate this before the backups) # 3 ERRORED-JOBS — every body fresh, a job errored recently; alert delivered +# 4 DEGRADED — no job errored, but an OK task logged an ERROR (fs-freeze refused = +# crash-consistent image only); alert delivered # # 3 is deliberately not 1. A nightly job that errored is worth a look; a stale # backup body is worth a page. Reporting both as "STALE" over a fleet whose @@ -83,6 +85,9 @@ printf '%s\n' "$out" if [ "$rc" -eq 3 ]; then subject="🟡 Backup jobs errored ($(date '+%Y-%m-%d')) — all bodies fresh" lead="A vzdump job errored recently. Every backup body on the fleet is FRESH; nothing is stale." +elif [ "$rc" -eq 4 ]; then + subject="🟠 Backup DEGRADED ($(date '+%Y-%m-%d')) — an OK job logged an ERROR" + lead="Every backup body is FRESH and no job failed, but a backup logged an ERROR inside a task PVE calls OK. An fs-freeze failure there means that image is crash-consistent only, not application-consistent." else subject="🔴 Backup freshness ALERT ($(date '+%Y-%m-%d'))" lead="Automated daily backup-freshness check found STALE or DOWN backup layer(s) on the PFI fleet." diff --git a/scripts/check-backup-freshness.sh b/scripts/check-backup-freshness.sh index 7b6dafd..99c2977 100755 --- a/scripts/check-backup-freshness.sh +++ b/scripts/check-backup-freshness.sh @@ -8,6 +8,9 @@ # 0 all backups fresh # 1 STALE — a backup body is past the threshold, or an endpoint is down # 3 ERRORED-JOBS — every body is fresh, but a vzdump job errored recently +# 4 DEGRADED — no job errored, but a backup in the window logged an ERROR inside a +# task PVE still calls OK; above all an fs-freeze refusal, which means the image +# is crash-consistent, NOT application-consistent (see the layer below) # # 1 and 3 are deliberately different. Collapsing them prints "STALE" over a # fleet whose every backup is current, which is how an alarm earns being @@ -55,6 +58,8 @@ declare -A NS_NODE=( [ana-pve]=infra-ops@10.250.250.31 [esh-pve]=infra-ops@10.0.250.35 [nh3-pve]=infra-ops@10.100.250.60 + [nh3-pve-2]=infra-ops@10.100.250.62 # standalone (not clustered): own jobs.cfg, own namespace. Added 2026-10-05; + # until then its job results were never read at all ) declare -A NS_JOBS=() # ns → one "enabled|all|vmids|excludes" line per job declare -A NS_KNOWN=() # ns → 1 once coverage was successfully read @@ -80,9 +85,17 @@ PVE_NODES=( "esh-pve infra-ops@10.0.250.35" "esh-nas-pve infra-ops@10.0.50.55" "nh3-pve infra-ops@10.100.250.60" + "nh3-pve-2 infra-ops@10.100.250.62" ) JOB_WINDOW_H="${BACKUP_JOB_WINDOW_HOURS:-36}" jobfail=() +degraded=() +acked=() +# Acknowledged OK-but-ERROR tasks: a known, deliberate cause (a drill), printed every run in their own section and +# never counted. Key = the full UPID, value = why. Keep the paper trail; the entries age out with the window. +declare -A ACKED_TASKS=( + [UPID:nh3-pve-2:0004FFBA:00719F66:6AC4628F:vzdump:111:root@pam:]="2026-10-05 svos backup drill B: freeze refused ON PURPOSE by a temporary hook (servers/nh3-svos/README.md)" +) load_jobs() { # $1 = namespace local ns="$1" node="${NS_NODE[$1]:-}" raw @@ -257,6 +270,57 @@ for entry in "${PVE_NODES[@]}"; do done <<<"$bad" done +# --- Layer: did the jobs that SUCCEEDED actually quiesce? -------------------- +# +# ⚠ A task PVE records as OK can still have failed the thing that matters. Measured 2026-10-05 (nh3-pve-2, VM 111, +# svos backup drill): when the guest REFUSES fs-freeze, vzdump logs +# ERROR: unable to freeze guest fs - child process has failed to execute fsfreeze hook: exit status 1 +# then backs up UNFROZEN and ends "Backup job finished successfully", task status OK, exit 0. The image is +# crash-consistent, not application-consistent, and the job-success layer above cannot see it. So: read the task +# LOG of every OK vzdump task in the window, and report any ERROR line. Freeze/thaw errors are named as +# application-consistency failures; anything else is reported as an ERROR inside an OK task. +# The remote side stays a plain grep per file (see load_jobs on why embedded awk is avoided). A task log that +# cannot be read is an error, never a silent pass. +for entry in "${PVE_NODES[@]}"; do + set -- $entry; node_label="$1"; node_addr="$2" + idx=$($SSH "$node_addr" "sudo -n cat /var/log/pve/tasks/index" 2>/dev/null) + [ -n "$idx" ] || continue # already reported UNREADABLE by the layer above + okups=$(awk -v now="$now" -v win="$((JOB_WINDOW_H * 3600))" ' + { n = split($1, f, ":"); if (n < 7 || f[6] != "vzdump") next + if (now - strtonum("0x" f[5]) > win) next + st = $0; sub(/^[^ ]+ [^ ]+ /, "", st); if (st == "OK") print $1 }' <<<"$idx") + [ -n "$okups" ] || continue + script="" + while IFS= read -r u; do + [ -n "$u" ] || continue + sh=$(cut -d: -f5 <<<"$u"); d=${sh: -1} + script+="f='/var/log/pve/tasks/$d/$u'; if [ -r \"\$f\" ]; then grep -E '^(INFO: Starting Backup of VM|ERROR:)' \"\$f\" | sed 's|^|$u\t|'; else printf 'MISSING\t%s\n' '$u'; fi +" + done <<<"$okups" + hits=$($SSH "$node_addr" "sudo -n bash -s" <<<"$script" 2>/dev/null) || { errors+=("vzdump task LOGS on $node_label unreadable — freeze failures there are invisible"); continue; } + # attribute each ERROR to the guest whose "Starting Backup of VM" line came last in that task + while IFS=$'\t' read -r u line; do + [ -n "$u" ] || continue + if [ "$u" = "MISSING" ]; then errors+=("vzdump task log missing on $node_label: $line"); continue; fi + case "$line" in + "INFO: Starting Backup of VM "*) cur=${line#INFO: Starting Backup of VM }; cur=${cur%% *}; continue ;; + esac + gid=${cur:-$(cut -d: -f7 <<<"$u")} + ts=$(date -d "@$((16#$(cut -d: -f5 <<<"$u")))" '+%m-%d %H:%M' 2>/dev/null) + if grep -qiE 'freeze|thaw' <<<"$line"; then + msg="$node_label guest $gid ($ts): APPLICATION-CONSISTENCY FAILURE, image is crash-consistent only :: ${line#ERROR: }" + else + msg="$node_label guest $gid ($ts): ERROR inside a task PVE calls OK :: ${line#ERROR: }" + fi + if [ -n "${ACKED_TASKS[$u]:-}" ] && [ -z "${BACKUP_IGNORE_ACKS:-}" ]; then + acked+=("$msg [ack: ${ACKED_TASKS[$u]}]") + else + degraded+=("$msg") + fi + done <<<"$hits" + cur="" +done + # --- restic CONTENT assertion (operator ruling 2026-09-22) ------------------- # # The checks above read a snapshot directory's MTIME on the NAS. That proves a @@ -347,6 +411,14 @@ if [ "${#jobfail[@]}" -gt 0 ]; then echo; echo "BACKUP JOBS THAT ERRORED in the last ${JOB_WINDOW_H}h (${#jobfail[@]}):" printf ' ❌ %s\n' "${jobfail[@]}" fi +if [ "${#degraded[@]}" -gt 0 ]; then + echo; echo "DEGRADED — task OK, but the log has an ERROR (last ${JOB_WINDOW_H}h, ${#degraded[@]}):" + printf ' 🟠 %s\n' "${degraded[@]}" +fi +if [ "${#acked[@]}" -gt 0 ]; then + echo; echo "ACKNOWLEDGED (known cause, not counted; BACKUP_IGNORE_ACKS=1 counts them):" + printf ' ☑ %s\n' "${acked[@]}" +fi if [ "${#stale[@]}" -gt 0 ] || [ "${#errors[@]}" -gt 0 ]; then echo; echo "STALE / PROBLEMS (${#stale[@]}+${#errors[@]}):" [ "${#stale[@]}" -gt 0 ] && printf ' 🔴 %s\n' "${stale[@]}" @@ -375,5 +447,11 @@ if [ "${#jobfail[@]}" -gt 0 ]; then echo " Errors older than the window age out on their own." exit 3 fi +if [ "${#degraded[@]}" -gt 0 ]; then + echo; echo "RESULT: DEGRADED — every body is FRESH and no job errored, but ${#degraded[@]} backup(s) logged an" + echo " ERROR inside an OK task. An fs-freeze failure means that image is crash-consistent ONLY:" + echo " do not count it as application-consistent. Worth a look, not a page." + exit 4 +fi echo; echo "RESULT: all backups fresh" exit 0 diff --git a/servers/nh3-svos/README.md b/servers/nh3-svos/README.md index 47c717f..cbc9192 100644 --- a/servers/nh3-svos/README.md +++ b/servers/nh3-svos/README.md @@ -20,7 +20,7 @@ configuration, enrollment and acceptance. | Releases | `/opt/svos` root:svc-svos 2775, with `releases/` for svos-dev's layout. The git checkout is not a runtime dependency | | Docker | none, by design | | Backups | nh3-pve-2 vzdump job, nightly **21:30**, `all 1` → pbs-ana namespace `nh3-pve-2` (mirrored to pbs-nh3 at 06:30), snapshot mode with guest-agent fs-freeze | -| App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events` | +| App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events`. **Monitoring:** `scripts/check-backup-freshness.sh` now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict **DEGRADED, exit 4** (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script | | Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 | ## Build (reproducible) @@ -49,6 +49,8 @@ scripts/elway infra-ops@10.100.50.43 --playbook playbooks/nh3-svos-host.yaml `caddy hash-password --algorithm argon2id` on stdin (never `--plaintext`), rewrite the accounts file, `secret put`, then `systemctl reload caddy-svos`. - **Credential-free offline shell (svos-dev, 2026-10-05):** the service worker fetches its hash-pinned INVENTORY with `credentials:'omit'`, so exactly those files plus `/sw.js` are served BEFORE Basic auth (GET/HEAD, exact files, no fallback). Public build bytes only; `/`, `/status`, `/api/*`, the connector and everything else stay gated. The path list is **generated** into `/etc/caddy/svos-public-inventory` by **`/usr/local/sbin/svos-front-inventory`** (`configs/svos-front/`): it hash-checks every file against INVENTORY, validates, then reloads. ⚠ **Run it as root after every SVOS release**, or the next build's hashed asset names stay gated and worker install fails. `--check` changes nothing. +- **Previous release's assets across a rollout:** the same tool writes `/etc/caddy/svos-public-previous`: when `/opt/svos/current` has moved, the release named in the old inventory header becomes "previous", and its hashed `/assets/*` files that the new build lacks are served credential-free from **that release's own static dir** (hash-checked). One previous release at most; comments only when there is none (true at 2026-10-05, one release). Replica-tested: old asset 200 with the old INVENTORY hash, new asset with the new hash, POST to the old asset 401, unknown asset 401/404; tampered files and an unsafe path make the tool refuse. +- **Hosting bounds (svos-dev, per the web contract):** request bodies capped at exactly **1 MiB** on every route (413 over); **headers within 5 s, whole body within 15 s** (Caddy `read_header` / `read_body`); **no** write, response or idle deadline, so `/api/stream` (SSE) stays open. Live 2026-10-05 2014–2021 from fv-ml1: 1 MiB+1 → 413 on `/api/verbs` and `/verbs/pin`; exactly 1 MiB passes Caddy (kernel then answers); unfinished headers closed at 5.0 s; stalled body cut at 15.0 s; a signed `app-interface` ping returned pong before and DURING a stalled body; an SSE stream stayed open 41–45 s (client hung up) alongside all of it. ⚠ **A stalled body over HTTP/1.1 can come back as Caddy's empty `200 OK` (Content-Length 0)** instead of 504: replica 3/3 buffered, 2/3 unbuffered; live 2 of 3. Over h2, which browsers use, it is 502 (3/3 replica, 2/2 live). The app never gets a complete request either way; an empty 200 is not success. `handle_errors 499` and `request_buffers` do not change it (tested). - **Static routes (gated), per svoperatingsystem `web/README.md` Hosting contract:** `/` and `/status` → index.html; `/assets/*`, `/icons/*`, `/favicon.svg`, `/manifest.webmanifest`, `/index.html`, `/sw.js` → exact file, missing → 404; other HTML document requests (`Sec-Fetch-Dest: document` or `Accept: text/html`) → index.html; anything else → 404. Unversioned files get `Cache-Control: no-cache`. - **Acceptance matrix:** `configs/svos-front/acceptance-matrix.sh` (35 checks; usage in its header). 35/35 pass live at 1957 2026-10-05 from fv-ml1; against the previous config it reports 16 FAIL (positive control). - **Headers:** client copies of `Tailscale-User-*`, `X-Forwarded-*`, `Tailscale-Headers-Info`, `X-Real-IP`, `Forwarded` @@ -71,7 +73,6 @@ scripts/elway infra-ops@10.100.50.43 --playbook playbooks/nh3-svos-host.yaml headers, `X-Forwarded-Proto: https`, `Host: svos.phasefinal.com`, and path `/health` (the `/api` prefix stripped); - connector route with right credentials → 502 (auth passed; the connector was not running); cert verifies; LAN 10.100.50.43:443 → connection refused. -- **Open (svos-dev's numbers needed):** the hosting contract also asks the boundary to bound request body size and read duration for the connector; not set yet. ### Original requirements (svos-dev)