feat(svos-front,backups): SVOS hosting bounds and previous-release assets; backup check flags fs-freeze failures and covers nh3-pve-2
Front door: request bodies capped at 1 MiB, headers within 5 s, body within 15 s, no response deadlines (SSE stays open). Validated live: 413 over the cap, stalled body cut at 15 s, signed ping and an SSE stream unaffected. svos-front-inventory also serves the previous release's hashed assets credential-free from that release's static dir, hash-checked. Backup check: reads the logs of vzdump tasks PVE records as OK and reports ERROR lines; an fs-freeze/thaw error is an application-consistency failure, new verdict DEGRADED (exit 4), wired into the alert wrapper. nh3-pve-2 added to the node and namespace lists (its job results were never read). The svos drill task is acknowledged by UPID; BACKUP_IGNORE_ACKS=1 counts it (positive control: exit 4).
This commit is contained in:
@@ -15,6 +15,16 @@
|
||||
email lkraven@lkraven.com
|
||||
auto_https disable_redirects
|
||||
admin localhost:2019
|
||||
# Hosting bounds (svos-dev 2026-10-05, svoperatingsystem web/README.md Hosting contract): request headers within 5 s,
|
||||
# the whole request body within 15 s. NO write/response deadline: /api/stream (SSE) must stay open. Measured on a
|
||||
# replica 2026-10-05 before deploy: SSE 25 s and a 20 s slow response survive on h1 and h2; a stalled body is cut at
|
||||
# 15 s (h1 504, h2 502); unfinished headers are closed at 5 s.
|
||||
servers {
|
||||
timeouts {
|
||||
read_header 5s
|
||||
read_body 15s
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
https://svos.phasefinal.com:443 {
|
||||
@@ -33,6 +43,12 @@ https://svos.phasefinal.com:443 {
|
||||
request_header -Forwarded
|
||||
request_header -X-SVOS-Peer
|
||||
|
||||
# Request bodies capped at exactly 1 MiB (1,048,576 B, = the kernel dispatcher's own cap) on every route;
|
||||
# only /api/* and the connector read bodies at all. Over the cap -> 413. Responses are not limited.
|
||||
request_body {
|
||||
max_size 1MiB
|
||||
}
|
||||
|
||||
# Credential-free offline shell (svos-dev 2026-10-05): the service worker fetches its hash-pinned INVENTORY
|
||||
# with credentials:'omit', so exactly those files plus /sw.js are served BEFORE Basic auth, GET/HEAD only,
|
||||
# exact files (no fallback). Public build bytes only; no API or connector path. The path list is GENERATED
|
||||
@@ -47,6 +63,10 @@ https://svos.phasefinal.com:443 {
|
||||
root * /opt/svos/current/static
|
||||
file_server
|
||||
}
|
||||
# Previous release's hashed assets across a rollout (svos-dev 2026-10-05): open workers restore evicted inventory
|
||||
# credential-free. Generated by the same tool: a matcher + handle rooted in THAT release's static dir, or
|
||||
# comments only when there is no previous release.
|
||||
import /etc/caddy/svos-public-previous
|
||||
|
||||
basic_auth argon2id SVOS {
|
||||
import /etc/caddy/svos-basic-auth.accounts
|
||||
|
||||
@@ -1,17 +1,26 @@
|
||||
#!/usr/bin/env python3
|
||||
"""svos-front-inventory: regenerate the SVOS front door's credential-free shell paths from the live release.
|
||||
"""svos-front-inventory: regenerate the SVOS front door's credential-free shell paths from the live release(s).
|
||||
|
||||
eshpfi configs/svos-front/, installed at /usr/local/sbin/svos-front-inventory on nh3-svos. Run as root after every
|
||||
SVOS release (after /opt/svos/current moves), or the service worker's install breaks: its INVENTORY names hashed
|
||||
asset files, and only those exact paths (plus /sw.js) are served before Basic auth (svos-dev, 2026-10-05).
|
||||
|
||||
What it does:
|
||||
1. Reads INVENTORY from /opt/svos/current/static/sw.js (a JSON object of path -> sha256).
|
||||
2. Refuses unless every path is a plain absolute file path and every file exists with exactly that sha256.
|
||||
3. Writes /etc/caddy/svos-public-inventory (one Caddyfile `path` line, root:caddy 0640) atomically.
|
||||
4. Validates the whole Caddyfile as the caddy user with the unit's EnvironmentFile (no secret on any argv),
|
||||
restores the previous inventory file if validation fails, and otherwise reloads caddy-svos.
|
||||
--check: steps 1-2 only, plus a diff against the installed file; changes nothing.
|
||||
Two generated files, both imported by /etc/caddy/Caddyfile, both root:caddy 0640:
|
||||
/etc/caddy/svos-public-inventory `path ...` of the CURRENT release's INVENTORY + /sw.js
|
||||
/etc/caddy/svos-public-previous a matcher + handle serving the PREVIOUS release's hashed /assets/* files that
|
||||
the current build no longer has, from that release's own static dir. Already-
|
||||
open workers restore evicted inventory with credentials:'omit', so the old
|
||||
hashed assets must stay reachable credential-free across a rollout. Empty
|
||||
(comments only) when there is no previous release.
|
||||
Which release is "previous" is explicit state, not a guess: it is the release named in the header of the installed
|
||||
svos-public-inventory when current has moved on from it; otherwise the previous file's own header is kept. One
|
||||
previous release at most (the web contract's one current/previous build record).
|
||||
|
||||
Every path must be a plain absolute file path and every file must exist with exactly its INVENTORY sha256, in its
|
||||
own release, or nothing is written. After writing, the whole Caddyfile is validated as the caddy user with the
|
||||
unit's EnvironmentFile (no secret on any argv); on failure both previous files are restored and caddy is not
|
||||
reloaded. --check: verify and diff only, change nothing.
|
||||
Testing: SVOS_FRONT_TEST_ROOT=<dir> re-roots /opt/svos and /etc/caddy under <dir> and skips validate/reload/chown.
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
@@ -22,9 +31,13 @@ import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
STATIC = Path("/opt/svos/current/static")
|
||||
OUT = Path("/etc/caddy/svos-public-inventory")
|
||||
TEST_ROOT = os.environ.get("SVOS_FRONT_TEST_ROOT")
|
||||
BASE = Path(TEST_ROOT) if TEST_ROOT else Path("/")
|
||||
CURRENT = BASE / "opt/svos/current"
|
||||
OUT = BASE / "etc/caddy/svos-public-inventory"
|
||||
PREV_OUT = BASE / "etc/caddy/svos-public-previous"
|
||||
SAFE = re.compile(r"^/[A-Za-z0-9._-]+(/[A-Za-z0-9._-]+)*$")
|
||||
HEADER = re.compile(r"^# GENERATED by \S+ from (\S+)/static/sw\.js", re.M)
|
||||
VALIDATE = ["systemd-run", "--quiet", "--wait", "--pipe", "--collect", "-p", "User=caddy", "-p", "Group=caddy",
|
||||
"-p", "EnvironmentFile=/etc/caddy/env",
|
||||
"-p", "Environment=XDG_DATA_HOME=/var/lib/caddy XDG_CONFIG_HOME=/var/lib/caddy",
|
||||
@@ -35,56 +48,103 @@ def die(msg):
|
||||
sys.exit(f"svos-front-inventory: {msg}")
|
||||
|
||||
|
||||
def inventory():
|
||||
src = (STATIC / "sw.js").read_text(encoding="utf-8")
|
||||
def inventory(release):
|
||||
"""path -> sha256 from <release>/static/sw.js, unchecked."""
|
||||
src = (Path(release) / "static/sw.js").read_text(encoding="utf-8")
|
||||
m = re.search(r"^const INVENTORY = (\{.*?\});$", src, re.M)
|
||||
if not m:
|
||||
die("no `const INVENTORY = {...};` line in sw.js")
|
||||
die(f"no `const INVENTORY = {{...}};` line in {release}/static/sw.js")
|
||||
inv = json.loads(m.group(1))
|
||||
if not inv:
|
||||
die("INVENTORY is empty")
|
||||
for path, digest in inv.items():
|
||||
die(f"INVENTORY is empty in {release}")
|
||||
return inv
|
||||
|
||||
|
||||
def verify(release, inv, paths):
|
||||
static = Path(release) / "static"
|
||||
for path in paths:
|
||||
if not SAFE.match(path) or ".." in path.split("/"):
|
||||
die(f"unsafe inventory path {path!r}")
|
||||
f = STATIC / path.lstrip("/")
|
||||
die(f"unsafe inventory path {path!r} in {release}")
|
||||
f = static / path.lstrip("/")
|
||||
if not f.is_file() or f.is_symlink():
|
||||
die(f"{path} is not a regular file in {STATIC}")
|
||||
die(f"{path} is not a regular file in {static}")
|
||||
actual = hashlib.sha256(f.read_bytes()).hexdigest()
|
||||
if actual != digest:
|
||||
die(f"{path} sha256 {actual} != INVENTORY {digest}")
|
||||
return sorted(inv)
|
||||
if actual != inv[path]:
|
||||
die(f"{path} in {release}: sha256 {actual} != INVENTORY {inv[path]}")
|
||||
|
||||
|
||||
def render(paths):
|
||||
release = os.path.realpath(STATIC.parent)
|
||||
return (f"# GENERATED by /usr/local/sbin/svos-front-inventory from {release}/static/sw.js; do not edit.\n"
|
||||
f"# Exact credential-free shell paths (hash-checked), plus /sw.js.\n"
|
||||
f"path {' '.join(paths + ['/sw.js'])}\n")
|
||||
def header_release(text):
|
||||
m = HEADER.search(text)
|
||||
return m.group(1) if m else None
|
||||
|
||||
|
||||
def shown(release):
|
||||
"""The release path as Caddy and humans see it on the host (without the test re-root)."""
|
||||
return "/" + os.path.relpath(release, BASE) if TEST_ROOT else release
|
||||
|
||||
|
||||
def real(shown_path):
|
||||
return str(BASE / shown_path.lstrip("/")) if TEST_ROOT else shown_path
|
||||
|
||||
|
||||
def main():
|
||||
new = render(inventory())
|
||||
old = OUT.read_text(encoding="utf-8") if OUT.exists() else ""
|
||||
if "--check" in sys.argv[1:]:
|
||||
print("inventory ok;", "installed file is current" if old == new else "installed file DIFFERS:\n" + new)
|
||||
check = "--check" in sys.argv[1:]
|
||||
cur = os.path.realpath(CURRENT)
|
||||
cur_inv = inventory(cur)
|
||||
verify(cur, cur_inv, sorted(cur_inv))
|
||||
new_cur = (f"# GENERATED by /usr/local/sbin/svos-front-inventory from {shown(cur)}/static/sw.js; do not edit.\n"
|
||||
f"# Exact credential-free shell paths (hash-checked), plus /sw.js.\n"
|
||||
f"path {' '.join(sorted(cur_inv) + ['/sw.js'])}\n")
|
||||
|
||||
old_cur = OUT.read_text(encoding="utf-8") if OUT.exists() else ""
|
||||
old_prev = PREV_OUT.read_text(encoding="utf-8") if PREV_OUT.exists() else ""
|
||||
candidates = [header_release(old_cur), header_release(old_prev)]
|
||||
prev = next((r for r in candidates if r and real(r) != cur and os.path.isdir(real(r))), None)
|
||||
|
||||
if prev:
|
||||
prev_inv = inventory(real(prev))
|
||||
keep = sorted(p for p in prev_inv if p.startswith("/assets/") and p not in cur_inv)
|
||||
verify(real(prev), prev_inv, keep)
|
||||
if prev and keep:
|
||||
new_prev = (f"# GENERATED by /usr/local/sbin/svos-front-inventory from {prev}/static/sw.js; do not edit.\n"
|
||||
f"# Previous release's hashed assets the current build no longer has (hash-checked), served\n"
|
||||
f"# credential-free from that release's own static dir.\n"
|
||||
f"@public_previous {{\n\tmethod GET HEAD\n\tpath {' '.join(keep)}\n}}\n"
|
||||
f"handle @public_previous {{\n\troot * {prev}/static\n\tfile_server\n}}\n")
|
||||
else:
|
||||
new_prev = ("# GENERATED by /usr/local/sbin/svos-front-inventory; do not edit.\n"
|
||||
"# No previous release with assets to keep: nothing is served from here.\n")
|
||||
|
||||
if check:
|
||||
for name, old, new in (("current", old_cur, new_cur), ("previous", old_prev, new_prev)):
|
||||
print(f"{name}: " + ("installed file is current" if old == new else "installed file DIFFERS:\n" + new))
|
||||
return
|
||||
if old == new:
|
||||
if old_cur == new_cur and old_prev == new_prev:
|
||||
print("unchanged; nothing to do")
|
||||
return
|
||||
tmp = OUT.with_name(OUT.name + ".new")
|
||||
tmp.write_text(new, encoding="utf-8")
|
||||
shutil.chown(tmp, "root", "caddy")
|
||||
os.chmod(tmp, 0o640)
|
||||
backup = OUT.with_name(OUT.name + ".prev")
|
||||
if OUT.exists():
|
||||
shutil.copy2(OUT, backup)
|
||||
os.replace(tmp, OUT)
|
||||
|
||||
saved = {}
|
||||
for path, text in ((OUT, new_cur), (PREV_OUT, new_prev)):
|
||||
if path.exists():
|
||||
saved[path] = path.read_text(encoding="utf-8")
|
||||
tmp = path.with_name(path.name + ".new")
|
||||
tmp.write_text(text, encoding="utf-8")
|
||||
if not TEST_ROOT:
|
||||
shutil.chown(tmp, "root", "caddy")
|
||||
os.chmod(tmp, 0o640)
|
||||
os.replace(tmp, path)
|
||||
if TEST_ROOT:
|
||||
print("TEST: files written, validate/reload skipped")
|
||||
return
|
||||
if subprocess.run(VALIDATE).returncode != 0:
|
||||
if backup.exists():
|
||||
os.replace(backup, OUT)
|
||||
die("caddy validate FAILED; previous inventory restored, caddy not reloaded")
|
||||
for path in (OUT, PREV_OUT):
|
||||
if path in saved:
|
||||
path.write_text(saved[path], encoding="utf-8")
|
||||
else:
|
||||
path.unlink()
|
||||
die("caddy validate FAILED; previous files restored, caddy not reloaded")
|
||||
subprocess.run(["systemctl", "reload", "caddy-svos"], check=True)
|
||||
print("inventory updated and caddy-svos reloaded:\n" + new)
|
||||
print("updated and caddy-svos reloaded:\n" + new_cur + new_prev)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -301,7 +301,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-10-05]` **nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only `https://svos.phasefinal.com` with HTTP Basic auth, login `Vuong` (Prime, direct)** — WhoIs gate retired (all mesh nodes are login `pfi`; Prime declined a mesh restructure). Password vaulted `nh3-svos/basic-auth-vuong`. qemu-ga freeze hooks were never run on Debian; now `configs/qemu-ga/fsfreeze-dispatch` (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"). Offline shell assets served before auth; rerun `svos-front-inventory` after every SVOS release. → `servers/nh3-svos/README.md`.
|
||||
- `[2026-10-05]` **nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only `https://svos.phasefinal.com` with HTTP Basic auth, login `Vuong` (Prime, direct)** — WhoIs gate retired (all mesh nodes are login `pfi`; Prime declined a mesh restructure). Password vaulted `nh3-svos/basic-auth-vuong`. qemu-ga freeze hooks were never run on Debian; now `configs/qemu-ga/fsfreeze-dispatch` (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"); the backup check now flags that as DEGRADED (exit 4) and covers nh3-pve-2, which it had never read. Offline shell (+ previous release's assets) served before auth; rerun `svos-front-inventory` after every SVOS release. Hosting bounds 1 MiB / 15 s body / 5 s headers. → `servers/nh3-svos/README.md`.
|
||||
- `[2026-10-05]` **fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE)** — run 1 aborted on its own guard (live rsync walked into an overlay `merged` view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (`-x`, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → `servers/fv-ml1/README.md`
|
||||
- `[2026-10-05]` **ana-docker: Docker CE repo bullseye → bookworm, 29.8.2 (Prime)** — with `live-restore: true` the engine upgrade kept containers up, but 6 that bind-mount `docker.sock` (crowdsec, traefik, dozzle, beszel-agent, dockge, gitea-runner) went stale and needed restarts. → `servers/ana-docker/README.md`
|
||||
- `[2026-10-05]` **RMM wildcard cert now auto-renews (Prime)** — `*.phasefinal.com` on pfi-tacticalrmm was certbot `manual` (could never renew; was 16 days from expiry). Now dns-cloudflare with a phasefinal.com-only token (vault `pfi-tacticalrmm/cloudflare-certbot-dns01-token`); reissued to 2027-01-03, dry-run OK. → `servers/pfi-tacticalrmm/README.md`
|
||||
|
||||
@@ -32,6 +32,8 @@
|
||||
# 1 backups STALE/DOWN, and the alert was delivered
|
||||
# 2 the ALERT PATH ITSELF FAILED (investigate this before the backups)
|
||||
# 3 ERRORED-JOBS — every body fresh, a job errored recently; alert delivered
|
||||
# 4 DEGRADED — no job errored, but an OK task logged an ERROR (fs-freeze refused =
|
||||
# crash-consistent image only); alert delivered
|
||||
#
|
||||
# 3 is deliberately not 1. A nightly job that errored is worth a look; a stale
|
||||
# backup body is worth a page. Reporting both as "STALE" over a fleet whose
|
||||
@@ -83,6 +85,9 @@ printf '%s\n' "$out"
|
||||
if [ "$rc" -eq 3 ]; then
|
||||
subject="🟡 Backup jobs errored ($(date '+%Y-%m-%d')) — all bodies fresh"
|
||||
lead="A vzdump job errored recently. Every backup body on the fleet is FRESH; nothing is stale."
|
||||
elif [ "$rc" -eq 4 ]; then
|
||||
subject="🟠 Backup DEGRADED ($(date '+%Y-%m-%d')) — an OK job logged an ERROR"
|
||||
lead="Every backup body is FRESH and no job failed, but a backup logged an ERROR inside a task PVE calls OK. An fs-freeze failure there means that image is crash-consistent only, not application-consistent."
|
||||
else
|
||||
subject="🔴 Backup freshness ALERT ($(date '+%Y-%m-%d'))"
|
||||
lead="Automated daily backup-freshness check found STALE or DOWN backup layer(s) on the PFI fleet."
|
||||
|
||||
@@ -8,6 +8,9 @@
|
||||
# 0 all backups fresh
|
||||
# 1 STALE — a backup body is past the threshold, or an endpoint is down
|
||||
# 3 ERRORED-JOBS — every body is fresh, but a vzdump job errored recently
|
||||
# 4 DEGRADED — no job errored, but a backup in the window logged an ERROR inside a
|
||||
# task PVE still calls OK; above all an fs-freeze refusal, which means the image
|
||||
# is crash-consistent, NOT application-consistent (see the layer below)
|
||||
#
|
||||
# 1 and 3 are deliberately different. Collapsing them prints "STALE" over a
|
||||
# fleet whose every backup is current, which is how an alarm earns being
|
||||
@@ -55,6 +58,8 @@ declare -A NS_NODE=(
|
||||
[ana-pve]=infra-ops@10.250.250.31
|
||||
[esh-pve]=infra-ops@10.0.250.35
|
||||
[nh3-pve]=infra-ops@10.100.250.60
|
||||
[nh3-pve-2]=infra-ops@10.100.250.62 # standalone (not clustered): own jobs.cfg, own namespace. Added 2026-10-05;
|
||||
# until then its job results were never read at all
|
||||
)
|
||||
declare -A NS_JOBS=() # ns → one "enabled|all|vmids|excludes" line per job
|
||||
declare -A NS_KNOWN=() # ns → 1 once coverage was successfully read
|
||||
@@ -80,9 +85,17 @@ PVE_NODES=(
|
||||
"esh-pve infra-ops@10.0.250.35"
|
||||
"esh-nas-pve infra-ops@10.0.50.55"
|
||||
"nh3-pve infra-ops@10.100.250.60"
|
||||
"nh3-pve-2 infra-ops@10.100.250.62"
|
||||
)
|
||||
JOB_WINDOW_H="${BACKUP_JOB_WINDOW_HOURS:-36}"
|
||||
jobfail=()
|
||||
degraded=()
|
||||
acked=()
|
||||
# Acknowledged OK-but-ERROR tasks: a known, deliberate cause (a drill), printed every run in their own section and
|
||||
# never counted. Key = the full UPID, value = why. Keep the paper trail; the entries age out with the window.
|
||||
declare -A ACKED_TASKS=(
|
||||
[UPID:nh3-pve-2:0004FFBA:00719F66:6AC4628F:vzdump:111:root@pam:]="2026-10-05 svos backup drill B: freeze refused ON PURPOSE by a temporary hook (servers/nh3-svos/README.md)"
|
||||
)
|
||||
|
||||
load_jobs() { # $1 = namespace
|
||||
local ns="$1" node="${NS_NODE[$1]:-}" raw
|
||||
@@ -257,6 +270,57 @@ for entry in "${PVE_NODES[@]}"; do
|
||||
done <<<"$bad"
|
||||
done
|
||||
|
||||
# --- Layer: did the jobs that SUCCEEDED actually quiesce? --------------------
|
||||
#
|
||||
# ⚠ A task PVE records as OK can still have failed the thing that matters. Measured 2026-10-05 (nh3-pve-2, VM 111,
|
||||
# svos backup drill): when the guest REFUSES fs-freeze, vzdump logs
|
||||
# ERROR: unable to freeze guest fs - child process has failed to execute fsfreeze hook: exit status 1
|
||||
# then backs up UNFROZEN and ends "Backup job finished successfully", task status OK, exit 0. The image is
|
||||
# crash-consistent, not application-consistent, and the job-success layer above cannot see it. So: read the task
|
||||
# LOG of every OK vzdump task in the window, and report any ERROR line. Freeze/thaw errors are named as
|
||||
# application-consistency failures; anything else is reported as an ERROR inside an OK task.
|
||||
# The remote side stays a plain grep per file (see load_jobs on why embedded awk is avoided). A task log that
|
||||
# cannot be read is an error, never a silent pass.
|
||||
for entry in "${PVE_NODES[@]}"; do
|
||||
set -- $entry; node_label="$1"; node_addr="$2"
|
||||
idx=$($SSH "$node_addr" "sudo -n cat /var/log/pve/tasks/index" 2>/dev/null)
|
||||
[ -n "$idx" ] || continue # already reported UNREADABLE by the layer above
|
||||
okups=$(awk -v now="$now" -v win="$((JOB_WINDOW_H * 3600))" '
|
||||
{ n = split($1, f, ":"); if (n < 7 || f[6] != "vzdump") next
|
||||
if (now - strtonum("0x" f[5]) > win) next
|
||||
st = $0; sub(/^[^ ]+ [^ ]+ /, "", st); if (st == "OK") print $1 }' <<<"$idx")
|
||||
[ -n "$okups" ] || continue
|
||||
script=""
|
||||
while IFS= read -r u; do
|
||||
[ -n "$u" ] || continue
|
||||
sh=$(cut -d: -f5 <<<"$u"); d=${sh: -1}
|
||||
script+="f='/var/log/pve/tasks/$d/$u'; if [ -r \"\$f\" ]; then grep -E '^(INFO: Starting Backup of VM|ERROR:)' \"\$f\" | sed 's|^|$u\t|'; else printf 'MISSING\t%s\n' '$u'; fi
|
||||
"
|
||||
done <<<"$okups"
|
||||
hits=$($SSH "$node_addr" "sudo -n bash -s" <<<"$script" 2>/dev/null) || { errors+=("vzdump task LOGS on $node_label unreadable — freeze failures there are invisible"); continue; }
|
||||
# attribute each ERROR to the guest whose "Starting Backup of VM" line came last in that task
|
||||
while IFS=$'\t' read -r u line; do
|
||||
[ -n "$u" ] || continue
|
||||
if [ "$u" = "MISSING" ]; then errors+=("vzdump task log missing on $node_label: $line"); continue; fi
|
||||
case "$line" in
|
||||
"INFO: Starting Backup of VM "*) cur=${line#INFO: Starting Backup of VM }; cur=${cur%% *}; continue ;;
|
||||
esac
|
||||
gid=${cur:-$(cut -d: -f7 <<<"$u")}
|
||||
ts=$(date -d "@$((16#$(cut -d: -f5 <<<"$u")))" '+%m-%d %H:%M' 2>/dev/null)
|
||||
if grep -qiE 'freeze|thaw' <<<"$line"; then
|
||||
msg="$node_label guest $gid ($ts): APPLICATION-CONSISTENCY FAILURE, image is crash-consistent only :: ${line#ERROR: }"
|
||||
else
|
||||
msg="$node_label guest $gid ($ts): ERROR inside a task PVE calls OK :: ${line#ERROR: }"
|
||||
fi
|
||||
if [ -n "${ACKED_TASKS[$u]:-}" ] && [ -z "${BACKUP_IGNORE_ACKS:-}" ]; then
|
||||
acked+=("$msg [ack: ${ACKED_TASKS[$u]}]")
|
||||
else
|
||||
degraded+=("$msg")
|
||||
fi
|
||||
done <<<"$hits"
|
||||
cur=""
|
||||
done
|
||||
|
||||
# --- restic CONTENT assertion (operator ruling 2026-09-22) -------------------
|
||||
#
|
||||
# The checks above read a snapshot directory's MTIME on the NAS. That proves a
|
||||
@@ -347,6 +411,14 @@ if [ "${#jobfail[@]}" -gt 0 ]; then
|
||||
echo; echo "BACKUP JOBS THAT ERRORED in the last ${JOB_WINDOW_H}h (${#jobfail[@]}):"
|
||||
printf ' ❌ %s\n' "${jobfail[@]}"
|
||||
fi
|
||||
if [ "${#degraded[@]}" -gt 0 ]; then
|
||||
echo; echo "DEGRADED — task OK, but the log has an ERROR (last ${JOB_WINDOW_H}h, ${#degraded[@]}):"
|
||||
printf ' 🟠 %s\n' "${degraded[@]}"
|
||||
fi
|
||||
if [ "${#acked[@]}" -gt 0 ]; then
|
||||
echo; echo "ACKNOWLEDGED (known cause, not counted; BACKUP_IGNORE_ACKS=1 counts them):"
|
||||
printf ' ☑ %s\n' "${acked[@]}"
|
||||
fi
|
||||
if [ "${#stale[@]}" -gt 0 ] || [ "${#errors[@]}" -gt 0 ]; then
|
||||
echo; echo "STALE / PROBLEMS (${#stale[@]}+${#errors[@]}):"
|
||||
[ "${#stale[@]}" -gt 0 ] && printf ' 🔴 %s\n' "${stale[@]}"
|
||||
@@ -375,5 +447,11 @@ if [ "${#jobfail[@]}" -gt 0 ]; then
|
||||
echo " Errors older than the window age out on their own."
|
||||
exit 3
|
||||
fi
|
||||
if [ "${#degraded[@]}" -gt 0 ]; then
|
||||
echo; echo "RESULT: DEGRADED — every body is FRESH and no job errored, but ${#degraded[@]} backup(s) logged an"
|
||||
echo " ERROR inside an OK task. An fs-freeze failure means that image is crash-consistent ONLY:"
|
||||
echo " do not count it as application-consistent. Worth a look, not a page."
|
||||
exit 4
|
||||
fi
|
||||
echo; echo "RESULT: all backups fresh"
|
||||
exit 0
|
||||
|
||||
@@ -20,7 +20,7 @@ configuration, enrollment and acceptance.
|
||||
| Releases | `/opt/svos` root:svc-svos 2775, with `releases/` for svos-dev's layout. The git checkout is not a runtime dependency |
|
||||
| Docker | none, by design |
|
||||
| Backups | nh3-pve-2 vzdump job, nightly **21:30**, `all 1` → pbs-ana namespace `nh3-pve-2` (mirrored to pbs-nh3 at 06:30), snapshot mode with guest-agent fs-freeze |
|
||||
| App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events` |
|
||||
| App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events`. **Monitoring:** `scripts/check-backup-freshness.sh` now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict **DEGRADED, exit 4** (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script |
|
||||
| Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 |
|
||||
|
||||
## Build (reproducible)
|
||||
@@ -49,6 +49,8 @@ scripts/elway infra-ops@10.100.50.43 --playbook playbooks/nh3-svos-host.yaml
|
||||
`caddy hash-password --algorithm argon2id` on stdin (never `--plaintext`), rewrite the accounts file, `secret put`,
|
||||
then `systemctl reload caddy-svos`.
|
||||
- **Credential-free offline shell (svos-dev, 2026-10-05):** the service worker fetches its hash-pinned INVENTORY with `credentials:'omit'`, so exactly those files plus `/sw.js` are served BEFORE Basic auth (GET/HEAD, exact files, no fallback). Public build bytes only; `/`, `/status`, `/api/*`, the connector and everything else stay gated. The path list is **generated** into `/etc/caddy/svos-public-inventory` by **`/usr/local/sbin/svos-front-inventory`** (`configs/svos-front/`): it hash-checks every file against INVENTORY, validates, then reloads. ⚠ **Run it as root after every SVOS release**, or the next build's hashed asset names stay gated and worker install fails. `--check` changes nothing.
|
||||
- **Previous release's assets across a rollout:** the same tool writes `/etc/caddy/svos-public-previous`: when `/opt/svos/current` has moved, the release named in the old inventory header becomes "previous", and its hashed `/assets/*` files that the new build lacks are served credential-free from **that release's own static dir** (hash-checked). One previous release at most; comments only when there is none (true at 2026-10-05, one release). Replica-tested: old asset 200 with the old INVENTORY hash, new asset with the new hash, POST to the old asset 401, unknown asset 401/404; tampered files and an unsafe path make the tool refuse.
|
||||
- **Hosting bounds (svos-dev, per the web contract):** request bodies capped at exactly **1 MiB** on every route (413 over); **headers within 5 s, whole body within 15 s** (Caddy `read_header` / `read_body`); **no** write, response or idle deadline, so `/api/stream` (SSE) stays open. Live 2026-10-05 2014–2021 from fv-ml1: 1 MiB+1 → 413 on `/api/verbs` and `/verbs/pin`; exactly 1 MiB passes Caddy (kernel then answers); unfinished headers closed at 5.0 s; stalled body cut at 15.0 s; a signed `app-interface` ping returned pong before and DURING a stalled body; an SSE stream stayed open 41–45 s (client hung up) alongside all of it. ⚠ **A stalled body over HTTP/1.1 can come back as Caddy's empty `200 OK` (Content-Length 0)** instead of 504: replica 3/3 buffered, 2/3 unbuffered; live 2 of 3. Over h2, which browsers use, it is 502 (3/3 replica, 2/2 live). The app never gets a complete request either way; an empty 200 is not success. `handle_errors 499` and `request_buffers` do not change it (tested).
|
||||
- **Static routes (gated), per svoperatingsystem `web/README.md` Hosting contract:** `/` and `/status` → index.html; `/assets/*`, `/icons/*`, `/favicon.svg`, `/manifest.webmanifest`, `/index.html`, `/sw.js` → exact file, missing → 404; other HTML document requests (`Sec-Fetch-Dest: document` or `Accept: text/html`) → index.html; anything else → 404. Unversioned files get `Cache-Control: no-cache`.
|
||||
- **Acceptance matrix:** `configs/svos-front/acceptance-matrix.sh` (35 checks; usage in its header). 35/35 pass live at 1957 2026-10-05 from fv-ml1; against the previous config it reports 16 FAIL (positive control).
|
||||
- **Headers:** client copies of `Tailscale-User-*`, `X-Forwarded-*`, `Tailscale-Headers-Info`, `X-Real-IP`, `Forwarded`
|
||||
@@ -71,7 +73,6 @@ scripts/elway infra-ops@10.100.50.43 --playbook playbooks/nh3-svos-host.yaml
|
||||
headers, `X-Forwarded-Proto: https`, `Host: svos.phasefinal.com`, and path `/health` (the `/api` prefix stripped);
|
||||
- connector route with right credentials → 502 (auth passed; the connector was not running); cert verifies; LAN
|
||||
10.100.50.43:443 → connection refused.
|
||||
- **Open (svos-dev's numbers needed):** the hosting contract also asks the boundary to bound request body size and read duration for the connector; not set yet.
|
||||
|
||||
### Original requirements (svos-dev)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user