feat(svos-front,backups): SVOS hosting bounds and previous-release assets; backup check flags fs-freeze failures and covers nh3-pve-2
Front door: request bodies capped at 1 MiB, headers within 5 s, body within 15 s, no response deadlines (SSE stays open). Validated live: 413 over the cap, stalled body cut at 15 s, signed ping and an SSE stream unaffected. svos-front-inventory also serves the previous release's hashed assets credential-free from that release's static dir, hash-checked. Backup check: reads the logs of vzdump tasks PVE records as OK and reports ERROR lines; an fs-freeze/thaw error is an application-consistency failure, new verdict DEGRADED (exit 4), wired into the alert wrapper. nh3-pve-2 added to the node and namespace lists (its job results were never read). The svos drill task is acknowledged by UPID; BACKUP_IGNORE_ACKS=1 counts it (positive control: exit 4).
This commit is contained in:
@@ -15,6 +15,16 @@
|
||||
email lkraven@lkraven.com
|
||||
auto_https disable_redirects
|
||||
admin localhost:2019
|
||||
# Hosting bounds (svos-dev 2026-10-05, svoperatingsystem web/README.md Hosting contract): request headers within 5 s,
|
||||
# the whole request body within 15 s. NO write/response deadline: /api/stream (SSE) must stay open. Measured on a
|
||||
# replica 2026-10-05 before deploy: SSE 25 s and a 20 s slow response survive on h1 and h2; a stalled body is cut at
|
||||
# 15 s (h1 504, h2 502); unfinished headers are closed at 5 s.
|
||||
servers {
|
||||
timeouts {
|
||||
read_header 5s
|
||||
read_body 15s
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
https://svos.phasefinal.com:443 {
|
||||
@@ -33,6 +43,12 @@ https://svos.phasefinal.com:443 {
|
||||
request_header -Forwarded
|
||||
request_header -X-SVOS-Peer
|
||||
|
||||
# Request bodies capped at exactly 1 MiB (1,048,576 B, = the kernel dispatcher's own cap) on every route;
|
||||
# only /api/* and the connector read bodies at all. Over the cap -> 413. Responses are not limited.
|
||||
request_body {
|
||||
max_size 1MiB
|
||||
}
|
||||
|
||||
# Credential-free offline shell (svos-dev 2026-10-05): the service worker fetches its hash-pinned INVENTORY
|
||||
# with credentials:'omit', so exactly those files plus /sw.js are served BEFORE Basic auth, GET/HEAD only,
|
||||
# exact files (no fallback). Public build bytes only; no API or connector path. The path list is GENERATED
|
||||
@@ -47,6 +63,10 @@ https://svos.phasefinal.com:443 {
|
||||
root * /opt/svos/current/static
|
||||
file_server
|
||||
}
|
||||
# Previous release's hashed assets across a rollout (svos-dev 2026-10-05): open workers restore evicted inventory
|
||||
# credential-free. Generated by the same tool: a matcher + handle rooted in THAT release's static dir, or
|
||||
# comments only when there is no previous release.
|
||||
import /etc/caddy/svos-public-previous
|
||||
|
||||
basic_auth argon2id SVOS {
|
||||
import /etc/caddy/svos-basic-auth.accounts
|
||||
|
||||
@@ -1,17 +1,26 @@
|
||||
#!/usr/bin/env python3
|
||||
"""svos-front-inventory: regenerate the SVOS front door's credential-free shell paths from the live release.
|
||||
"""svos-front-inventory: regenerate the SVOS front door's credential-free shell paths from the live release(s).
|
||||
|
||||
eshpfi configs/svos-front/, installed at /usr/local/sbin/svos-front-inventory on nh3-svos. Run as root after every
|
||||
SVOS release (after /opt/svos/current moves), or the service worker's install breaks: its INVENTORY names hashed
|
||||
asset files, and only those exact paths (plus /sw.js) are served before Basic auth (svos-dev, 2026-10-05).
|
||||
|
||||
What it does:
|
||||
1. Reads INVENTORY from /opt/svos/current/static/sw.js (a JSON object of path -> sha256).
|
||||
2. Refuses unless every path is a plain absolute file path and every file exists with exactly that sha256.
|
||||
3. Writes /etc/caddy/svos-public-inventory (one Caddyfile `path` line, root:caddy 0640) atomically.
|
||||
4. Validates the whole Caddyfile as the caddy user with the unit's EnvironmentFile (no secret on any argv),
|
||||
restores the previous inventory file if validation fails, and otherwise reloads caddy-svos.
|
||||
--check: steps 1-2 only, plus a diff against the installed file; changes nothing.
|
||||
Two generated files, both imported by /etc/caddy/Caddyfile, both root:caddy 0640:
|
||||
/etc/caddy/svos-public-inventory `path ...` of the CURRENT release's INVENTORY + /sw.js
|
||||
/etc/caddy/svos-public-previous a matcher + handle serving the PREVIOUS release's hashed /assets/* files that
|
||||
the current build no longer has, from that release's own static dir. Already-
|
||||
open workers restore evicted inventory with credentials:'omit', so the old
|
||||
hashed assets must stay reachable credential-free across a rollout. Empty
|
||||
(comments only) when there is no previous release.
|
||||
Which release is "previous" is explicit state, not a guess: it is the release named in the header of the installed
|
||||
svos-public-inventory when current has moved on from it; otherwise the previous file's own header is kept. One
|
||||
previous release at most (the web contract's one current/previous build record).
|
||||
|
||||
Every path must be a plain absolute file path and every file must exist with exactly its INVENTORY sha256, in its
|
||||
own release, or nothing is written. After writing, the whole Caddyfile is validated as the caddy user with the
|
||||
unit's EnvironmentFile (no secret on any argv); on failure both previous files are restored and caddy is not
|
||||
reloaded. --check: verify and diff only, change nothing.
|
||||
Testing: SVOS_FRONT_TEST_ROOT=<dir> re-roots /opt/svos and /etc/caddy under <dir> and skips validate/reload/chown.
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
@@ -22,9 +31,13 @@ import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
STATIC = Path("/opt/svos/current/static")
|
||||
OUT = Path("/etc/caddy/svos-public-inventory")
|
||||
TEST_ROOT = os.environ.get("SVOS_FRONT_TEST_ROOT")
|
||||
BASE = Path(TEST_ROOT) if TEST_ROOT else Path("/")
|
||||
CURRENT = BASE / "opt/svos/current"
|
||||
OUT = BASE / "etc/caddy/svos-public-inventory"
|
||||
PREV_OUT = BASE / "etc/caddy/svos-public-previous"
|
||||
SAFE = re.compile(r"^/[A-Za-z0-9._-]+(/[A-Za-z0-9._-]+)*$")
|
||||
HEADER = re.compile(r"^# GENERATED by \S+ from (\S+)/static/sw\.js", re.M)
|
||||
VALIDATE = ["systemd-run", "--quiet", "--wait", "--pipe", "--collect", "-p", "User=caddy", "-p", "Group=caddy",
|
||||
"-p", "EnvironmentFile=/etc/caddy/env",
|
||||
"-p", "Environment=XDG_DATA_HOME=/var/lib/caddy XDG_CONFIG_HOME=/var/lib/caddy",
|
||||
@@ -35,56 +48,103 @@ def die(msg):
|
||||
sys.exit(f"svos-front-inventory: {msg}")
|
||||
|
||||
|
||||
def inventory():
|
||||
src = (STATIC / "sw.js").read_text(encoding="utf-8")
|
||||
def inventory(release):
|
||||
"""path -> sha256 from <release>/static/sw.js, unchecked."""
|
||||
src = (Path(release) / "static/sw.js").read_text(encoding="utf-8")
|
||||
m = re.search(r"^const INVENTORY = (\{.*?\});$", src, re.M)
|
||||
if not m:
|
||||
die("no `const INVENTORY = {...};` line in sw.js")
|
||||
die(f"no `const INVENTORY = {{...}};` line in {release}/static/sw.js")
|
||||
inv = json.loads(m.group(1))
|
||||
if not inv:
|
||||
die("INVENTORY is empty")
|
||||
for path, digest in inv.items():
|
||||
die(f"INVENTORY is empty in {release}")
|
||||
return inv
|
||||
|
||||
|
||||
def verify(release, inv, paths):
|
||||
static = Path(release) / "static"
|
||||
for path in paths:
|
||||
if not SAFE.match(path) or ".." in path.split("/"):
|
||||
die(f"unsafe inventory path {path!r}")
|
||||
f = STATIC / path.lstrip("/")
|
||||
die(f"unsafe inventory path {path!r} in {release}")
|
||||
f = static / path.lstrip("/")
|
||||
if not f.is_file() or f.is_symlink():
|
||||
die(f"{path} is not a regular file in {STATIC}")
|
||||
die(f"{path} is not a regular file in {static}")
|
||||
actual = hashlib.sha256(f.read_bytes()).hexdigest()
|
||||
if actual != digest:
|
||||
die(f"{path} sha256 {actual} != INVENTORY {digest}")
|
||||
return sorted(inv)
|
||||
if actual != inv[path]:
|
||||
die(f"{path} in {release}: sha256 {actual} != INVENTORY {inv[path]}")
|
||||
|
||||
|
||||
def render(paths):
|
||||
release = os.path.realpath(STATIC.parent)
|
||||
return (f"# GENERATED by /usr/local/sbin/svos-front-inventory from {release}/static/sw.js; do not edit.\n"
|
||||
f"# Exact credential-free shell paths (hash-checked), plus /sw.js.\n"
|
||||
f"path {' '.join(paths + ['/sw.js'])}\n")
|
||||
def header_release(text):
|
||||
m = HEADER.search(text)
|
||||
return m.group(1) if m else None
|
||||
|
||||
|
||||
def shown(release):
|
||||
"""The release path as Caddy and humans see it on the host (without the test re-root)."""
|
||||
return "/" + os.path.relpath(release, BASE) if TEST_ROOT else release
|
||||
|
||||
|
||||
def real(shown_path):
|
||||
return str(BASE / shown_path.lstrip("/")) if TEST_ROOT else shown_path
|
||||
|
||||
|
||||
def main():
|
||||
new = render(inventory())
|
||||
old = OUT.read_text(encoding="utf-8") if OUT.exists() else ""
|
||||
if "--check" in sys.argv[1:]:
|
||||
print("inventory ok;", "installed file is current" if old == new else "installed file DIFFERS:\n" + new)
|
||||
check = "--check" in sys.argv[1:]
|
||||
cur = os.path.realpath(CURRENT)
|
||||
cur_inv = inventory(cur)
|
||||
verify(cur, cur_inv, sorted(cur_inv))
|
||||
new_cur = (f"# GENERATED by /usr/local/sbin/svos-front-inventory from {shown(cur)}/static/sw.js; do not edit.\n"
|
||||
f"# Exact credential-free shell paths (hash-checked), plus /sw.js.\n"
|
||||
f"path {' '.join(sorted(cur_inv) + ['/sw.js'])}\n")
|
||||
|
||||
old_cur = OUT.read_text(encoding="utf-8") if OUT.exists() else ""
|
||||
old_prev = PREV_OUT.read_text(encoding="utf-8") if PREV_OUT.exists() else ""
|
||||
candidates = [header_release(old_cur), header_release(old_prev)]
|
||||
prev = next((r for r in candidates if r and real(r) != cur and os.path.isdir(real(r))), None)
|
||||
|
||||
if prev:
|
||||
prev_inv = inventory(real(prev))
|
||||
keep = sorted(p for p in prev_inv if p.startswith("/assets/") and p not in cur_inv)
|
||||
verify(real(prev), prev_inv, keep)
|
||||
if prev and keep:
|
||||
new_prev = (f"# GENERATED by /usr/local/sbin/svos-front-inventory from {prev}/static/sw.js; do not edit.\n"
|
||||
f"# Previous release's hashed assets the current build no longer has (hash-checked), served\n"
|
||||
f"# credential-free from that release's own static dir.\n"
|
||||
f"@public_previous {{\n\tmethod GET HEAD\n\tpath {' '.join(keep)}\n}}\n"
|
||||
f"handle @public_previous {{\n\troot * {prev}/static\n\tfile_server\n}}\n")
|
||||
else:
|
||||
new_prev = ("# GENERATED by /usr/local/sbin/svos-front-inventory; do not edit.\n"
|
||||
"# No previous release with assets to keep: nothing is served from here.\n")
|
||||
|
||||
if check:
|
||||
for name, old, new in (("current", old_cur, new_cur), ("previous", old_prev, new_prev)):
|
||||
print(f"{name}: " + ("installed file is current" if old == new else "installed file DIFFERS:\n" + new))
|
||||
return
|
||||
if old == new:
|
||||
if old_cur == new_cur and old_prev == new_prev:
|
||||
print("unchanged; nothing to do")
|
||||
return
|
||||
tmp = OUT.with_name(OUT.name + ".new")
|
||||
tmp.write_text(new, encoding="utf-8")
|
||||
shutil.chown(tmp, "root", "caddy")
|
||||
os.chmod(tmp, 0o640)
|
||||
backup = OUT.with_name(OUT.name + ".prev")
|
||||
if OUT.exists():
|
||||
shutil.copy2(OUT, backup)
|
||||
os.replace(tmp, OUT)
|
||||
|
||||
saved = {}
|
||||
for path, text in ((OUT, new_cur), (PREV_OUT, new_prev)):
|
||||
if path.exists():
|
||||
saved[path] = path.read_text(encoding="utf-8")
|
||||
tmp = path.with_name(path.name + ".new")
|
||||
tmp.write_text(text, encoding="utf-8")
|
||||
if not TEST_ROOT:
|
||||
shutil.chown(tmp, "root", "caddy")
|
||||
os.chmod(tmp, 0o640)
|
||||
os.replace(tmp, path)
|
||||
if TEST_ROOT:
|
||||
print("TEST: files written, validate/reload skipped")
|
||||
return
|
||||
if subprocess.run(VALIDATE).returncode != 0:
|
||||
if backup.exists():
|
||||
os.replace(backup, OUT)
|
||||
die("caddy validate FAILED; previous inventory restored, caddy not reloaded")
|
||||
for path in (OUT, PREV_OUT):
|
||||
if path in saved:
|
||||
path.write_text(saved[path], encoding="utf-8")
|
||||
else:
|
||||
path.unlink()
|
||||
die("caddy validate FAILED; previous files restored, caddy not reloaded")
|
||||
subprocess.run(["systemctl", "reload", "caddy-svos"], check=True)
|
||||
print("inventory updated and caddy-svos reloaded:\n" + new)
|
||||
print("updated and caddy-svos reloaded:\n" + new_cur + new_prev)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
Reference in New Issue
Block a user