feat(nh3-svos): credential-free SVOS offline shell before Basic auth; contract static routes; backup drill results

svos-dev: the service worker fetches its hash-pinned INVENTORY with credentials omitted. Exactly those files plus /sw.js are now served before basic_auth (GET/HEAD, exact files). The list is generated from the release's sw.js by svos-front-inventory, which hash-checks each file and validates before reload. Gated static routes follow the web/README hosting contract: missing assets 404 instead of index.html, other document requests get index.html, everything else 404, unversioned files no-cache. acceptance-matrix.sh: 35/35 live; 16 FAIL against the previous config (positive control). git added to the nh3-svos playbook (kernel's remote check). Live backup drill: quiesced backup ~1-2 s outage; a refused freeze does NOT abort vzdump (unfrozen backup, job reported successful); both images restore-mounted read-only and verified.
This commit is contained in:
vh
2026-10-05 19:58:19 -07:00
parent 7c3eee877a
commit e369b1c769
6 changed files with 186 additions and 7 deletions
+39 -2
View File
@@ -33,6 +33,21 @@ https://svos.phasefinal.com:443 {
request_header -Forwarded
request_header -X-SVOS-Peer
# Credential-free offline shell (svos-dev 2026-10-05): the service worker fetches its hash-pinned INVENTORY
# with credentials:'omit', so exactly those files plus /sw.js are served BEFORE Basic auth, GET/HEAD only,
# exact files (no fallback). Public build bytes only; no API or connector path. The path list is GENERATED
# from the live release's sw.js by /usr/local/sbin/svos-front-inventory, which must run after every release.
@public_shell {
method GET HEAD
import /etc/caddy/svos-public-inventory
}
handle @public_shell {
@revalidate path /sw.js /index.html /manifest.webmanifest /favicon.svg /icons/*
header @revalidate Cache-Control no-cache
root * /opt/svos/current/static
file_server
}
basic_auth argon2id SVOS {
import /etc/caddy/svos-basic-auth.accounts
}
@@ -46,10 +61,32 @@ https://svos.phasefinal.com:443 {
handle @connector {
reverse_proxy 127.0.0.1:8403
}
handle {
# Static routes per svoperatingsystem web/README.md "Hosting contract" (gated):
# / and /status -> index.html; the static route space -> exact files, a missing file is 404 (never index.html);
# other HTML document requests -> index.html (the app renders Page not found); everything else -> 404.
@shell_doc path / /status
handle @shell_doc {
header Cache-Control no-cache
root * /opt/svos/current/static
try_files {path} /index.html
rewrite * /index.html
file_server
}
@static path /assets/* /icons/* /favicon.svg /manifest.webmanifest /index.html /sw.js
handle @static {
@revalidate path /sw.js /index.html /manifest.webmanifest /favicon.svg /icons/*
header @revalidate Cache-Control no-cache
root * /opt/svos/current/static
file_server
}
@other_doc expression header({'Sec-Fetch-Dest': 'document'}) || header({'Accept': '*text/html*'})
handle @other_doc {
header Cache-Control no-cache
root * /opt/svos/current/static
rewrite * /index.html
file_server
}
handle {
respond 404
}
}
}
+47
View File
@@ -0,0 +1,47 @@
#!/bin/bash
# SVOS front-door acceptance matrix (eshpfi configs/svos-front/), 2026-10-05.
# Run from a MESH peer. Feed the variables on stdin ahead of this file so no credential touches an argv:
# pw=$(secret get nh3-svos/basic-auth-vuong)
# { printf 'BASE=https://svos.phasefinal.com\nCURLX="--resolve svos.phasefinal.com:443:100.64.0.9"\nCRED=%s\nINV=%q\nIDX_SHA=%s\n' \
# "Vuong:$pw" "$(<lines 'path sha256' from the release sw.js INVENTORY>)" "$(<sha256 of static/index.html>)"
# cat configs/svos-front/acceptance-matrix.sh; } | ssh infra-ops@<mesh-peer> 'bash -s'
# Positive control (2026-10-05): against the previous Caddyfile (no public shell, try_files fallback) it reports
# 16 FAIL, so it can see the behaviours it checks.
# expects: BASE, CURLX (extra curl args), CRED (user:pass), INV (lines "path sha256"), IDX_SHA (sha256 of index.html)
pass=0; fail=0
ok() { pass=$((pass+1)); printf 'PASS %s\n' "$1"; }
bad() { fail=$((fail+1)); printf 'FAIL %s :: %s\n' "$1" "$2"; }
req() { # $1 name, $2 auth(y/n), then curl args; sets CODE REALM CT CC SHA
local name=$1 auth=$2; shift 2
local hdr=$(mktemp) body=$(mktemp)
if [ "$auth" = y ]; then CODE=$(curl -s $CURLX -K <(printf 'user = "%s"\n' "$CRED") -D "$hdr" -o "$body" -w '%{http_code}' "$@")
else CODE=$(curl -s $CURLX -D "$hdr" -o "$body" -w '%{http_code}' "$@"); fi
REALM=$(grep -ci '^www-authenticate: Basic realm="SVOS"' "$hdr"); CT=$(grep -i '^content-type:' "$hdr" | tr -d '\r' | cut -d' ' -f2-)
CC=$(grep -i '^cache-control:' "$hdr" | tr -d '\r' | cut -d' ' -f2-); SHA=$(sha256sum "$body" | cut -d' ' -f1); rm -f "$hdr" "$body"
}
gated() { req "$1" n "${@:2}"; [ "$CODE" = 401 ] && [ "$REALM" = 1 ] && ok "$1 -> 401 realm" || bad "$1" "code=$CODE realm=$REALM"; }
echo "== unauthenticated: exact inventory is public, hash-preserved"
while read -r p h; do req "U1 GET $p" n "$BASE$p"; [ "$CODE" = 200 ] && [ "$SHA" = "$h" ] && ok "U1 GET $p 200 sha ok cc=[$CC]" || bad "U1 GET $p" "code=$CODE sha=$SHA"; done <<< "$INV"
req "U1 GET /sw.js" n "$BASE/sw.js"; [ "$CODE" = 200 ] && echo "$CT" | grep -q javascript && [ "$CC" = no-cache ] && ok "U1 GET /sw.js 200 ct=[$CT] cc=[$CC]" || bad "U1 /sw.js" "code=$CODE ct=$CT cc=$CC"
req "U2 HEAD /index.html" n -I "$BASE/index.html"; [ "$CODE" = 200 ] && ok "U2 HEAD /index.html 200" || bad "U2" "code=$CODE"
gated "U3 POST /index.html" -X POST "$BASE/index.html"
gated "U4 GET /assets/missing.js" "$BASE/assets/missing.js"
gated "U5 GET /" "$BASE/"
gated "U5 GET /status" "$BASE/status"
for p in /api/health /health /openapi.json /verbs/ping /verbs/pins; do gated "U6 GET $p" "$BASE$p"; done
gated "U7 GET /assets/../api/health (path-as-is)" --path-as-is "$BASE/assets/../api/health"
gated "U8 GET /index.html/" "$BASE/index.html/"
req "U9 GET /INDEX.HTML" n "$BASE/INDEX.HTML"; [ "$CODE" = 404 ] || [ "$CODE" = 401 ] && ok "U9 /INDEX.HTML -> $CODE (no bytes served)" || bad "U9" "code=$CODE"
gated "U10 GET /%2e%2e/etc/passwd" --path-as-is "$BASE/%2e%2e/etc/passwd"
gated "U11 GET /icons/" "$BASE/icons/"
echo "== authenticated"
for p in / /status; do req "A1 GET $p" y -H 'Sec-Fetch-Dest: document' "$BASE$p"; [ "$CODE" = 200 ] && [ "$SHA" = "$IDX_SHA" ] && [ "$CC" = no-cache ] && ok "A1 GET $p -> index.html cc=[$CC]" || bad "A1 $p" "code=$CODE sha=$SHA cc=$CC"; done
req "A3 GET /assets/missing.js" y "$BASE/assets/missing.js"; [ "$CODE" = 404 ] && [ "$SHA" != "$IDX_SHA" ] && ok "A3 missing asset -> 404 (not index)" || bad "A3" "code=$CODE"
req "A3b GET /assets/missing.js as document" y -H 'Sec-Fetch-Dest: document' "$BASE/assets/missing.js"; [ "$CODE" = 404 ] && ok "A3b missing asset as document -> 404" || bad "A3b" "code=$CODE"
req "A4 GET /icons/" y "$BASE/icons/"; [ "$CODE" = 404 ] && ok "A4 /icons/ -> 404" || bad "A4" "code=$CODE"
req "A5 GET /nowhere (no doc headers)" y "$BASE/nowhere"; [ "$CODE" = 404 ] && ok "A5 /nowhere -> 404" || bad "A5" "code=$CODE"
req "A6 GET /nowhere Sec-Fetch-Dest" y -H 'Sec-Fetch-Dest: document' "$BASE/nowhere"; [ "$CODE" = 200 ] && [ "$SHA" = "$IDX_SHA" ] && ok "A6 /nowhere document -> index.html" || bad "A6" "code=$CODE"
req "A7 GET /nowhere Accept html" y -H 'Accept: text/html,application/xhtml+xml' "$BASE/nowhere"; [ "$CODE" = 200 ] && [ "$SHA" = "$IDX_SHA" ] && ok "A7 /nowhere Accept html -> index.html" || bad "A7" "code=$CODE"
for p in /api/health /health; do req "A8 GET $p" y "$BASE$p"; [ "$REALM" = 0 ] && ok "A8 GET $p passes Caddy auth (no Basic challenge) -> upstream $CODE" || bad "A8 $p" "code=$CODE realm=$REALM"; done
req "A9 POST /index.html" y -X POST "$BASE/index.html"; [ "$CODE" = 405 ] || [ "$CODE" = 404 ] && ok "A9 POST /index.html authed -> $CODE" || bad "A9" "code=$CODE"
echo "== $pass pass, $fail fail"
+91
View File
@@ -0,0 +1,91 @@
#!/usr/bin/env python3
"""svos-front-inventory: regenerate the SVOS front door's credential-free shell paths from the live release.
eshpfi configs/svos-front/, installed at /usr/local/sbin/svos-front-inventory on nh3-svos. Run as root after every
SVOS release (after /opt/svos/current moves), or the service worker's install breaks: its INVENTORY names hashed
asset files, and only those exact paths (plus /sw.js) are served before Basic auth (svos-dev, 2026-10-05).
What it does:
1. Reads INVENTORY from /opt/svos/current/static/sw.js (a JSON object of path -> sha256).
2. Refuses unless every path is a plain absolute file path and every file exists with exactly that sha256.
3. Writes /etc/caddy/svos-public-inventory (one Caddyfile `path` line, root:caddy 0640) atomically.
4. Validates the whole Caddyfile as the caddy user with the unit's EnvironmentFile (no secret on any argv),
restores the previous inventory file if validation fails, and otherwise reloads caddy-svos.
--check: steps 1-2 only, plus a diff against the installed file; changes nothing.
"""
import hashlib
import json
import os
import re
import shutil
import subprocess
import sys
from pathlib import Path
STATIC = Path("/opt/svos/current/static")
OUT = Path("/etc/caddy/svos-public-inventory")
SAFE = re.compile(r"^/[A-Za-z0-9._-]+(/[A-Za-z0-9._-]+)*$")
VALIDATE = ["systemd-run", "--quiet", "--wait", "--pipe", "--collect", "-p", "User=caddy", "-p", "Group=caddy",
"-p", "EnvironmentFile=/etc/caddy/env",
"-p", "Environment=XDG_DATA_HOME=/var/lib/caddy XDG_CONFIG_HOME=/var/lib/caddy",
"/usr/local/bin/caddy", "validate", "--config", "/etc/caddy/Caddyfile", "--adapter", "caddyfile"]
def die(msg):
sys.exit(f"svos-front-inventory: {msg}")
def inventory():
src = (STATIC / "sw.js").read_text(encoding="utf-8")
m = re.search(r"^const INVENTORY = (\{.*?\});$", src, re.M)
if not m:
die("no `const INVENTORY = {...};` line in sw.js")
inv = json.loads(m.group(1))
if not inv:
die("INVENTORY is empty")
for path, digest in inv.items():
if not SAFE.match(path) or ".." in path.split("/"):
die(f"unsafe inventory path {path!r}")
f = STATIC / path.lstrip("/")
if not f.is_file() or f.is_symlink():
die(f"{path} is not a regular file in {STATIC}")
actual = hashlib.sha256(f.read_bytes()).hexdigest()
if actual != digest:
die(f"{path} sha256 {actual} != INVENTORY {digest}")
return sorted(inv)
def render(paths):
release = os.path.realpath(STATIC.parent)
return (f"# GENERATED by /usr/local/sbin/svos-front-inventory from {release}/static/sw.js; do not edit.\n"
f"# Exact credential-free shell paths (hash-checked), plus /sw.js.\n"
f"path {' '.join(paths + ['/sw.js'])}\n")
def main():
new = render(inventory())
old = OUT.read_text(encoding="utf-8") if OUT.exists() else ""
if "--check" in sys.argv[1:]:
print("inventory ok;", "installed file is current" if old == new else "installed file DIFFERS:\n" + new)
return
if old == new:
print("unchanged; nothing to do")
return
tmp = OUT.with_name(OUT.name + ".new")
tmp.write_text(new, encoding="utf-8")
shutil.chown(tmp, "root", "caddy")
os.chmod(tmp, 0o640)
backup = OUT.with_name(OUT.name + ".prev")
if OUT.exists():
shutil.copy2(OUT, backup)
os.replace(tmp, OUT)
if subprocess.run(VALIDATE).returncode != 0:
if backup.exists():
os.replace(backup, OUT)
die("caddy validate FAILED; previous inventory restored, caddy not reloaded")
subprocess.run(["systemctl", "reload", "caddy-svos"], check=True)
print("inventory updated and caddy-svos reloaded:\n" + new)
if __name__ == "__main__":
main()
+1 -1
View File
@@ -301,7 +301,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
## Recent decisions
- `[2026-10-05]` **nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only `https://svos.phasefinal.com` with HTTP Basic auth, login `Vuong` (Prime, direct)** — WhoIs gate retired (all mesh nodes are login `pfi`; Prime declined a mesh restructure). Password vaulted `nh3-svos/basic-auth-vuong`. qemu-ga freeze hooks were never run on Debian; now `configs/qemu-ga/fsfreeze-dispatch` (fails the freeze on a hook failure). Live-service backup drill pending. → `servers/nh3-svos/README.md`.
- `[2026-10-05]` **nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only `https://svos.phasefinal.com` with HTTP Basic auth, login `Vuong` (Prime, direct)** — WhoIs gate retired (all mesh nodes are login `pfi`; Prime declined a mesh restructure). Password vaulted `nh3-svos/basic-auth-vuong`. qemu-ga freeze hooks were never run on Debian; now `configs/qemu-ga/fsfreeze-dispatch` (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"). Offline shell assets served before auth; rerun `svos-front-inventory` after every SVOS release. → `servers/nh3-svos/README.md`.
- `[2026-10-05]` **fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE)** — run 1 aborted on its own guard (live rsync walked into an overlay `merged` view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (`-x`, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → `servers/fv-ml1/README.md`
- `[2026-10-05]` **ana-docker: Docker CE repo bullseye → bookworm, 29.8.2 (Prime)** — with `live-restore: true` the engine upgrade kept containers up, but 6 that bind-mount `docker.sock` (crowdsec, traefik, dozzle, beszel-agent, dockge, gitea-runner) went stale and needed restarts. → `servers/ana-docker/README.md`
- `[2026-10-05]` **RMM wildcard cert now auto-renews (Prime)** — `*.phasefinal.com` on pfi-tacticalrmm was certbot `manual` (could never renew; was 16 days from expiry). Now dns-cloudflare with a phasefinal.com-only token (vault `pfi-tacticalrmm/cloudflare-certbot-dns01-token`); reissued to 2027-01-03, dry-run OK. → `servers/pfi-tacticalrmm/README.md`
+3 -2
View File
@@ -9,6 +9,7 @@
# - /var/lib/svos svc-svos 0700, LOCAL ext4 (the root disk on vmstore): SQLite WAL, one writer, never shared/NFS
# - /opt/svos root:svc-svos 2775, the release tree (releases/<id>/ + a `current` link; svos-dev lays it out)
# - Python 3.13 (Debian 13) + venv, sqlite3, qemu-guest-agent (vzdump freeze/thaw), timesyncd checked
# - git: the SVOS kernel's data-root production-remote check needs it (svos-dev, 2026-10-05); no checkout lives here
# - /etc/qemu/fsfreeze-hook.d/ exists for the application-consistent snapshot hook svos-dev will add (the guest
# agent runs those scripts on vzdump's freeze/thaw)
# No Docker: nothing here needs it.
@@ -30,8 +31,8 @@ steps:
- name: base packages
sudo: true
shell: DEBIAN_FRONTEND=noninteractive apt-get install -y -q qemu-guest-agent python3 python3-venv sqlite3 rsync jq curl ca-certificates
when: "! dpkg -s qemu-guest-agent python3-venv sqlite3 rsync jq >/dev/null 2>&1"
shell: DEBIAN_FRONTEND=noninteractive apt-get install -y -q qemu-guest-agent python3 python3-venv sqlite3 rsync jq curl ca-certificates git
when: "! dpkg -s qemu-guest-agent python3-venv sqlite3 rsync jq git >/dev/null 2>&1"
- name: qemu-guest-agent running
sudo: true
+5 -2
View File
@@ -20,7 +20,7 @@ configuration, enrollment and acceptance.
| Releases | `/opt/svos` root:svc-svos 2775, with `releases/` for svos-dev's layout. The git checkout is not a runtime dependency |
| Docker | none, by design |
| Backups | nh3-pve-2 vzdump job, nightly **21:30**, `all 1` → pbs-ana namespace `nh3-pve-2` (mirrored to pbs-nh3 at 06:30), snapshot mode with guest-agent fs-freeze |
| App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. Tested 2026-10-05 via `qm guest cmd 111 fsfreeze-*`, both success and failure paths. The live-service backup drill is still pending |
| App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events` |
| Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 |
## Build (reproducible)
@@ -48,6 +48,9 @@ scripts/elway infra-ops@10.100.50.43 --playbook playbooks/nh3-svos-host.yaml
The plaintext is vaulted at **`nh3-svos/basic-auth-vuong`**. To rotate: pipe the new password to
`caddy hash-password --algorithm argon2id` on stdin (never `--plaintext`), rewrite the accounts file, `secret put`,
then `systemctl reload caddy-svos`.
- **Credential-free offline shell (svos-dev, 2026-10-05):** the service worker fetches its hash-pinned INVENTORY with `credentials:'omit'`, so exactly those files plus `/sw.js` are served BEFORE Basic auth (GET/HEAD, exact files, no fallback). Public build bytes only; `/`, `/status`, `/api/*`, the connector and everything else stay gated. The path list is **generated** into `/etc/caddy/svos-public-inventory` by **`/usr/local/sbin/svos-front-inventory`** (`configs/svos-front/`): it hash-checks every file against INVENTORY, validates, then reloads. ⚠ **Run it as root after every SVOS release**, or the next build's hashed asset names stay gated and worker install fails. `--check` changes nothing.
- **Static routes (gated), per svoperatingsystem `web/README.md` Hosting contract:** `/` and `/status` → index.html; `/assets/*`, `/icons/*`, `/favicon.svg`, `/manifest.webmanifest`, `/index.html`, `/sw.js` → exact file, missing → 404; other HTML document requests (`Sec-Fetch-Dest: document` or `Accept: text/html`) → index.html; anything else → 404. Unversioned files get `Cache-Control: no-cache`.
- **Acceptance matrix:** `configs/svos-front/acceptance-matrix.sh` (35 checks; usage in its header). 35/35 pass live at 1957 2026-10-05 from fv-ml1; against the previous config it reports 16 FAIL (positive control).
- **Headers:** client copies of `Tailscale-User-*`, `X-Forwarded-*`, `Tailscale-Headers-Info`, `X-Real-IP`, `Forwarded`
and `X-SVOS-Peer` are deleted before auth. After auth, `Tailscale-User-Login` is SET to the authenticated user, and
**`Authorization` is deleted**. That deletion is load-bearing: the kernel's human path (svoperatingsystem
@@ -68,7 +71,7 @@ scripts/elway infra-ops@10.100.50.43 --playbook playbooks/nh3-svos-host.yaml
headers, `X-Forwarded-Proto: https`, `Host: svos.phasefinal.com`, and path `/health` (the `/api` prefix stripped);
- connector route with right credentials → 502 (auth passed; the connector was not running); cert verifies; LAN
10.100.50.43:443 → connection refused.
- **Still to run:** the live-service vzdump + restore-mount drill, once svos-dev runs the services.
- **Open (svos-dev's numbers needed):** the hosting contract also asks the boundary to bound request body size and read duration for the connector; not set yet.
### Original requirements (svos-dev)