vh
454b689a54
svos-dev review: the Caddyfile deleted named headers only. Prefix deletes cover the families; the named strays (Tailscale-Headers-Info, X-Real-IP, Forwarded, X-SVOS-Peer) stay. Wildcard semantics checked on a scratch Caddy (case-insensitive; look-alikes outside the family survive). Deployed, validated via systemd-run with EnvironmentFile, reloaded; deny tests re-run (403). README: safe validate recipe, pending acceptance list.
79 lines
7.3 KiB
Markdown
79 lines
7.3 KiB
Markdown
# nh3-svos: host for the NEW SVOS operating system (VM 111 on nh3-pve-2, NH3)
|
|
|
|
Built 2026-10-05 at svos-dev's request, relaying Prime ("stand up a dedicated VM for it on nh3-pve-2"). It runs the
|
|
new SVOS (repo `corviduo/svoperatingsystem`): a FastAPI/uvicorn kernel (SQLite WAL, synchronous FULL, one writer),
|
|
an interface connector, an optional voice adapter and a static phone/desktop UI.
|
|
**It is a separate instance.** The existing High Seat (`svos.service` on nh3-dev, `:8770`) and Miranda are NOT
|
|
migrated by this host.
|
|
|
|
**Ownership split:** infra-ops owns the VM, OS, access, backups and ingress. svos-dev owns the application: install,
|
|
configuration, enrollment and acceptance.
|
|
|
|
| | |
|
|
|---|---|
|
|
| Address | `10.100.50.43`, VLAN 50 (nh3-servers), static via cloud-init; `nh3-svos.nh3.internal` |
|
|
| Guest | VM **111** on **nh3-pve-2**, `onboot 1`, tag `svos` |
|
|
| Shape | 4 vCPU `x86-64-v3`, 8 GiB, 100 GB on `vmstore` (local NVMe LVM-thin, discard on). nh3-pve-2 had 47 GB RAM and 946 GB vmstore free at build |
|
|
| OS | Debian 13 genericcloud, Python 3.13 + venv, SQLite 3.46, systemd-timesyncd (synced), qemu-guest-agent |
|
|
| Identities | `vh` 1000 (operator keys + lkraven@nh3-dev, so svos-dev's session can ssh in; NOPASSWD sudo from cloud-init, plus `/etc/sudoers.d/91-vh-verifypw` so `sudo -v` agrees), `infra-ops` 850, **`svc-svos` 801** (nologin; the only writer of the data root) |
|
|
| Data root | `/var/lib/svos`, svc-svos 0700, on the local root ext4. Never NFS, never shared |
|
|
| Releases | `/opt/svos` root:svc-svos 2775, with `releases/` for svos-dev's layout. The git checkout is not a runtime dependency |
|
|
| Docker | none, by design |
|
|
| Backups | nh3-pve-2 vzdump job, nightly **21:30**, `all 1` → pbs-ana namespace `nh3-pve-2` (mirrored to pbs-nh3 at 06:30), snapshot mode with guest-agent fs-freeze |
|
|
| App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. Tested 2026-10-05 via `qm guest cmd 111 fsfreeze-*`, both success and failure paths. The live-service backup drill is still pending |
|
|
| Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 |
|
|
|
|
## Build (reproducible)
|
|
|
|
```bash
|
|
scripts/elway infra-ops@10.100.250.62 --playbook playbooks/nh3-pve-2-cloudinit-vm.yaml --var vmid=111 \
|
|
--var name=nh3-svos --var ip=10.100.50.43/24 --var cores=4 --var memory=8192 --var disk=100G --var tags=svos \
|
|
--var "description=..."
|
|
scripts/elway vh@10.100.50.43 --playbook playbooks/bootstrap-infra-ops-user.yaml
|
|
scripts/elway infra-ops@10.100.50.43 --playbook playbooks/nh3-svos-host.yaml
|
|
```
|
|
|
|
## Front door: built 2026-10-05, mesh-only `https://svos.phasefinal.com` (Prime). Fail-closed until the owner identity is set
|
|
|
|
- **Mesh:** tailscaled joined headscale as `nh3-svos`, **100.64.0.9**, `--accept-dns=false --accept-routes=false`.
|
|
- **Name:** `svos.phasefinal.com` exists ONLY as a headscale MagicDNS extra record (`/var/lib/headscale/extra-records.json`
|
|
on nh3-headscale, hot-reloaded) → 100.64.0.9. There is no public record and no internal AdGuard record.
|
|
- **TLS:** Caddy 2.11.4 (the fleet `caddy-cf` build with the Cloudflare DNS module) at `/usr/local/bin/caddy`, unit
|
|
`caddy-svos.service`, config `/etc/caddy/Caddyfile` (canonical `configs/svos-front/Caddyfile`). It listens **only on
|
|
100.64.0.9:443**, with no `:80`. The LE cert for `svos.phasefinal.com` comes via DNS-01, token `nh3-svos-caddy-dns01`
|
|
(phasefinal.com only), in `/etc/caddy/env` (root:caddy 0640), vaulted `nh3-svos/cloudflare-caddy-dns01-token`.
|
|
- **Identity:** `svos-whois-auth.service` (`configs/svos-front/svos-whois-auth.py`, DynamicUser, 127.0.0.1:8405). Caddy's
|
|
`forward_auth` passes the SOCKET peer; the responder asks tailscaled WhoIs and answers 200 + `Tailscale-User-Login` only
|
|
for logins in **`/etc/svos-front/allowed-logins`** (empty = nobody). Client identity/forwarding headers are deleted
|
|
first, as whole families by prefix (`Tailscale-User-*`, `X-Forwarded-*`, svos-dev's ask) plus `Tailscale-Headers-Info`,
|
|
`X-Real-IP`, `Forwarded` and `X-SVOS-Peer`. The wildcard was tested on a scratch Caddy first: case-insensitive, and
|
|
look-alikes outside the family (`X-Forwardedness`, `Tailscale-Other`) survive.
|
|
- **Config changes:** validate WITHOUT putting the token on a command line, then reload:
|
|
`sudo systemd-run --wait --pipe --collect -p User=caddy -p Group=caddy -p EnvironmentFile=/etc/caddy/env
|
|
-p "Environment=XDG_DATA_HOME=/var/lib/caddy XDG_CONFIG_HOME=/var/lib/caddy" /usr/local/bin/caddy validate --config
|
|
<file> --adapter caddyfile`, then `systemctl reload caddy-svos`. (`env $(cat /etc/caddy/env) caddy …` leaked the first
|
|
token into sudo's argv log on 2026-10-05; it was revoked.)
|
|
- **Tested 2026-10-05:** LAN → no connection; mesh peer fv-ml1 (login `pfi`) → 403; forged `Tailscale-User-Login`,
|
|
`Tailscale-User-Whatever` and `X-Forwarded-For` → 403; the responder directly with a bogus, garbage or empty peer → 403.
|
|
- **Still to run (handoff to svos-dev's acceptance):** once a login is allowed, an allowed device gets 200 and its forged
|
|
`Tailscale-User-*` / `X-Forwarded-*` headers (arbitrary family members included) are replaced or gone at the app; a
|
|
stopped tailscaled gives 403; and the live-service vzdump + restore-mount drill once the SVOS services run.
|
|
- ⚠ **Every mesh node, servers included, has WhoIs LoginName `pfi`** (headscale has one user). `pfi` must never go in
|
|
the allow-list: it would admit every server and LAN traffic SNAT'd through a subnet router. Prime's decision pending:
|
|
a separate headscale user for his devices (recommended) or a device allow-list.
|
|
|
|
### Original requirements (svos-dev)
|
|
|
|
App side, per svos-dev 2026-10-05: kernel on `127.0.0.1:8400 --trust-loopback` (its Unix-socket mode needs a 0700
|
|
kernel-owned dir and creates a 0666 socket, so no shared-group socket in this build), interface connector on
|
|
`127.0.0.1:8403`. Static UI at `/opt/svos/current/static`. `/api/*` → kernel with the prefix stripped (`/api/health` is
|
|
the kernel's); `/health`, `/verbs/ping|pins|pin|unpin`, `/openapi.json` → connector unchanged. ⚠ With
|
|
`--trust-loopback`, every local process is inside the identity trust boundary, so run no untrusted workloads here.
|
|
Agreed design (option A): Caddy on the tailnet address only, `forward_auth` to a loopback WhoIs responder fed the real
|
|
socket peer (`{http.request.remote.host}:{port}`), fail closed, allow-list of logins, client identity/forwarding headers
|
|
stripped, Origin passed through untouched.
|
|
|
|
App listeners stay private (loopback or a Unix socket). The front door has to give the browser **HTTPS**: the UI and the
|
|
voice adapter need a secure context for the microphone. It also has to give the kernel's adapter a **verified identity**
|
|
in `Tailscale-User-Login`, with any browser-supplied copy stripped. See the svos-dev thread of 2026-10-05.
|