Commit Graph
174 Commits
Author SHA1 Message Date
vh 5e8307ed7a ops(svos): v0.0.11 (452a592) on nh3-svos + nh3-dev — kernel 0.0.11, schema 13, U15 [tasks] config; terminal 0.0.4
Built in-session from a clean archive of the tag (kernel, connector base and
terminal wheels; everything else carried from 7f95e54). Ran after the 21:30
vzdump of VM 111, with an offline backup of the schema 12 database first.
check-config accepted the new [tasks] table and the tasks artifact mapping
(host nh3-svos, the terminal's artifact_host). Kernel READY in 1.17 s,
schema 13, apps up 7. nh3-dev's current moved to 452a592; svos-terminal 0.0.4
registered manifest revision 4, which was pinned without a card.
2026-10-13 21:33:16 -07:00
vh 7ca2f76d21 feat(albok): personal/inbox wing on nh3-albok + /srv/nemi/inbox capture root on nh3-dev (unit 4.4 slice C, gate 2)
Prime ruled albok subsumes note intake (1530). The wing is like personal/notes
(albok-personal 1511, not service-restricted); health reads 7 wings ready.
Nemi's token was re-minted with read+write on the new wing and vaulted over
albok/token-nemi. /srv/nemi/inbox is lkraven:nemi-drop 2770 with a default ACL,
plus u:lkraven:rx on /srv/nemi, so Nemi reaches it from lkraven's user manager,
whose groups predate nemi-drop. Both paths are verified by the playbook.
2026-10-13 17:31:49 -07:00
vh 30a7220b2c ops(esh-nas): infra-ops identity bootstrapped in CT 103 via pct from esh-pve-nas
Prime's ruling 2026-10-13. esh-nas (10.0.50.50) had no infra-ops user, so key
login was refused. The same steps as bootstrap-infra-ops-user.yaml (uid/gid
850, fleet key, NOPASSWD sudoers with log_output and the BULK_DATA carve-out,
visudo-validated) ran inside the container from its host. Verified with a
direct ssh infra-ops@10.0.50.50 and sudo -n whoami = root.
2026-10-13 13:25:21 -07:00
vh fa567fa150 ops(svos): U13 phase B amendment — full kernel-origin path list, Miranda's env and origin on nh3-dev, enrollment secrets, app-high-seat declared
Per svoperatingsystem dc241f8 u13-svos-edit.md items 1, 4, 5 and 6 (Prime's
go stands): the kernel origin passes exactly the method-pinned list (apps and
Miranda's agent connector; 15 pairs reach the kernel, the rest are Caddy 404);
nh3-dev's apps env is rebuilt as dc241f8 with svos-miranda 0.0.3 and
svos-agent 0.0.3 and gains the svos-miranda origin (nh3-svos only); three
enrollment secrets minted into the vault; app-high-seat declared with one
reload. Also fixes nh3-dev's fleet Caddy sharing admin localhost:2019 with
ttyd-caddy via SO_REUSEPORT (reloads could land on either): now :2021.
2026-10-10 16:56:47 -07:00
vh feef8330ca ops(svos): U13 phase B — nh3-svos on v0.0.9 (schema 12), kernel app origin, seat-host and high-seat origins, bridge env on nh3-dev
Prime approved svos-dev's U13 pre-edit ask (2026-10-10 1639).
- B1: release afab123 on nh3-svos (kernel 0.0.9, migration 11->12) after a
  vzdump and an offline svos.db copy; app-terminal and app-postoffice held on
  b588510 by drop-ins until the edit.
- B2: --app-listen 127.0.0.1:8410 (drop-in) fronted by Caddy as
  svos-kernel.phasefinal.com on the LAN, allowlisted to nh3-dev; the path list
  adds /auth/kernel-key, which every connector client needs. DNS rewrite on
  the three internal AdGuards.
- B3/B4: svos-seathost (-> :8404) and svos-highseat (-> :8773) on nh3-dev's
  fleet Caddy, allowlisted to nh3-svos; /opt/svos apps env on nh3-dev with
  svos-postoffice[push] (althing-core 3.7.0 from althing 7a2a1c1) and
  svos-terminal. The fleet Caddyfile is now tracked in configs/.
2026-10-10 16:50:18 -07:00
vh 51da9ff5ea docs(nh3-svos): kernel 0.0.8 reader pool live; LimitNOFILE stopgap removed; soak flat 2026-10-09 05:45:14 -07:00
vh d2f9f561c2 docs(nh3-svos): refreshed to SVOS ac42ace — kernel 0.0.7 + Type=notify pair, thaw race fixed (drill 2 -> 0) 2026-10-08 15:17:56 -07:00
vh b04cee5579 feat(nh3-svos): SVOS voice (app-voice) live — /voice route before Basic auth (lease bearer), unit + quiesce hook; app credentials vaulted 2026-10-08 13:55:13 -07:00
vh a3184f66b9 docs(nh3-svos): front door on mesh + LAN, internal DNS answers the LAN address; fv-ml1 off-mesh by design 2026-10-08 13:32:03 -07:00
vh cab00585ca docs(nh3-svos): svos names now also rewritten on the three internal AdGuards (mesh-reachable only) 2026-10-08 13:26:53 -07:00
vh 423bc946df docs(fv-ml1): patched to kernel 6.12.111, ZFS 2.3.9, NVIDIA 580.178.04, toolkit 1.20.1; ZBM boot and the hand-verified driver path recorded 2026-10-08 00:42:50 -07:00
vh 469b3ba3cf feat(nh3-svos): SVOS a737a06 refresh — app-terminal on its own mesh origin (svos-terminal.phasefinal.com), app-knowledge, app-postoffice units; quiesce hook covers the new units 2026-10-07 23:17:18 -07:00
vh 1210b4a0d5 feat(svos-front,backups): SVOS hosting bounds and previous-release assets; backup check flags fs-freeze failures and covers nh3-pve-2
Front door: request bodies capped at 1 MiB, headers within 5 s, body within 15 s, no response deadlines (SSE stays open). Validated live: 413 over the cap, stalled body cut at 15 s, signed ping and an SSE stream unaffected. svos-front-inventory also serves the previous release's hashed assets credential-free from that release's static dir, hash-checked. Backup check: reads the logs of vzdump tasks PVE records as OK and reports ERROR lines; an fs-freeze/thaw error is an application-consistency failure, new verdict DEGRADED (exit 4), wired into the alert wrapper. nh3-pve-2 added to the node and namespace lists (its job results were never read). The svos drill task is acknowledged by UPID; BACKUP_IGNORE_ACKS=1 counts it (positive control: exit 4).
2026-10-05 20:22:16 -07:00
vh e369b1c769 feat(nh3-svos): credential-free SVOS offline shell before Basic auth; contract static routes; backup drill results
svos-dev: the service worker fetches its hash-pinned INVENTORY with credentials omitted. Exactly those files plus /sw.js are now served before basic_auth (GET/HEAD, exact files). The list is generated from the release's sw.js by svos-front-inventory, which hash-checks each file and validates before reload. Gated static routes follow the web/README hosting contract: missing assets 404 instead of index.html, other document requests get index.html, everything else 404, unversioned files no-cache. acceptance-matrix.sh: 35/35 live; 16 FAIL against the previous config (positive control). git added to the nh3-svos playbook (kernel's remote check). Live backup drill: quiesced backup ~1-2 s outage; a refused freeze does NOT abort vzdump (unfrozen backup, job reported successful); both images restore-mounted read-only and verified.
2026-10-05 19:58:19 -07:00
vh 7c3eee877a docs(nh3-svos): correct the basic-auth test time 2026-10-05 19:47:44 -07:00
vh c523806566 feat(nh3-svos): SVOS front door uses HTTP Basic auth (login Vuong); WhoIs gate retired
Prime's ruling: basic auth saved in the browser, mesh-only access. basic_auth argon2id with the account hash in a root:caddy file outside git (plaintext vaulted). After auth, Tailscale-User-Login is set from the authenticated user and Authorization is deleted, since the kernel's human path refuses requests that carry it. Removed svos-whois-auth (every mesh node is login pfi). Verified from a mesh peer against a temporary header echo: 401 without/with wrong credentials, 200 with exactly one Tailscale-User-Login: Vuong and no forged or Authorization headers upstream.
2026-10-05 19:47:19 -07:00
vh 454b689a54 fix(nh3-svos): strip Tailscale-User-* and X-Forwarded-* as whole families at the SVOS front door
svos-dev review: the Caddyfile deleted named headers only. Prefix deletes cover the families; the named strays (Tailscale-Headers-Info, X-Real-IP, Forwarded, X-SVOS-Peer) stay. Wildcard semantics checked on a scratch Caddy (case-insensitive; look-alikes outside the family survive). Deployed, validated via systemd-run with EnvironmentFile, reloaded; deny tests re-run (403). README: safe validate recipe, pending acceptance list.
2026-10-05 19:35:10 -07:00
vh ce398685c0 feat(nh3-svos): mesh-only front door for svos.phasefinal.com (Caddy + WhoIs fail-closed auth), headscale extra record 2026-10-05 19:31:41 -07:00
vh 9175fbd539 feat(qemu-ga): failure-propagating fsfreeze dispatcher; enabled on nh3-svos (hooks were never run) 2026-10-05 19:25:14 -07:00
vh 36289f701a ops(beszel): native agents + hub registration for nh3-svos and nh3-albok; nh3-svos app/ingress notes 2026-10-05 19:04:13 -07:00
vh d7e845eb64 feat(nh3-svos): VM 111 on nh3-pve-2 for the new SVOS (svos-dev, Prime); generic cloud-init VM playbook 2026-10-05 18:59:53 -07:00
vh 2c4c7ca9ae ops(esh-docker-vm): Homepage on https via a file-provider Traefik router; note the dead Route53 ACME resolver 2026-10-05 18:38:08 -07:00
vh 3429b5c55d ops(pfi-pteradactyl): docker.io/containerd upgraded (manual dockerd + wings restart); docs 2026-10-05 14:18:57 -07:00
vh 5301ede9d2 ops(ana-docker): docker-ce from bookworm repo (29.8.2); live-restore stale-socket lesson; docs 2026-10-05 13:31:30 -07:00
vh aa86c37954 ops(pfi-pteradactyl): mask leftover apache2, patch (MariaDB 11.8), engine held; docs 2026-10-05 10:41:23 -07:00
vh ac068dc2bd ops(pfi-tacticalrmm): *.phasefinal.com renews via dns-cloudflare (was manual; 16 days from expiry) 2026-10-05 08:43:15 -07:00
vh 67c72ed228 docs(ana-docker): crowdsec sqlite WAL fix for the bouncer fail-closed 403s 2026-10-05 04:19:52 -07:00
vh 49c90f3083 docs(fv-ml1): drop the stale 'docker still on zroot' line 2026-10-05 02:43:31 -07:00
vh 23371954f8 docs(fv-ml1): docker on tank zvol, held packages, kernel metas removed; memory 2026-10-05 02:43:19 -07:00
vh 7f34fe5a7c docs(irv-ml1, ana-docker): storage cleanup, text encoders on worktank, Traefik readTimeout 10m 2026-10-05 01:19:36 -07:00
vh c0d3384c68 feat(litellm): embedder primary/failover split — qwen3-embedding on esh-ml1, new qwen3-embedding-bulk on nh3-ml1
Prime's ruling 2026-10-05 (relayed by albok-dev, confirmed directly) replaces
the 2026-09-26 simple-shuffle load-share for embeddings only; the reranker
groups are unchanged. Routing uses LiteLLM per-deployment `order` (semantics
read at source in v1.97.0): lowest order among healthy deployments, and a
failed request retries at the next order.

Verified after the restart (43 s gateway outage): 6/6 calls per alias landed
on its primary. With nh3-ml1 tei-embed stopped, qwen3-embedding-bulk served 6/6
from esh-ml1 with 200s; once nh3-ml1 was back, bulk returned 6/6 to nh3-ml1.
The albok-service key now covers qwen3-embedding-bulk (200); the Nemi key does
not (403).
2026-10-05 00:32:42 -07:00
vh a7a068f56f albok: move closed — acceptance passed, old nh3-docker store deleted, first VM 110 backup taken 2026-10-05 00:21:15 -07:00
vh d3f59aa2c5 feat(albok): move albok-service to a dedicated VM, nh3-albok (VM 110 on nh3-pve-2)
albok-service leaves nh3-docker before unit 4 puts the personal vault and
licensed books into restricted wings. On nh3-docker, portainer, portainer_agent,
dockge (socket rw) and dockerd on plaintext tcp/2375 were each root over
/srv/albok. The new host runs only albok-service: no socket containers and no
TCP Docker API, so its Homepage tile is a static services.yaml entry.

- nh3-pve-2: vmbr0 made VLAN-aware (applied by reboot, no guests running);
  pbs-ana storage in the new namespace nh3-pve-2, plus a 21:30 vzdump job
  (pbs-nh3's recursive sync mirrors it)
- VM 110 nh3-albok, 10.100.50.44 on VLAN 50, Debian 13 cloud-init, 4 vCPU
  x86-64-v3 / 12 GiB / 100 GB; built to fleet conventions (vh 1000,
  infra-ops 850, docker 851)
- compose: mem_limit 4g -> 8g, homepage labels removed; host port 8392 kept
- cutover by tar stream, verified by digest (contents plus owner, group, mode,
  mtime and type identical); albok.nh3.internal now points at nh3-albok
- old /srv/albok stays on nh3-docker (container removed, compose renamed)
  until acceptance and the operator's approval to delete it
- sudo-io archive hosts and ops-log STACK_HOSTS include nh3-albok
2026-10-04 23:33:24 -07:00
vh 008bd685aa feat(sudo-io-archive): weekly fleet archive of sudo session recordings (pack, verify, then clear); fv-ml1 HF_HOME on tank and root-pool cleanup 2026-10-04 22:48:05 -07:00
vh 513f6ef38f ops: phasefinal.com moved to Cloudflare Pages; pfi-ana-webhost destroyed and removed from inventory, DNS and the FortiGate 2026-10-04 17:28:29 -07:00
vh 55e4d01b49 ops(pfi-ana-webhost): retired (VM 110 stopped, FortiGate policy 18 disabled); dragonfireacoustics.com located on Cloudflare Pages 2026-10-04 16:57:49 -07:00
vh 0e3ab447af ops: retire VM 101 (DC) and VM 104 (nh3-laser) from running + backups; infra-ops login on pbs-ana/pbs-nh3; nh3-pve-2 follow-ups verified done 2026-10-04 16:37:15 -07:00
vh d0c1bc628a ops(pbs): pbs-nh3 + pbs-ana upgraded to PBS 4.2.7 / Debian 13; guest-agent driver script, plan and log 2026-10-04 11:44:57 -07:00
vh 73129a0564 ops(esh-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20 (one reboot, Prime on site); ESH cluster uniform on 9 2026-10-04 07:45:05 -07:00
vh b67ab48833 ops(pfi-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20; reboot test + post-check recorded; pre-reboot checks corrected for PVE 9 2026-10-04 07:16:21 -07:00
vh 5f8a1311a5 memory: snapshot — 2026-10-03 ~1440 (nh3-pve clean on 9, RustDesk 1.1.16, esh-nas-pve PVE 9 assessed) 2026-10-03 15:23:19 -07:00
vh d4675f2cb5 ops(ana-docker): rustdesk-server 1.1.14 -> 1.1.16; server key vaulted 2026-10-03 14:24:32 -07:00
vh 22224b5335 ops(nh3-pve-2): register host, network/AMT/disk layout and the re-IP lesson 2026-10-03 13:17:46 -07:00
vh 00a6fbd849 feat(cira-tunnel-watchdog): auto-drop stale AMT CIRA tunnels on MeshCentral's MPS 2026-10-03 12:34:35 -07:00
vh e4fcd1398d docs(meshcentral): stale CIRA tunnel, third occurrence (nh3-pve-2) 2026-10-03 12:19:42 -07:00
vh 36c52d8860 docs(meshcentral): stale CIRA tunnel reproduced on a single shutdown 2026-10-02 22:53:30 -07:00
vh ad0322bf83 ops(meshcentral): diagnose and fix HW Connect stuck at Setup (stale CIRA tunnel); add relay probe 2026-10-02 22:49:27 -07:00
vh 55004090d1 ops(esh-pve-2): register host, AMT phoning home to MeshCentral; note MeshCentral first-CIRA crash race 2026-10-02 22:25:37 -07:00
vh 56c372dd6f docs(pfi-tacticalrmm): rmm-mesh nginx upload limit raised to 4G (was the 1 MB default) 2026-10-02 17:08:21 -07:00
vh 5c0d5f0a73 docs(pfi-tacticalrmm): MeshCentral site-admin account lkraven 2026-10-02 17:06:19 -07:00