Commit Graph
1616 Commits
Author SHA1 Message Date
vh 1210b4a0d5 feat(svos-front,backups): SVOS hosting bounds and previous-release assets; backup check flags fs-freeze failures and covers nh3-pve-2
Front door: request bodies capped at 1 MiB, headers within 5 s, body within 15 s, no response deadlines (SSE stays open). Validated live: 413 over the cap, stalled body cut at 15 s, signed ping and an SSE stream unaffected. svos-front-inventory also serves the previous release's hashed assets credential-free from that release's static dir, hash-checked. Backup check: reads the logs of vzdump tasks PVE records as OK and reports ERROR lines; an fs-freeze/thaw error is an application-consistency failure, new verdict DEGRADED (exit 4), wired into the alert wrapper. nh3-pve-2 added to the node and namespace lists (its job results were never read). The svos drill task is acknowledged by UPID; BACKUP_IGNORE_ACKS=1 counts it (positive control: exit 4).
2026-10-05 20:22:16 -07:00
vh e369b1c769 feat(nh3-svos): credential-free SVOS offline shell before Basic auth; contract static routes; backup drill results
svos-dev: the service worker fetches its hash-pinned INVENTORY with credentials omitted. Exactly those files plus /sw.js are now served before basic_auth (GET/HEAD, exact files). The list is generated from the release's sw.js by svos-front-inventory, which hash-checks each file and validates before reload. Gated static routes follow the web/README hosting contract: missing assets 404 instead of index.html, other document requests get index.html, everything else 404, unversioned files no-cache. acceptance-matrix.sh: 35/35 live; 16 FAIL against the previous config (positive control). git added to the nh3-svos playbook (kernel's remote check). Live backup drill: quiesced backup ~1-2 s outage; a refused freeze does NOT abort vzdump (unfrozen backup, job reported successful); both images restore-mounted read-only and verified.
2026-10-05 19:58:19 -07:00
vh 7c3eee877a docs(nh3-svos): correct the basic-auth test time 2026-10-05 19:47:44 -07:00
vh c523806566 feat(nh3-svos): SVOS front door uses HTTP Basic auth (login Vuong); WhoIs gate retired
Prime's ruling: basic auth saved in the browser, mesh-only access. basic_auth argon2id with the account hash in a root:caddy file outside git (plaintext vaulted). After auth, Tailscale-User-Login is set from the authenticated user and Authorization is deleted, since the kernel's human path refuses requests that carry it. Removed svos-whois-auth (every mesh node is login pfi). Verified from a mesh peer against a temporary header echo: 401 without/with wrong credentials, 200 with exactly one Tailscale-User-Login: Vuong and no forged or Authorization headers upstream.
2026-10-05 19:47:19 -07:00
vh 454b689a54 fix(nh3-svos): strip Tailscale-User-* and X-Forwarded-* as whole families at the SVOS front door
svos-dev review: the Caddyfile deleted named headers only. Prefix deletes cover the families; the named strays (Tailscale-Headers-Info, X-Real-IP, Forwarded, X-SVOS-Peer) stay. Wildcard semantics checked on a scratch Caddy (case-insensitive; look-alikes outside the family survive). Deployed, validated via systemd-run with EnvironmentFile, reloaded; deny tests re-run (403). README: safe validate recipe, pending acceptance list.
2026-10-05 19:35:10 -07:00
vh ce398685c0 feat(nh3-svos): mesh-only front door for svos.phasefinal.com (Caddy + WhoIs fail-closed auth), headscale extra record 2026-10-05 19:31:41 -07:00
vh 63736bd40a vendor(qemu-ga): svos-dev's nh3-svos quiesce hook + its 6 controls beside the dispatcher 2026-10-05 19:28:02 -07:00
vh 9175fbd539 feat(qemu-ga): failure-propagating fsfreeze dispatcher; enabled on nh3-svos (hooks were never run) 2026-10-05 19:25:14 -07:00
vh 36289f701a ops(beszel): native agents + hub registration for nh3-svos and nh3-albok; nh3-svos app/ingress notes 2026-10-05 19:04:13 -07:00
vh d7e845eb64 feat(nh3-svos): VM 111 on nh3-pve-2 for the new SVOS (svos-dev, Prime); generic cloud-init VM playbook 2026-10-05 18:59:53 -07:00
vh 2c4c7ca9ae ops(esh-docker-vm): Homepage on https via a file-provider Traefik router; note the dead Route53 ACME resolver 2026-10-05 18:38:08 -07:00
vh 3429b5c55d ops(pfi-pteradactyl): docker.io/containerd upgraded (manual dockerd + wings restart); docs 2026-10-05 14:18:57 -07:00
vh 58bd802872 ops: docker engine 29.8.2 on esh-ml1, nh3-ml1, nh3-dev, corviduo-dev; docs 2026-10-05 14:15:12 -07:00
vh 5301ede9d2 ops(ana-docker): docker-ce from bookworm repo (29.8.2); live-restore stale-socket lesson; docs 2026-10-05 13:31:30 -07:00
vh 258a3eea68 docs: pfi-gx10 update recorded; held-engine list and Waiting-on-Prime refreshed 2026-10-05 11:34:00 -07:00
vh 06d7184158 ops: pfi-gx10 DGX OTA 7.6 + reboot, vm-esh-nas + ana-docker reboots (engine 29.8.0), crowdsec bouncer updateMaxFailure 3; docs 2026-10-05 11:33:43 -07:00
vh f6fe6f67f8 runbook(park 100): pfi-tacticalrmm rebooted + TRMM 1.5.2 (Prime) 2026-10-05 11:13:34 -07:00
vh 912d932917 runbook(park 100): 4 kernel reboots done (nh3-extdev, corviduo-dev, esh-vm-db, pfi-pteradactyl) 2026-10-05 11:09:17 -07:00
vh a808b349b8 runbook(park 100): pfi-tacticalrmm patched; reboot list now 8 hosts 2026-10-05 10:55:27 -07:00
vh aa86c37954 ops(pfi-pteradactyl): mask leftover apache2, patch (MariaDB 11.8), engine held; docs 2026-10-05 10:41:23 -07:00
vh 599b3e17f7 albok-service: fold 0.1.10 image pin from the live host 2026-10-05 10:24:41 -07:00
vh ac068dc2bd ops(pfi-tacticalrmm): *.phasefinal.com renews via dns-cloudflare (was manual; 16 days from expiry) 2026-10-05 08:43:15 -07:00
vh c9ace41340 feat(check-backup-freshness): attach the documented reason to each policy exclusion
The policy section said WHICH guests are uncovered but not WHY, so every
reader had to reconstruct the ruling from memory or the bus. A missing
reason now prints as 'NO DOCUMENTED REASON', which turns an undocumented
new exclusion into a visible flag instead of a silent line.
2026-10-05 08:03:47 -07:00
vh 67c72ed228 docs(ana-docker): crowdsec sqlite WAL fix for the bouncer fail-closed 403s 2026-10-05 04:19:52 -07:00
vh 241b14bc66 runbook(park 100): results — 15 patched, parks and deferrals; memory 2026-10-05 03:55:58 -07:00
vh 49c90f3083 docs(fv-ml1): drop the stale 'docker still on zroot' line 2026-10-05 02:43:31 -07:00
vh 23371954f8 docs(fv-ml1): docker on tank zvol, held packages, kernel metas removed; memory 2026-10-05 02:43:19 -07:00
vh fde50b7ca7 runbook(park 100): hold kernel metas on NVIDIA-DKMS hosts (fv-ml1 6.12.111 DKMS failure) 2026-10-05 02:42:41 -07:00
vh 67b91eff46 fix(skaldsong): group_add the host docker gid so the app can write its root:docker state dirs
The db/ and runs/ bind mounts are root:docker 2775 since the 2026-09-14 tree
normalisation; the app runs as 1000:1000, so SQLite could not create its journal
and every write returned 500 (found at v0.40.2).
2026-10-05 02:32:08 -07:00
vh af2f77a9c6 fix(oneoff): fv-ml1 docker move run 2 — one-filesystem rsync, skip overlay merged views, path-list verify, auto-restore before swap 2026-10-05 02:30:24 -07:00
vh 65d808642d albok-service: fold 0.1.9 (image pin, embedding_write on qwen3-embedding-bulk) from the live host 2026-10-05 02:17:18 -07:00
vh bf46beaa96 docs(runbooks): fleet guest patch campaign 2026-10-05 (park 100); fv-ml1 docker-to-tank one-off 2026-10-05 01:50:55 -07:00
vh 7346e4623c feat(comfyui): HF_HUB_CACHE=/basedir/models/foley-hf-cache for the Foley nodes
HunyuanVideo-Foley, MMAudio and ControlFoley load their Hugging Face dependencies
from pinned snapshots staged on the model store. Without this they look in the
container's ephemeral cache and download again on every recreate. Requested by
comfy-dev; HF_HUB_OFFLINE deliberately not set.
2026-10-05 01:37:21 -07:00
vh 7f34fe5a7c docs(irv-ml1, ana-docker): storage cleanup, text encoders on worktank, Traefik readTimeout 10m 2026-10-05 01:19:36 -07:00
vh 88c3d8a001 feat(skaldsong): pass SKALDSONG_COMPLETER_API_KEY to the container
Since skaldsong v0.39.0 the structured pipeline passes call the LiteLLM
gateway directly; with no key every story generation fails 401 at its first
pass. Operator ruling 2026-10-05: the shared all-agents key until Skaldsong
v1.0 final, then a project-scoped key. The stack passes an explicit
environment list, so the .env line alone does not reach the container.
2026-10-05 00:46:24 -07:00
vh afb9d3eef1 memory: irv-ml1 storetank 85% alert is the planned foley stage (200 GiB reserve guard) 2026-10-05 00:36:19 -07:00
vh c0d3384c68 feat(litellm): embedder primary/failover split — qwen3-embedding on esh-ml1, new qwen3-embedding-bulk on nh3-ml1
Prime's ruling 2026-10-05 (relayed by albok-dev, confirmed directly) replaces
the 2026-09-26 simple-shuffle load-share for embeddings only; the reranker
groups are unchanged. Routing uses LiteLLM per-deployment `order` (semantics
read at source in v1.97.0): lowest order among healthy deployments, and a
failed request retries at the next order.

Verified after the restart (43 s gateway outage): 6/6 calls per alias landed
on its primary. With nh3-ml1 tei-embed stopped, qwen3-embedding-bulk served 6/6
from esh-ml1 with 200s; once nh3-ml1 was back, bulk returned 6/6 to nh3-ml1.
The albok-service key now covers qwen3-embedding-bulk (200); the Nemi key does
not (403).
2026-10-05 00:32:42 -07:00
vh a7a068f56f albok: move closed — acceptance passed, old nh3-docker store deleted, first VM 110 backup taken 2026-10-05 00:21:15 -07:00
vh 94ca792a97 memory: albok old-store deletion approved by Prime, conditional on acceptance 2026-10-04 23:49:48 -07:00
vh d3f59aa2c5 feat(albok): move albok-service to a dedicated VM, nh3-albok (VM 110 on nh3-pve-2)
albok-service leaves nh3-docker before unit 4 puts the personal vault and
licensed books into restricted wings. On nh3-docker, portainer, portainer_agent,
dockge (socket rw) and dockerd on plaintext tcp/2375 were each root over
/srv/albok. The new host runs only albok-service: no socket containers and no
TCP Docker API, so its Homepage tile is a static services.yaml entry.

- nh3-pve-2: vmbr0 made VLAN-aware (applied by reboot, no guests running);
  pbs-ana storage in the new namespace nh3-pve-2, plus a 21:30 vzdump job
  (pbs-nh3's recursive sync mirrors it)
- VM 110 nh3-albok, 10.100.50.44 on VLAN 50, Debian 13 cloud-init, 4 vCPU
  x86-64-v3 / 12 GiB / 100 GB; built to fleet conventions (vh 1000,
  infra-ops 850, docker 851)
- compose: mem_limit 4g -> 8g, homepage labels removed; host port 8392 kept
- cutover by tar stream, verified by digest (contents plus owner, group, mode,
  mtime and type identical); albok.nh3.internal now points at nh3-albok
- old /srv/albok stays on nh3-docker (container removed, compose renamed)
  until acceptance and the operator's approval to delete it
- sudo-io archive hosts and ops-log STACK_HOSTS include nh3-albok
2026-10-04 23:33:24 -07:00
vh 867f33d672 docs(pfi): amend 10-04 bench doc per brokkr review
- Fit gap restated on nvidia-smi accounting (~17 MiB rest / ~130 MiB peak
  over, vs the live seat's 15,220/15,437 headroom); the 1.2 GiB figure
  conflated the torch pool fraction with nvidia-smi totals
- Footnote: baseline peak row is suite-only, not shape-matched to the
  candidate gate peaks
- namedef cell labeled as the description-reversal stand-in, not the
  loaded-name variant of arXiv 2609.26758; columns are overlaps
2026-10-04 23:12:41 -07:00
vh 523f086939 docs(pfi): Jev candidate bench 2026-10-04 — H2O-lightning-4b and Clef-flash vs Intern baseline
Vendor-path benches on fv-ml1 GPU 3 at the 14.4 GiB / 32k cap. Neither
candidate replaces Intern-Decision-4B: H2O reproduces its vendor figure
(205x3) and has the best calibration, but rests 1 GiB over the cap and
is 2.2x slower at the 21-criteria shape; Clef loses on accuracy (191-192),
hard tier (72-73/111), and ships a silent 16,384-token input slice.
Adds the namedef cell and JB_AUTH wiring to the 09-30 harness client.
2026-10-04 23:10:36 -07:00
vh f76429c457 memory: snapshot — 2026-10-04 2300 (PVE 9 + PBS 4 fleet-wide, webhost destroyed, phasefinal on Pages, sudo-io archiver; 7 settled entries archived) 2026-10-04 23:03:58 -07:00
vh d9d1dd9325 feat(sudo-io): stop recording bulk data (sudoers BULK_DATA !log_output fleet-wide), drop >64 MiB streams from archives, prune existing bulk recordings 2026-10-04 22:55:49 -07:00
vh 008bd685aa feat(sudo-io-archive): weekly fleet archive of sudo session recordings (pack, verify, then clear); fv-ml1 HF_HOME on tank and root-pool cleanup 2026-10-04 22:48:05 -07:00
vh 626e0c7f94 memory: dragonfireacoustics-site now on gitea; fv-ml1 unused vLLM images removed 2026-10-04 17:59:21 -07:00
vh 905862a58e ops(phasefinal-web): dedicated infra-ops-pages token; apex->www as a Cloudflare Redirect Rule 2026-10-04 17:37:32 -07:00
vh 513f6ef38f ops: phasefinal.com moved to Cloudflare Pages; pfi-ana-webhost destroyed and removed from inventory, DNS and the FortiGate 2026-10-04 17:28:29 -07:00
vh 55e4d01b49 ops(pfi-ana-webhost): retired (VM 110 stopped, FortiGate policy 18 disabled); dragonfireacoustics.com located on Cloudflare Pages 2026-10-04 16:57:49 -07:00
vh 0e3ab447af ops: retire VM 101 (DC) and VM 104 (nh3-laser) from running + backups; infra-ops login on pbs-ana/pbs-nh3; nh3-pve-2 follow-ups verified done 2026-10-04 16:37:15 -07:00