vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
1,612 Commits 2 Branches 5 Tags
454b689a54263a9aa21eef51cdc6273ddaac39ef
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
vh 454b689a54 fix(nh3-svos): strip Tailscale-User-* and X-Forwarded-* as whole families at the SVOS front door 
svos-dev review: the Caddyfile deleted named headers only. Prefix deletes cover the families; the named strays (Tailscale-Headers-Info, X-Real-IP, Forwarded, X-SVOS-Peer) stay. Wildcard semantics checked on a scratch Caddy (case-insensitive; look-alikes outside the family survive). Deployed, validated via systemd-run with EnvironmentFile, reloaded; deny tests re-run (403). README: safe validate recipe, pending acceptance list.
 2026-10-05 19:35:10 -07:00
vh ce398685c0 feat(nh3-svos): mesh-only front door for svos.phasefinal.com (Caddy + WhoIs fail-closed auth), headscale extra record 2026-10-05 19:31:41 -07:00
vh 63736bd40a vendor(qemu-ga): svos-dev's nh3-svos quiesce hook + its 6 controls beside the dispatcher 2026-10-05 19:28:02 -07:00
vh 9175fbd539 feat(qemu-ga): failure-propagating fsfreeze dispatcher; enabled on nh3-svos (hooks were never run) 2026-10-05 19:25:14 -07:00
vh 36289f701a ops(beszel): native agents + hub registration for nh3-svos and nh3-albok; nh3-svos app/ingress notes 2026-10-05 19:04:13 -07:00
vh d7e845eb64 feat(nh3-svos): VM 111 on nh3-pve-2 for the new SVOS (svos-dev, Prime); generic cloud-init VM playbook 2026-10-05 18:59:53 -07:00
vh 2c4c7ca9ae ops(esh-docker-vm): Homepage on https via a file-provider Traefik router; note the dead Route53 ACME resolver 2026-10-05 18:38:08 -07:00
vh 3429b5c55d ops(pfi-pteradactyl): docker.io/containerd upgraded (manual dockerd + wings restart); docs 2026-10-05 14:18:57 -07:00
vh 58bd802872 ops: docker engine 29.8.2 on esh-ml1, nh3-ml1, nh3-dev, corviduo-dev; docs 2026-10-05 14:15:12 -07:00
vh 5301ede9d2 ops(ana-docker): docker-ce from bookworm repo (29.8.2); live-restore stale-socket lesson; docs 2026-10-05 13:31:30 -07:00
vh 258a3eea68 docs: pfi-gx10 update recorded; held-engine list and Waiting-on-Prime refreshed 2026-10-05 11:34:00 -07:00
vh 06d7184158 ops: pfi-gx10 DGX OTA 7.6 + reboot, vm-esh-nas + ana-docker reboots (engine 29.8.0), crowdsec bouncer updateMaxFailure 3; docs 2026-10-05 11:33:43 -07:00
vh f6fe6f67f8 runbook(park 100): pfi-tacticalrmm rebooted + TRMM 1.5.2 (Prime) 2026-10-05 11:13:34 -07:00
vh 912d932917 runbook(park 100): 4 kernel reboots done (nh3-extdev, corviduo-dev, esh-vm-db, pfi-pteradactyl) 2026-10-05 11:09:17 -07:00
vh a808b349b8 runbook(park 100): pfi-tacticalrmm patched; reboot list now 8 hosts 2026-10-05 10:55:27 -07:00
vh aa86c37954 ops(pfi-pteradactyl): mask leftover apache2, patch (MariaDB 11.8), engine held; docs 2026-10-05 10:41:23 -07:00
vh 599b3e17f7 albok-service: fold 0.1.10 image pin from the live host 2026-10-05 10:24:41 -07:00
vh ac068dc2bd ops(pfi-tacticalrmm): *.phasefinal.com renews via dns-cloudflare (was manual; 16 days from expiry) 2026-10-05 08:43:15 -07:00
vh c9ace41340 feat(check-backup-freshness): attach the documented reason to each policy exclusion 
The policy section said WHICH guests are uncovered but not WHY, so every
reader had to reconstruct the ruling from memory or the bus. A missing
reason now prints as 'NO DOCUMENTED REASON', which turns an undocumented
new exclusion into a visible flag instead of a silent line.
 2026-10-05 08:03:47 -07:00
vh 67c72ed228 docs(ana-docker): crowdsec sqlite WAL fix for the bouncer fail-closed 403s 2026-10-05 04:19:52 -07:00
vh 241b14bc66 runbook(park 100): results — 15 patched, parks and deferrals; memory 2026-10-05 03:55:58 -07:00
vh 49c90f3083 docs(fv-ml1): drop the stale 'docker still on zroot' line 2026-10-05 02:43:31 -07:00
vh 23371954f8 docs(fv-ml1): docker on tank zvol, held packages, kernel metas removed; memory 2026-10-05 02:43:19 -07:00
vh fde50b7ca7 runbook(park 100): hold kernel metas on NVIDIA-DKMS hosts (fv-ml1 6.12.111 DKMS failure) 2026-10-05 02:42:41 -07:00
vh 67b91eff46 fix(skaldsong): group_add the host docker gid so the app can write its root:docker state dirs 
The db/ and runs/ bind mounts are root:docker 2775 since the 2026-09-14 tree
normalisation; the app runs as 1000:1000, so SQLite could not create its journal
and every write returned 500 (found at v0.40.2).
 2026-10-05 02:32:08 -07:00
vh af2f77a9c6 fix(oneoff): fv-ml1 docker move run 2 — one-filesystem rsync, skip overlay merged views, path-list verify, auto-restore before swap 2026-10-05 02:30:24 -07:00
vh 65d808642d albok-service: fold 0.1.9 (image pin, embedding_write on qwen3-embedding-bulk) from the live host 2026-10-05 02:17:18 -07:00
vh bf46beaa96 docs(runbooks): fleet guest patch campaign 2026-10-05 (park 100); fv-ml1 docker-to-tank one-off 2026-10-05 01:50:55 -07:00
vh 7346e4623c feat(comfyui): HF_HUB_CACHE=/basedir/models/foley-hf-cache for the Foley nodes 
HunyuanVideo-Foley, MMAudio and ControlFoley load their Hugging Face dependencies
from pinned snapshots staged on the model store. Without this they look in the
container's ephemeral cache and download again on every recreate. Requested by
comfy-dev; HF_HUB_OFFLINE deliberately not set.
 2026-10-05 01:37:21 -07:00
vh 7f34fe5a7c docs(irv-ml1, ana-docker): storage cleanup, text encoders on worktank, Traefik readTimeout 10m 2026-10-05 01:19:36 -07:00
vh 88c3d8a001 feat(skaldsong): pass SKALDSONG_COMPLETER_API_KEY to the container 
Since skaldsong v0.39.0 the structured pipeline passes call the LiteLLM
gateway directly; with no key every story generation fails 401 at its first
pass. Operator ruling 2026-10-05: the shared all-agents key until Skaldsong
v1.0 final, then a project-scoped key. The stack passes an explicit
environment list, so the .env line alone does not reach the container.
 2026-10-05 00:46:24 -07:00
vh afb9d3eef1 memory: irv-ml1 storetank 85% alert is the planned foley stage (200 GiB reserve guard) 2026-10-05 00:36:19 -07:00
vh c0d3384c68 feat(litellm): embedder primary/failover split — qwen3-embedding on esh-ml1, new qwen3-embedding-bulk on nh3-ml1 
Prime's ruling 2026-10-05 (relayed by albok-dev, confirmed directly) replaces
the 2026-09-26 simple-shuffle load-share for embeddings only; the reranker
groups are unchanged. Routing uses LiteLLM per-deployment `order` (semantics
read at source in v1.97.0): lowest order among healthy deployments, and a
failed request retries at the next order.

Verified after the restart (43 s gateway outage): 6/6 calls per alias landed
on its primary. With nh3-ml1 tei-embed stopped, qwen3-embedding-bulk served 6/6
from esh-ml1 with 200s; once nh3-ml1 was back, bulk returned 6/6 to nh3-ml1.
The albok-service key now covers qwen3-embedding-bulk (200); the Nemi key does
not (403).
 2026-10-05 00:32:42 -07:00
vh a7a068f56f albok: move closed — acceptance passed, old nh3-docker store deleted, first VM 110 backup taken 2026-10-05 00:21:15 -07:00
vh 94ca792a97 memory: albok old-store deletion approved by Prime, conditional on acceptance 2026-10-04 23:49:48 -07:00
vh d3f59aa2c5 feat(albok): move albok-service to a dedicated VM, nh3-albok (VM 110 on nh3-pve-2) 
albok-service leaves nh3-docker before unit 4 puts the personal vault and
licensed books into restricted wings. On nh3-docker, portainer, portainer_agent,
dockge (socket rw) and dockerd on plaintext tcp/2375 were each root over
/srv/albok. The new host runs only albok-service: no socket containers and no
TCP Docker API, so its Homepage tile is a static services.yaml entry.

- nh3-pve-2: vmbr0 made VLAN-aware (applied by reboot, no guests running);
  pbs-ana storage in the new namespace nh3-pve-2, plus a 21:30 vzdump job
  (pbs-nh3's recursive sync mirrors it)
- VM 110 nh3-albok, 10.100.50.44 on VLAN 50, Debian 13 cloud-init, 4 vCPU
  x86-64-v3 / 12 GiB / 100 GB; built to fleet conventions (vh 1000,
  infra-ops 850, docker 851)
- compose: mem_limit 4g -> 8g, homepage labels removed; host port 8392 kept
- cutover by tar stream, verified by digest (contents plus owner, group, mode,
  mtime and type identical); albok.nh3.internal now points at nh3-albok
- old /srv/albok stays on nh3-docker (container removed, compose renamed)
  until acceptance and the operator's approval to delete it
- sudo-io archive hosts and ops-log STACK_HOSTS include nh3-albok
 2026-10-04 23:33:24 -07:00
vh 867f33d672 docs(pfi): amend 10-04 bench doc per brokkr review 
- Fit gap restated on nvidia-smi accounting (~17 MiB rest / ~130 MiB peak
  over, vs the live seat's 15,220/15,437 headroom); the 1.2 GiB figure
  conflated the torch pool fraction with nvidia-smi totals
- Footnote: baseline peak row is suite-only, not shape-matched to the
  candidate gate peaks
- namedef cell labeled as the description-reversal stand-in, not the
  loaded-name variant of arXiv 2609.26758; columns are overlaps
 2026-10-04 23:12:41 -07:00
vh 523f086939 docs(pfi): Jev candidate bench 2026-10-04 — H2O-lightning-4b and Clef-flash vs Intern baseline 
Vendor-path benches on fv-ml1 GPU 3 at the 14.4 GiB / 32k cap. Neither
candidate replaces Intern-Decision-4B: H2O reproduces its vendor figure
(205x3) and has the best calibration, but rests 1 GiB over the cap and
is 2.2x slower at the 21-criteria shape; Clef loses on accuracy (191-192),
hard tier (72-73/111), and ships a silent 16,384-token input slice.
Adds the namedef cell and JB_AUTH wiring to the 09-30 harness client.
 2026-10-04 23:10:36 -07:00
vh f76429c457 memory: snapshot — 2026-10-04 2300 (PVE 9 + PBS 4 fleet-wide, webhost destroyed, phasefinal on Pages, sudo-io archiver; 7 settled entries archived) 2026-10-04 23:03:58 -07:00
vh d9d1dd9325 feat(sudo-io): stop recording bulk data (sudoers BULK_DATA !log_output fleet-wide), drop >64 MiB streams from archives, prune existing bulk recordings 2026-10-04 22:55:49 -07:00
vh 008bd685aa feat(sudo-io-archive): weekly fleet archive of sudo session recordings (pack, verify, then clear); fv-ml1 HF_HOME on tank and root-pool cleanup 2026-10-04 22:48:05 -07:00
vh 626e0c7f94 memory: dragonfireacoustics-site now on gitea; fv-ml1 unused vLLM images removed 2026-10-04 17:59:21 -07:00
vh 905862a58e ops(phasefinal-web): dedicated infra-ops-pages token; apex->www as a Cloudflare Redirect Rule 2026-10-04 17:37:32 -07:00
vh 513f6ef38f ops: phasefinal.com moved to Cloudflare Pages; pfi-ana-webhost destroyed and removed from inventory, DNS and the FortiGate 2026-10-04 17:28:29 -07:00
vh 55e4d01b49 ops(pfi-ana-webhost): retired (VM 110 stopped, FortiGate policy 18 disabled); dragonfireacoustics.com located on Cloudflare Pages 2026-10-04 16:57:49 -07:00
vh 0e3ab447af ops: retire VM 101 (DC) and VM 104 (nh3-laser) from running + backups; infra-ops login on pbs-ana/pbs-nh3; nh3-pve-2 follow-ups verified done 2026-10-04 16:37:15 -07:00
vh d0c1bc628a ops(pbs): pbs-nh3 + pbs-ana upgraded to PBS 4.2.7 / Debian 13; guest-agent driver script, plan and log 2026-10-04 11:44:57 -07:00
vh 4d4851acd2 docs(albok-service): fleet/memory reindex result on 0.1.8; reindex path shape note 2026-10-04 11:44:13 -07:00
vh fce4e1a583 chore(albok-service): 0.1.8 on nh3-docker (e653c29; fleet/memory reindex on new tag) 2026-10-04 11:16:24 -07:00
vh e8faa74512 docs(albok-service): fleet/memory reindex stood down pending 0.1.8 2026-10-04 08:18:42 -07:00
vh ebe0165acf chore(albok-service): 0.1.7 on nh3-docker (clean-archive build of 64f75dc) 2026-10-04 08:17:33 -07:00
vh 73129a0564 ops(esh-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20 (one reboot, Prime on site); ESH cluster uniform on 9 2026-10-04 07:45:05 -07:00
vh 9a6b23581e ops(esh-pve): PVE 9 prep done (8.4.21, microcode, systemd-boot out, removable shim, NIC pins, vmbr0 bridge-ports fixed) 2026-10-04 07:32:13 -07:00
vh b8535dfb52 homepage: add NH3-PVE-2 and its AMT (MeshCentral) cards 2026-10-04 07:23:28 -07:00
vh b67ab48833 ops(pfi-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20; reboot test + post-check recorded; pre-reboot checks corrected for PVE 9 2026-10-04 07:16:21 -07:00
vh 075ba0930e ops(pfi-pve): PVE 9 prep done (8.4.21, grub-efi-amd64 on the UEFI/LVM box, microcode, NIC pins); upgrade plan + playbooks 2026-10-04 06:11:13 -07:00
vh a4b7626a13 memory: snapshot — 2026-10-04 ~0546 (albok 0.1.6; 8 settled 09-19 entries archived) 2026-10-04 05:46:09 -07:00
vh 2449e8806f fix(albok-service): 0.1.6 on nh3-docker (review fold: locked index build) 2026-10-04 05:45:25 -07:00
vh 957cd80d5e docs(albok-service): 0.1.5 measured live (fleet/memory 3.97 s -> 0.22 s) 2026-10-03 22:49:53 -07:00
vh 53b3428d13 memory: snapshot — 2026-10-03 ~2045 (esh-nas-pve PVE 9, albok 0.1.5, QNAP LCD, open calls) 2026-10-03 20:40:50 -07:00
vh 0ed188be5e perf(albok-service): 0.1.5 on nh3-docker (search 4.5 s -> ~0.25 s) 2026-10-03 20:40:11 -07:00
vh d4c5ae8d00 memory: NH3 Mac mini asleep; fix tomorrow on site 2026-10-03 16:23:15 -07:00
vh 453c927ace feat(qnap-lcd): Proxmox-correct front-panel LCD menu for esh-nas-pve; albok redeploy note 2026-10-03 16:14:43 -07:00
vh f696aba10a fix(albok-service): 0.1.4 on nh3-docker (version header from metadata) 2026-10-03 16:12:01 -07:00
vh 9ba8962885 ops(esh-nas-pve): upgraded to PVE 9.2.21; /boot-LV-full trap and fixes recorded 2026-10-03 16:06:33 -07:00
vh ca18b04a22 fix(albok-service): 0.1.3 on nh3-docker (ingester stamp out of equivalence) 2026-10-03 15:59:10 -07:00
vh 241a668f3c ops(esh-nas-pve): PVE 9 upgrade playbook (snapshot, trixie sources, detached dist-upgrade) 2026-10-03 15:52:44 -07:00
vh 473c52dc6b docs(pve9 plan): keep ESH DNS up during the NAS reboot (unmount 2 shares instead of stopping VM 100) 2026-10-03 15:50:31 -07:00
vh 48f28cfe93 ops(esh-nas-pve): PVE 8 reboot test done; two restart foot-guns recorded 2026-10-03 15:46:37 -07:00
vh f648e18b1a ops(esh-nas-pve): PVE 9 no-downtime prep (8.4.21, microcode, systemd-boot out, NIC pins) 2026-10-03 15:31:00 -07:00
vh 5f8a1311a5 memory: snapshot — 2026-10-03 ~1440 (nh3-pve clean on 9, RustDesk 1.1.16, esh-nas-pve PVE 9 assessed) 2026-10-03 15:23:19 -07:00
vh d4675f2cb5 ops(ana-docker): rustdesk-server 1.1.14 -> 1.1.16; server key vaulted 2026-10-03 14:24:32 -07:00
vh 5e25d4bae1 docs(pve9 plan): headers must be present before the upgrade (nh3-pve lesson) 2026-10-03 14:09:47 -07:00
vh d9e09fed14 ops(nh3-pve): PVE 9 post-upgrade fixes (7.0 headers + nvidia DKMS, microcode, enterprise repo off); old VM102 snapshot removed 2026-10-03 14:09:41 -07:00
vh 08c3271084 memory: nh3-pve on PVE 9; post-check findings (nh3-ml1 down: no 7.0 headers) 2026-10-03 14:06:01 -07:00
vh 5abd43e1c8 memory: snapshot — 2026-10-03 in-flight (nh3-pve 8→9 under way), two-tier split of 27 entries, 7 archived 2026-10-03 13:36:42 -07:00
vh 4b2a81bf39 feat(pve-nag-patch): remove the subscription popup on the PVE hosts, re-applied by an apt hook 2026-10-03 13:27:11 -07:00
vh 22224b5335 ops(nh3-pve-2): register host, network/AMT/disk layout and the re-IP lesson 2026-10-03 13:17:46 -07:00
vh 9b1cfa7344 feat(nh3-pve-2): drop stale 1 TB boot entry, wipe it into LVM-thin storage vmstore 2026-10-03 13:17:28 -07:00
vh 00a6fbd849 feat(cira-tunnel-watchdog): auto-drop stale AMT CIRA tunnels on MeshCentral's MPS 2026-10-03 12:34:35 -07:00
vh fe1d472aae dns: nh3-pve-2 -> 10.100.250.62 (10G port, nh3-mgmt reservation) 2026-10-03 12:32:44 -07:00
vh e4fcd1398d docs(meshcentral): stale CIRA tunnel, third occurrence (nh3-pve-2) 2026-10-03 12:19:42 -07:00
vh 73da3175e2 docs(albok-service): pushed tag verified to match the deployed image 2026-10-03 10:00:03 -07:00
vh f8290f64bf memory: worldtree gate duty reframed as post-deletion regression check (infra-ops 01M413W4) 2026-10-03 07:50:47 -07:00
vh 9b71f6c2f3 memory: first post-deletion Worldtree gate batch PASS 2026-10-03 07:49:44 -07:00
vh 3a6d997c99 ops(esh-nas-pve): tank verified healthy after two scrubs; disk kept; PVE 9 blocker cleared 2026-10-03 06:28:48 -07:00
vh 6a274a412a docs(albok-nemi): nemi now prunes reports and takes a run lock 2026-10-03 06:00:23 -07:00
vh 70fa0fbd53 memory: Nemi timer installed 2026-10-03 05:58:12 -07:00
vh e887c6a4bc feat(albok-nemi): hourly Nemi walk timer on nh3-dev (albok-dev ask), with OnFailure hook 2026-10-03 05:57:54 -07:00
vh 8a4dbf2341 fix(albok-service): 0.1.2 on nh3-docker (fd leak); add restricted wing personal/agent-feedback; Nemi credentials 2026-10-03 02:54:18 -07:00
vh 1fb36b32d2 memory: ESH tank first scrub done, second scrub running 2026-10-03 01:18:24 -07:00
vh 3a91eff2e6 memory: ESH tank scrub repairing the Aug 20 replacement's unrepaired blocks 2026-10-02 23:33:07 -07:00
vh 36c52d8860 docs(meshcentral): stale CIRA tunnel reproduced on a single shutdown 2026-10-02 22:53:30 -07:00
vh ad0322bf83 ops(meshcentral): diagnose and fix HW Connect stuck at Setup (stale CIRA tunnel); add relay probe 2026-10-02 22:49:27 -07:00
vh 55004090d1 ops(esh-pve-2): register host, AMT phoning home to MeshCentral; note MeshCentral first-CIRA crash race 2026-10-02 22:25:37 -07:00
vh e707d87713 feat(esh-pve-2): drop stale 1 TB boot entry, wipe it into LVM-thin storage vmstore 2026-10-02 22:08:19 -07:00
vh 6315ce38d5 memory: ESH tank verified residual, cleared, full scrub running 2026-10-02 19:26:30 -07:00
vh de143364fd docs: fleet Proxmox host inventory snapshot 2026-10-02 (all sites) 2026-10-02 19:23:58 -07:00
vh 62254023f3 docs: esh-pve-cluster PVE 8->9 upgrade plan (not executed); ESH tank DEGRADED finding 2026-10-02 19:20:28 -07:00
vh 5439d5fdc1 albok-service: 0.1.1 (health fix) on nh3-docker 2026-10-02 18:04:57 -07:00