Commit Graph
100 Commits
Author SHA1 Message Date
vh 7346e4623c feat(comfyui): HF_HUB_CACHE=/basedir/models/foley-hf-cache for the Foley nodes
HunyuanVideo-Foley, MMAudio and ControlFoley load their Hugging Face dependencies
from pinned snapshots staged on the model store. Without this they look in the
container's ephemeral cache and download again on every recreate. Requested by
comfy-dev; HF_HUB_OFFLINE deliberately not set.
2026-10-05 01:37:21 -07:00
vh 7f34fe5a7c docs(irv-ml1, ana-docker): storage cleanup, text encoders on worktank, Traefik readTimeout 10m 2026-10-05 01:19:36 -07:00
vh 88c3d8a001 feat(skaldsong): pass SKALDSONG_COMPLETER_API_KEY to the container
Since skaldsong v0.39.0 the structured pipeline passes call the LiteLLM
gateway directly; with no key every story generation fails 401 at its first
pass. Operator ruling 2026-10-05: the shared all-agents key until Skaldsong
v1.0 final, then a project-scoped key. The stack passes an explicit
environment list, so the .env line alone does not reach the container.
2026-10-05 00:46:24 -07:00
vh afb9d3eef1 memory: irv-ml1 storetank 85% alert is the planned foley stage (200 GiB reserve guard) 2026-10-05 00:36:19 -07:00
vh c0d3384c68 feat(litellm): embedder primary/failover split — qwen3-embedding on esh-ml1, new qwen3-embedding-bulk on nh3-ml1
Prime's ruling 2026-10-05 (relayed by albok-dev, confirmed directly) replaces
the 2026-09-26 simple-shuffle load-share for embeddings only; the reranker
groups are unchanged. Routing uses LiteLLM per-deployment `order` (semantics
read at source in v1.97.0): lowest order among healthy deployments, and a
failed request retries at the next order.

Verified after the restart (43 s gateway outage): 6/6 calls per alias landed
on its primary. With nh3-ml1 tei-embed stopped, qwen3-embedding-bulk served 6/6
from esh-ml1 with 200s; once nh3-ml1 was back, bulk returned 6/6 to nh3-ml1.
The albok-service key now covers qwen3-embedding-bulk (200); the Nemi key does
not (403).
2026-10-05 00:32:42 -07:00
vh a7a068f56f albok: move closed — acceptance passed, old nh3-docker store deleted, first VM 110 backup taken 2026-10-05 00:21:15 -07:00
vh 94ca792a97 memory: albok old-store deletion approved by Prime, conditional on acceptance 2026-10-04 23:49:48 -07:00
vh d3f59aa2c5 feat(albok): move albok-service to a dedicated VM, nh3-albok (VM 110 on nh3-pve-2)
albok-service leaves nh3-docker before unit 4 puts the personal vault and
licensed books into restricted wings. On nh3-docker, portainer, portainer_agent,
dockge (socket rw) and dockerd on plaintext tcp/2375 were each root over
/srv/albok. The new host runs only albok-service: no socket containers and no
TCP Docker API, so its Homepage tile is a static services.yaml entry.

- nh3-pve-2: vmbr0 made VLAN-aware (applied by reboot, no guests running);
  pbs-ana storage in the new namespace nh3-pve-2, plus a 21:30 vzdump job
  (pbs-nh3's recursive sync mirrors it)
- VM 110 nh3-albok, 10.100.50.44 on VLAN 50, Debian 13 cloud-init, 4 vCPU
  x86-64-v3 / 12 GiB / 100 GB; built to fleet conventions (vh 1000,
  infra-ops 850, docker 851)
- compose: mem_limit 4g -> 8g, homepage labels removed; host port 8392 kept
- cutover by tar stream, verified by digest (contents plus owner, group, mode,
  mtime and type identical); albok.nh3.internal now points at nh3-albok
- old /srv/albok stays on nh3-docker (container removed, compose renamed)
  until acceptance and the operator's approval to delete it
- sudo-io archive hosts and ops-log STACK_HOSTS include nh3-albok
2026-10-04 23:33:24 -07:00
vh 867f33d672 docs(pfi): amend 10-04 bench doc per brokkr review
- Fit gap restated on nvidia-smi accounting (~17 MiB rest / ~130 MiB peak
  over, vs the live seat's 15,220/15,437 headroom); the 1.2 GiB figure
  conflated the torch pool fraction with nvidia-smi totals
- Footnote: baseline peak row is suite-only, not shape-matched to the
  candidate gate peaks
- namedef cell labeled as the description-reversal stand-in, not the
  loaded-name variant of arXiv 2609.26758; columns are overlaps
2026-10-04 23:12:41 -07:00
vh 523f086939 docs(pfi): Jev candidate bench 2026-10-04 — H2O-lightning-4b and Clef-flash vs Intern baseline
Vendor-path benches on fv-ml1 GPU 3 at the 14.4 GiB / 32k cap. Neither
candidate replaces Intern-Decision-4B: H2O reproduces its vendor figure
(205x3) and has the best calibration, but rests 1 GiB over the cap and
is 2.2x slower at the 21-criteria shape; Clef loses on accuracy (191-192),
hard tier (72-73/111), and ships a silent 16,384-token input slice.
Adds the namedef cell and JB_AUTH wiring to the 09-30 harness client.
2026-10-04 23:10:36 -07:00
vh f76429c457 memory: snapshot — 2026-10-04 2300 (PVE 9 + PBS 4 fleet-wide, webhost destroyed, phasefinal on Pages, sudo-io archiver; 7 settled entries archived) 2026-10-04 23:03:58 -07:00
vh d9d1dd9325 feat(sudo-io): stop recording bulk data (sudoers BULK_DATA !log_output fleet-wide), drop >64 MiB streams from archives, prune existing bulk recordings 2026-10-04 22:55:49 -07:00
vh 008bd685aa feat(sudo-io-archive): weekly fleet archive of sudo session recordings (pack, verify, then clear); fv-ml1 HF_HOME on tank and root-pool cleanup 2026-10-04 22:48:05 -07:00
vh 626e0c7f94 memory: dragonfireacoustics-site now on gitea; fv-ml1 unused vLLM images removed 2026-10-04 17:59:21 -07:00
vh 905862a58e ops(phasefinal-web): dedicated infra-ops-pages token; apex->www as a Cloudflare Redirect Rule 2026-10-04 17:37:32 -07:00
vh 513f6ef38f ops: phasefinal.com moved to Cloudflare Pages; pfi-ana-webhost destroyed and removed from inventory, DNS and the FortiGate 2026-10-04 17:28:29 -07:00
vh 55e4d01b49 ops(pfi-ana-webhost): retired (VM 110 stopped, FortiGate policy 18 disabled); dragonfireacoustics.com located on Cloudflare Pages 2026-10-04 16:57:49 -07:00
vh 0e3ab447af ops: retire VM 101 (DC) and VM 104 (nh3-laser) from running + backups; infra-ops login on pbs-ana/pbs-nh3; nh3-pve-2 follow-ups verified done 2026-10-04 16:37:15 -07:00
vh d0c1bc628a ops(pbs): pbs-nh3 + pbs-ana upgraded to PBS 4.2.7 / Debian 13; guest-agent driver script, plan and log 2026-10-04 11:44:57 -07:00
vh 4d4851acd2 docs(albok-service): fleet/memory reindex result on 0.1.8; reindex path shape note 2026-10-04 11:44:13 -07:00
vh fce4e1a583 chore(albok-service): 0.1.8 on nh3-docker (e653c29; fleet/memory reindex on new tag) 2026-10-04 11:16:24 -07:00
vh e8faa74512 docs(albok-service): fleet/memory reindex stood down pending 0.1.8 2026-10-04 08:18:42 -07:00
vh ebe0165acf chore(albok-service): 0.1.7 on nh3-docker (clean-archive build of 64f75dc) 2026-10-04 08:17:33 -07:00
vh 73129a0564 ops(esh-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20 (one reboot, Prime on site); ESH cluster uniform on 9 2026-10-04 07:45:05 -07:00
vh 9a6b23581e ops(esh-pve): PVE 9 prep done (8.4.21, microcode, systemd-boot out, removable shim, NIC pins, vmbr0 bridge-ports fixed) 2026-10-04 07:32:13 -07:00
vh b8535dfb52 homepage: add NH3-PVE-2 and its AMT (MeshCentral) cards 2026-10-04 07:23:28 -07:00
vh b67ab48833 ops(pfi-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20; reboot test + post-check recorded; pre-reboot checks corrected for PVE 9 2026-10-04 07:16:21 -07:00
vh 075ba0930e ops(pfi-pve): PVE 9 prep done (8.4.21, grub-efi-amd64 on the UEFI/LVM box, microcode, NIC pins); upgrade plan + playbooks 2026-10-04 06:11:13 -07:00
vh a4b7626a13 memory: snapshot — 2026-10-04 ~0546 (albok 0.1.6; 8 settled 09-19 entries archived) 2026-10-04 05:46:09 -07:00
vh 2449e8806f fix(albok-service): 0.1.6 on nh3-docker (review fold: locked index build) 2026-10-04 05:45:25 -07:00
vh 957cd80d5e docs(albok-service): 0.1.5 measured live (fleet/memory 3.97 s -> 0.22 s) 2026-10-03 22:49:53 -07:00
vh 53b3428d13 memory: snapshot — 2026-10-03 ~2045 (esh-nas-pve PVE 9, albok 0.1.5, QNAP LCD, open calls) 2026-10-03 20:40:50 -07:00
vh 0ed188be5e perf(albok-service): 0.1.5 on nh3-docker (search 4.5 s -> ~0.25 s) 2026-10-03 20:40:11 -07:00
vh d4c5ae8d00 memory: NH3 Mac mini asleep; fix tomorrow on site 2026-10-03 16:23:15 -07:00
vh 453c927ace feat(qnap-lcd): Proxmox-correct front-panel LCD menu for esh-nas-pve; albok redeploy note 2026-10-03 16:14:43 -07:00
vh f696aba10a fix(albok-service): 0.1.4 on nh3-docker (version header from metadata) 2026-10-03 16:12:01 -07:00
vh 9ba8962885 ops(esh-nas-pve): upgraded to PVE 9.2.21; /boot-LV-full trap and fixes recorded 2026-10-03 16:06:33 -07:00
vh ca18b04a22 fix(albok-service): 0.1.3 on nh3-docker (ingester stamp out of equivalence) 2026-10-03 15:59:10 -07:00
vh 241a668f3c ops(esh-nas-pve): PVE 9 upgrade playbook (snapshot, trixie sources, detached dist-upgrade) 2026-10-03 15:52:44 -07:00
vh 473c52dc6b docs(pve9 plan): keep ESH DNS up during the NAS reboot (unmount 2 shares instead of stopping VM 100) 2026-10-03 15:50:31 -07:00
vh 48f28cfe93 ops(esh-nas-pve): PVE 8 reboot test done; two restart foot-guns recorded 2026-10-03 15:46:37 -07:00
vh f648e18b1a ops(esh-nas-pve): PVE 9 no-downtime prep (8.4.21, microcode, systemd-boot out, NIC pins) 2026-10-03 15:31:00 -07:00
vh 5f8a1311a5 memory: snapshot — 2026-10-03 ~1440 (nh3-pve clean on 9, RustDesk 1.1.16, esh-nas-pve PVE 9 assessed) 2026-10-03 15:23:19 -07:00
vh d4675f2cb5 ops(ana-docker): rustdesk-server 1.1.14 -> 1.1.16; server key vaulted 2026-10-03 14:24:32 -07:00
vh 5e25d4bae1 docs(pve9 plan): headers must be present before the upgrade (nh3-pve lesson) 2026-10-03 14:09:47 -07:00
vh d9e09fed14 ops(nh3-pve): PVE 9 post-upgrade fixes (7.0 headers + nvidia DKMS, microcode, enterprise repo off); old VM102 snapshot removed 2026-10-03 14:09:41 -07:00
vh 08c3271084 memory: nh3-pve on PVE 9; post-check findings (nh3-ml1 down: no 7.0 headers) 2026-10-03 14:06:01 -07:00
vh 5abd43e1c8 memory: snapshot — 2026-10-03 in-flight (nh3-pve 8→9 under way), two-tier split of 27 entries, 7 archived 2026-10-03 13:36:42 -07:00
vh 4b2a81bf39 feat(pve-nag-patch): remove the subscription popup on the PVE hosts, re-applied by an apt hook 2026-10-03 13:27:11 -07:00
vh 22224b5335 ops(nh3-pve-2): register host, network/AMT/disk layout and the re-IP lesson 2026-10-03 13:17:46 -07:00
vh 9b1cfa7344 feat(nh3-pve-2): drop stale 1 TB boot entry, wipe it into LVM-thin storage vmstore 2026-10-03 13:17:28 -07:00
vh 00a6fbd849 feat(cira-tunnel-watchdog): auto-drop stale AMT CIRA tunnels on MeshCentral's MPS 2026-10-03 12:34:35 -07:00
vh fe1d472aae dns: nh3-pve-2 -> 10.100.250.62 (10G port, nh3-mgmt reservation) 2026-10-03 12:32:44 -07:00
vh e4fcd1398d docs(meshcentral): stale CIRA tunnel, third occurrence (nh3-pve-2) 2026-10-03 12:19:42 -07:00
vh 73da3175e2 docs(albok-service): pushed tag verified to match the deployed image 2026-10-03 10:00:03 -07:00
vh f8290f64bf memory: worldtree gate duty reframed as post-deletion regression check (infra-ops 01M413W4) 2026-10-03 07:50:47 -07:00
vh 9b71f6c2f3 memory: first post-deletion Worldtree gate batch PASS 2026-10-03 07:49:44 -07:00
vh 3a6d997c99 ops(esh-nas-pve): tank verified healthy after two scrubs; disk kept; PVE 9 blocker cleared 2026-10-03 06:28:48 -07:00
vh 6a274a412a docs(albok-nemi): nemi now prunes reports and takes a run lock 2026-10-03 06:00:23 -07:00
vh 70fa0fbd53 memory: Nemi timer installed 2026-10-03 05:58:12 -07:00
vh e887c6a4bc feat(albok-nemi): hourly Nemi walk timer on nh3-dev (albok-dev ask), with OnFailure hook 2026-10-03 05:57:54 -07:00
vh 8a4dbf2341 fix(albok-service): 0.1.2 on nh3-docker (fd leak); add restricted wing personal/agent-feedback; Nemi credentials 2026-10-03 02:54:18 -07:00
vh 1fb36b32d2 memory: ESH tank first scrub done, second scrub running 2026-10-03 01:18:24 -07:00
vh 3a91eff2e6 memory: ESH tank scrub repairing the Aug 20 replacement's unrepaired blocks 2026-10-02 23:33:07 -07:00
vh 36c52d8860 docs(meshcentral): stale CIRA tunnel reproduced on a single shutdown 2026-10-02 22:53:30 -07:00
vh ad0322bf83 ops(meshcentral): diagnose and fix HW Connect stuck at Setup (stale CIRA tunnel); add relay probe 2026-10-02 22:49:27 -07:00
vh 55004090d1 ops(esh-pve-2): register host, AMT phoning home to MeshCentral; note MeshCentral first-CIRA crash race 2026-10-02 22:25:37 -07:00
vh e707d87713 feat(esh-pve-2): drop stale 1 TB boot entry, wipe it into LVM-thin storage vmstore 2026-10-02 22:08:19 -07:00
vh 6315ce38d5 memory: ESH tank verified residual, cleared, full scrub running 2026-10-02 19:26:30 -07:00
vh de143364fd docs: fleet Proxmox host inventory snapshot 2026-10-02 (all sites) 2026-10-02 19:23:58 -07:00
vh 62254023f3 docs: esh-pve-cluster PVE 8->9 upgrade plan (not executed); ESH tank DEGRADED finding 2026-10-02 19:20:28 -07:00
vh 5439d5fdc1 albok-service: 0.1.1 (health fix) on nh3-docker 2026-10-02 18:04:57 -07:00
vh d3958655c1 feat(albok-service): deploy the fleet knowledgebase service on nh3-docker
albok-service 0.1.0 (vh/albok 0b37431), image pfi/albok-service pinned by
digest, published on host port 8392 because 8390 is the post office.
Host prep playbook creates the fixed ids (albok 1500, albok-read 1510,
albok-personal 1511) and the local store/private roots; the container
gets a mounted /etc/group and group_add so the service can resolve and
chgrp its wing dirs. The config carries a LiteLLM key scoped to
qwen3-embedding and lives outside the deploy-synced conf dir. DNS name
albok.nh3.internal.
2026-10-02 17:56:34 -07:00
vh 9c233ca255 memory: nh3-pve-2 end-of-day state and on-site reinstall checklist 2026-10-02 17:40:50 -07:00
vh 56c372dd6f docs(pfi-tacticalrmm): rmm-mesh nginx upload limit raised to 4G (was the 1 MB default) 2026-10-02 17:08:21 -07:00
vh 5c0d5f0a73 docs(pfi-tacticalrmm): MeshCentral site-admin account lkraven 2026-10-02 17:06:19 -07:00
vh 34659ae9e7 ops(nh3-pve-2): AMT 21 configured (KVM, no-consent, listener) and phoning home to MeshCentral 2026-10-02 14:54:56 -07:00
vh e73adfe7ae dns+memory: nh3-pve-2-amt 10.100.250.63 (UDM port 5 on nh3-mgmt, reservation) 2026-10-02 14:43:16 -07:00
vh 59c9c0715a memory: MS-03 roles — esh-dev inherits nh3-dev's sessions; nh3-pve-2 purpose TBD by design 2026-10-02 09:17:39 -07:00
vh 536230170e memory: one MS-03 deploys as nh3-pve-2 (AMT on DHCP for phone-home; addressing plan) 2026-10-02 09:10:31 -07:00
vh 9dba6baccf homepage: NH3-PVE-AMT card opens MeshCentral (AMT now phones home; LAN management is off) 2026-10-02 09:06:49 -07:00
vh 1a2d763de4 ops(nh3-pve): AMT phones home to MeshCentral (CIRA); AMT on DHCP; LAN management now dark by design 2026-10-02 09:06:24 -07:00
vh 9bcb9417fb feat(amt): amt-cira-setup.py — configure Intel AMT phone-home (CIRA) to MeshCentral over WS-Man
MeshCentral only pushes CIRA through an on-host agent, so this mirrors its
amtmanager.js sequence directly over the LAN: trust MeshCentral's root,
add the MPS (username = 16-char meshid prefix), a periodic policy,
BIOS+OS user-initiated connections and a random environment-detection
domain. Idempotent, with a read-only state report. Enumerate uses
Enumerate+Pull (AMT 16 ignores OptimizeEnumeration; caught by a
positive control that first read 0 instances of a class that has one).

Applied to nh3-pve's AMT 2026-10-02 alongside MeshCentral mpsPass and an
ana-gw VIP/policy for 4433; the AMT does not dial out yet (static IP).
2026-10-02 08:50:47 -07:00
vh 61ba384d28 memory: demo fix-forward 7ab6ae40 verified; deploy-gate gap filed as Worldtree #423 2026-10-02 08:24:33 -07:00
vh 1219cfaca9 memory: demo rolled back from broken c2d87263 to fa8bc51c; b193 retired-key list for demo 2026-10-02 08:17:24 -07:00
vh 8b81579bca ops(pfi-tacticalrmm): MeshCentral to hybrid mode; nh3-pve AMT added and connected 2026-10-02 08:14:19 -07:00
vh a967bb95a6 docs(pfi-tacticalrmm): MeshCentral facts — WAN-only mode silently drops AMT adds; CLI access via the vaulted login token 2026-10-02 08:09:02 -07:00
vh cd30592e55 fix(wt-memory-gate-batch): pass --legacy-mode only when the deployed harness has it
b193 (U11b) retired the legacy plane and removed --legacy-mode from the
harness; passing it there fails the daily batch. The wrapper now greps
core/memory_acceptance at the deployed sha and omits the flag when it is
gone, so it is correct on both sides of the b192 -> b193 deploy (checked:
7a83f2f -> passes it, c2d87263 -> omits it).
2026-10-02 07:59:03 -07:00
vh 1cc8ad3292 memory: U11b step 5 done — live legacy memory deleted on demo and personal (H2 0), stamp sent 2026-10-02 07:54:50 -07:00
vh b0789f4b29 memory: U11b — gate-batch wrapper drops --legacy-mode at b193 2026-10-02 07:50:38 -07:00
vh 6e15f610bc memory: U11b gate streak 3 of 3; step 5 held for Prime's direct go 2026-10-02 07:50:12 -07:00
vh 59fb6b77a8 ops(nh3-dev): seat healthz watchdog — the 5-day pull-only outage must page, not age
althing-seat-daemon took an outside SIGTERM on 2026-09-27 and exited 0, so
Restart=on-failure never revived it; hermes-gateway sat pull-only for five
days with a yellow verdict nobody was watching. Fixed the death mode with a
Restart=always drop-in (also on the jekyll twin, same latent bug) and added
this watchdog for every other death mode: 5-min timer, alert on the second
consecutive non-200 (~10 min sustained), re-alert at most hourly, recovery
mail on green, exit 2 distinguishes a broken alarm wire from a down seat
(backup-freshness precedent). Lifecycle exercised end-to-end against a dead
port before install.
2026-10-02 07:47:17 -07:00
vh 902e16630f feat(dev-backup): add daily and weekly retention (48 hourly + 30 daily + 12 weekly)
Prime's ruling 2026-10-02. retention.py picks the snapshots to delete:
the newest 48, plus the newest of each of the last 30 days and of each of
the last 12 ISO weeks, counting only days and weeks that have snapshots.
Names that are not exactly YYYY-MM-DD_HHMM are never selected, and the
NAS side refuses any path outside that pattern. The unit fails unless
the number kept equals the number expected. Live run: deleted 1, 0
errors, 48 kept as expected.
2026-10-02 07:42:45 -07:00
vh 49c8fdf383 memory: nh3-dev snapshot waits for a natural reboot (Prime); post-boot checks listed 2026-10-02 07:36:15 -07:00
vh 6ec76bf35c ops(nh3-dev): grow root into the 378 GB disk online; swap moved to /swapfile
Prime grew VM 102's scsi0 from 250 to 378 GB after the root alert hit 85%
twice in 14 h. The swap partition sat right after sda1 and blocked growth,
so the playbook moves swap to a 4 GB /swapfile, deletes sda5/sda2, grows
sda1 in place (start sector unchanged) and ext4 online, and sets initramfs
RESUME=none so boots don't wait for the vanished swap. Root is 372 GB, 58%.
2026-10-02 07:33:36 -07:00
vh 01c356e7fb docs(ana-docker): record the RustDesk server key and where the client setup steps live 2026-10-01 15:31:32 -07:00
vh abcf4c3166 fix(scriberr): serve over HTTPS via the fleet TLS caddy so the browser recorder works
The in-browser recorder calls getUserMedia, which browsers refuse on http://
origins, so it sat at "Initializing recorder...". scriberr.nh3.phasefinal.com
is now fronted by the fleet TLS caddy on nh3-dev (wildcard cert) and added to
Scriberr's ALLOWED_ORIGINS; the Homepage link points at it. The plain
http://10.251.50.54:8080 URL keeps working except for recording.
2026-10-01 13:07:18 -07:00
vh 474f78811c memory: U11b — writer/reader default to enabled at b193 (no instance action) 2026-10-01 12:35:57 -07:00
vh f37a50c11a memory: U11b C5 — memory_extractor and memory.escalation cleared to retire after b193 2026-10-01 11:25:01 -07:00
vh aa440708c4 memory: U11b C5 post-b193 config retirements listed; memory_extractor held open (escalation.model_role) 2026-10-01 11:23:32 -07:00