Commit Graph
1584 Commits
Author SHA1 Message Date
vh 7346e4623c feat(comfyui): HF_HUB_CACHE=/basedir/models/foley-hf-cache for the Foley nodes
HunyuanVideo-Foley, MMAudio and ControlFoley load their Hugging Face dependencies
from pinned snapshots staged on the model store. Without this they look in the
container's ephemeral cache and download again on every recreate. Requested by
comfy-dev; HF_HUB_OFFLINE deliberately not set.
2026-10-05 01:37:21 -07:00
vh 7f34fe5a7c docs(irv-ml1, ana-docker): storage cleanup, text encoders on worktank, Traefik readTimeout 10m 2026-10-05 01:19:36 -07:00
vh 88c3d8a001 feat(skaldsong): pass SKALDSONG_COMPLETER_API_KEY to the container
Since skaldsong v0.39.0 the structured pipeline passes call the LiteLLM
gateway directly; with no key every story generation fails 401 at its first
pass. Operator ruling 2026-10-05: the shared all-agents key until Skaldsong
v1.0 final, then a project-scoped key. The stack passes an explicit
environment list, so the .env line alone does not reach the container.
2026-10-05 00:46:24 -07:00
vh afb9d3eef1 memory: irv-ml1 storetank 85% alert is the planned foley stage (200 GiB reserve guard) 2026-10-05 00:36:19 -07:00
vh c0d3384c68 feat(litellm): embedder primary/failover split — qwen3-embedding on esh-ml1, new qwen3-embedding-bulk on nh3-ml1
Prime's ruling 2026-10-05 (relayed by albok-dev, confirmed directly) replaces
the 2026-09-26 simple-shuffle load-share for embeddings only; the reranker
groups are unchanged. Routing uses LiteLLM per-deployment `order` (semantics
read at source in v1.97.0): lowest order among healthy deployments, and a
failed request retries at the next order.

Verified after the restart (43 s gateway outage): 6/6 calls per alias landed
on its primary. With nh3-ml1 tei-embed stopped, qwen3-embedding-bulk served 6/6
from esh-ml1 with 200s; once nh3-ml1 was back, bulk returned 6/6 to nh3-ml1.
The albok-service key now covers qwen3-embedding-bulk (200); the Nemi key does
not (403).
2026-10-05 00:32:42 -07:00
vh a7a068f56f albok: move closed — acceptance passed, old nh3-docker store deleted, first VM 110 backup taken 2026-10-05 00:21:15 -07:00
vh 94ca792a97 memory: albok old-store deletion approved by Prime, conditional on acceptance 2026-10-04 23:49:48 -07:00
vh d3f59aa2c5 feat(albok): move albok-service to a dedicated VM, nh3-albok (VM 110 on nh3-pve-2)
albok-service leaves nh3-docker before unit 4 puts the personal vault and
licensed books into restricted wings. On nh3-docker, portainer, portainer_agent,
dockge (socket rw) and dockerd on plaintext tcp/2375 were each root over
/srv/albok. The new host runs only albok-service: no socket containers and no
TCP Docker API, so its Homepage tile is a static services.yaml entry.

- nh3-pve-2: vmbr0 made VLAN-aware (applied by reboot, no guests running);
  pbs-ana storage in the new namespace nh3-pve-2, plus a 21:30 vzdump job
  (pbs-nh3's recursive sync mirrors it)
- VM 110 nh3-albok, 10.100.50.44 on VLAN 50, Debian 13 cloud-init, 4 vCPU
  x86-64-v3 / 12 GiB / 100 GB; built to fleet conventions (vh 1000,
  infra-ops 850, docker 851)
- compose: mem_limit 4g -> 8g, homepage labels removed; host port 8392 kept
- cutover by tar stream, verified by digest (contents plus owner, group, mode,
  mtime and type identical); albok.nh3.internal now points at nh3-albok
- old /srv/albok stays on nh3-docker (container removed, compose renamed)
  until acceptance and the operator's approval to delete it
- sudo-io archive hosts and ops-log STACK_HOSTS include nh3-albok
2026-10-04 23:33:24 -07:00
vh 867f33d672 docs(pfi): amend 10-04 bench doc per brokkr review
- Fit gap restated on nvidia-smi accounting (~17 MiB rest / ~130 MiB peak
  over, vs the live seat's 15,220/15,437 headroom); the 1.2 GiB figure
  conflated the torch pool fraction with nvidia-smi totals
- Footnote: baseline peak row is suite-only, not shape-matched to the
  candidate gate peaks
- namedef cell labeled as the description-reversal stand-in, not the
  loaded-name variant of arXiv 2609.26758; columns are overlaps
2026-10-04 23:12:41 -07:00
vh 523f086939 docs(pfi): Jev candidate bench 2026-10-04 — H2O-lightning-4b and Clef-flash vs Intern baseline
Vendor-path benches on fv-ml1 GPU 3 at the 14.4 GiB / 32k cap. Neither
candidate replaces Intern-Decision-4B: H2O reproduces its vendor figure
(205x3) and has the best calibration, but rests 1 GiB over the cap and
is 2.2x slower at the 21-criteria shape; Clef loses on accuracy (191-192),
hard tier (72-73/111), and ships a silent 16,384-token input slice.
Adds the namedef cell and JB_AUTH wiring to the 09-30 harness client.
2026-10-04 23:10:36 -07:00
vh f76429c457 memory: snapshot — 2026-10-04 2300 (PVE 9 + PBS 4 fleet-wide, webhost destroyed, phasefinal on Pages, sudo-io archiver; 7 settled entries archived) 2026-10-04 23:03:58 -07:00
vh d9d1dd9325 feat(sudo-io): stop recording bulk data (sudoers BULK_DATA !log_output fleet-wide), drop >64 MiB streams from archives, prune existing bulk recordings 2026-10-04 22:55:49 -07:00
vh 008bd685aa feat(sudo-io-archive): weekly fleet archive of sudo session recordings (pack, verify, then clear); fv-ml1 HF_HOME on tank and root-pool cleanup 2026-10-04 22:48:05 -07:00
vh 626e0c7f94 memory: dragonfireacoustics-site now on gitea; fv-ml1 unused vLLM images removed 2026-10-04 17:59:21 -07:00
vh 905862a58e ops(phasefinal-web): dedicated infra-ops-pages token; apex->www as a Cloudflare Redirect Rule 2026-10-04 17:37:32 -07:00
vh 513f6ef38f ops: phasefinal.com moved to Cloudflare Pages; pfi-ana-webhost destroyed and removed from inventory, DNS and the FortiGate 2026-10-04 17:28:29 -07:00
vh 55e4d01b49 ops(pfi-ana-webhost): retired (VM 110 stopped, FortiGate policy 18 disabled); dragonfireacoustics.com located on Cloudflare Pages 2026-10-04 16:57:49 -07:00
vh 0e3ab447af ops: retire VM 101 (DC) and VM 104 (nh3-laser) from running + backups; infra-ops login on pbs-ana/pbs-nh3; nh3-pve-2 follow-ups verified done 2026-10-04 16:37:15 -07:00
vh d0c1bc628a ops(pbs): pbs-nh3 + pbs-ana upgraded to PBS 4.2.7 / Debian 13; guest-agent driver script, plan and log 2026-10-04 11:44:57 -07:00
vh 4d4851acd2 docs(albok-service): fleet/memory reindex result on 0.1.8; reindex path shape note 2026-10-04 11:44:13 -07:00
vh fce4e1a583 chore(albok-service): 0.1.8 on nh3-docker (e653c29; fleet/memory reindex on new tag) 2026-10-04 11:16:24 -07:00
vh e8faa74512 docs(albok-service): fleet/memory reindex stood down pending 0.1.8 2026-10-04 08:18:42 -07:00
vh ebe0165acf chore(albok-service): 0.1.7 on nh3-docker (clean-archive build of 64f75dc) 2026-10-04 08:17:33 -07:00
vh 73129a0564 ops(esh-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20 (one reboot, Prime on site); ESH cluster uniform on 9 2026-10-04 07:45:05 -07:00
vh 9a6b23581e ops(esh-pve): PVE 9 prep done (8.4.21, microcode, systemd-boot out, removable shim, NIC pins, vmbr0 bridge-ports fixed) 2026-10-04 07:32:13 -07:00
vh b8535dfb52 homepage: add NH3-PVE-2 and its AMT (MeshCentral) cards 2026-10-04 07:23:28 -07:00
vh b67ab48833 ops(pfi-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20; reboot test + post-check recorded; pre-reboot checks corrected for PVE 9 2026-10-04 07:16:21 -07:00
vh 075ba0930e ops(pfi-pve): PVE 9 prep done (8.4.21, grub-efi-amd64 on the UEFI/LVM box, microcode, NIC pins); upgrade plan + playbooks 2026-10-04 06:11:13 -07:00
vh a4b7626a13 memory: snapshot — 2026-10-04 ~0546 (albok 0.1.6; 8 settled 09-19 entries archived) 2026-10-04 05:46:09 -07:00
vh 2449e8806f fix(albok-service): 0.1.6 on nh3-docker (review fold: locked index build) 2026-10-04 05:45:25 -07:00
vh 957cd80d5e docs(albok-service): 0.1.5 measured live (fleet/memory 3.97 s -> 0.22 s) 2026-10-03 22:49:53 -07:00
vh 53b3428d13 memory: snapshot — 2026-10-03 ~2045 (esh-nas-pve PVE 9, albok 0.1.5, QNAP LCD, open calls) 2026-10-03 20:40:50 -07:00
vh 0ed188be5e perf(albok-service): 0.1.5 on nh3-docker (search 4.5 s -> ~0.25 s) 2026-10-03 20:40:11 -07:00
vh d4c5ae8d00 memory: NH3 Mac mini asleep; fix tomorrow on site 2026-10-03 16:23:15 -07:00
vh 453c927ace feat(qnap-lcd): Proxmox-correct front-panel LCD menu for esh-nas-pve; albok redeploy note 2026-10-03 16:14:43 -07:00
vh f696aba10a fix(albok-service): 0.1.4 on nh3-docker (version header from metadata) 2026-10-03 16:12:01 -07:00
vh 9ba8962885 ops(esh-nas-pve): upgraded to PVE 9.2.21; /boot-LV-full trap and fixes recorded 2026-10-03 16:06:33 -07:00
vh ca18b04a22 fix(albok-service): 0.1.3 on nh3-docker (ingester stamp out of equivalence) 2026-10-03 15:59:10 -07:00
vh 241a668f3c ops(esh-nas-pve): PVE 9 upgrade playbook (snapshot, trixie sources, detached dist-upgrade) 2026-10-03 15:52:44 -07:00
vh 473c52dc6b docs(pve9 plan): keep ESH DNS up during the NAS reboot (unmount 2 shares instead of stopping VM 100) 2026-10-03 15:50:31 -07:00
vh 48f28cfe93 ops(esh-nas-pve): PVE 8 reboot test done; two restart foot-guns recorded 2026-10-03 15:46:37 -07:00
vh f648e18b1a ops(esh-nas-pve): PVE 9 no-downtime prep (8.4.21, microcode, systemd-boot out, NIC pins) 2026-10-03 15:31:00 -07:00
vh 5f8a1311a5 memory: snapshot — 2026-10-03 ~1440 (nh3-pve clean on 9, RustDesk 1.1.16, esh-nas-pve PVE 9 assessed) 2026-10-03 15:23:19 -07:00
vh d4675f2cb5 ops(ana-docker): rustdesk-server 1.1.14 -> 1.1.16; server key vaulted 2026-10-03 14:24:32 -07:00
vh 5e25d4bae1 docs(pve9 plan): headers must be present before the upgrade (nh3-pve lesson) 2026-10-03 14:09:47 -07:00
vh d9e09fed14 ops(nh3-pve): PVE 9 post-upgrade fixes (7.0 headers + nvidia DKMS, microcode, enterprise repo off); old VM102 snapshot removed 2026-10-03 14:09:41 -07:00
vh 08c3271084 memory: nh3-pve on PVE 9; post-check findings (nh3-ml1 down: no 7.0 headers) 2026-10-03 14:06:01 -07:00
vh 5abd43e1c8 memory: snapshot — 2026-10-03 in-flight (nh3-pve 8→9 under way), two-tier split of 27 entries, 7 archived 2026-10-03 13:36:42 -07:00
vh 4b2a81bf39 feat(pve-nag-patch): remove the subscription popup on the PVE hosts, re-applied by an apt hook 2026-10-03 13:27:11 -07:00
vh 22224b5335 ops(nh3-pve-2): register host, network/AMT/disk layout and the re-IP lesson 2026-10-03 13:17:46 -07:00