Commit Graph
744 Commits
Author SHA1 Message Date
vh aa86c37954 ops(pfi-pteradactyl): mask leftover apache2, patch (MariaDB 11.8), engine held; docs 2026-10-05 10:41:23 -07:00
vh ac068dc2bd ops(pfi-tacticalrmm): *.phasefinal.com renews via dns-cloudflare (was manual; 16 days from expiry) 2026-10-05 08:43:15 -07:00
vh 67c72ed228 docs(ana-docker): crowdsec sqlite WAL fix for the bouncer fail-closed 403s 2026-10-05 04:19:52 -07:00
vh 241b14bc66 runbook(park 100): results — 15 patched, parks and deferrals; memory 2026-10-05 03:55:58 -07:00
vh 23371954f8 docs(fv-ml1): docker on tank zvol, held packages, kernel metas removed; memory 2026-10-05 02:43:19 -07:00
vh 7f34fe5a7c docs(irv-ml1, ana-docker): storage cleanup, text encoders on worktank, Traefik readTimeout 10m 2026-10-05 01:19:36 -07:00
vh afb9d3eef1 memory: irv-ml1 storetank 85% alert is the planned foley stage (200 GiB reserve guard) 2026-10-05 00:36:19 -07:00
vh c0d3384c68 feat(litellm): embedder primary/failover split — qwen3-embedding on esh-ml1, new qwen3-embedding-bulk on nh3-ml1
Prime's ruling 2026-10-05 (relayed by albok-dev, confirmed directly) replaces
the 2026-09-26 simple-shuffle load-share for embeddings only; the reranker
groups are unchanged. Routing uses LiteLLM per-deployment `order` (semantics
read at source in v1.97.0): lowest order among healthy deployments, and a
failed request retries at the next order.

Verified after the restart (43 s gateway outage): 6/6 calls per alias landed
on its primary. With nh3-ml1 tei-embed stopped, qwen3-embedding-bulk served 6/6
from esh-ml1 with 200s; once nh3-ml1 was back, bulk returned 6/6 to nh3-ml1.
The albok-service key now covers qwen3-embedding-bulk (200); the Nemi key does
not (403).
2026-10-05 00:32:42 -07:00
vh a7a068f56f albok: move closed — acceptance passed, old nh3-docker store deleted, first VM 110 backup taken 2026-10-05 00:21:15 -07:00
vh 94ca792a97 memory: albok old-store deletion approved by Prime, conditional on acceptance 2026-10-04 23:49:48 -07:00
vh d3f59aa2c5 feat(albok): move albok-service to a dedicated VM, nh3-albok (VM 110 on nh3-pve-2)
albok-service leaves nh3-docker before unit 4 puts the personal vault and
licensed books into restricted wings. On nh3-docker, portainer, portainer_agent,
dockge (socket rw) and dockerd on plaintext tcp/2375 were each root over
/srv/albok. The new host runs only albok-service: no socket containers and no
TCP Docker API, so its Homepage tile is a static services.yaml entry.

- nh3-pve-2: vmbr0 made VLAN-aware (applied by reboot, no guests running);
  pbs-ana storage in the new namespace nh3-pve-2, plus a 21:30 vzdump job
  (pbs-nh3's recursive sync mirrors it)
- VM 110 nh3-albok, 10.100.50.44 on VLAN 50, Debian 13 cloud-init, 4 vCPU
  x86-64-v3 / 12 GiB / 100 GB; built to fleet conventions (vh 1000,
  infra-ops 850, docker 851)
- compose: mem_limit 4g -> 8g, homepage labels removed; host port 8392 kept
- cutover by tar stream, verified by digest (contents plus owner, group, mode,
  mtime and type identical); albok.nh3.internal now points at nh3-albok
- old /srv/albok stays on nh3-docker (container removed, compose renamed)
  until acceptance and the operator's approval to delete it
- sudo-io archive hosts and ops-log STACK_HOSTS include nh3-albok
2026-10-04 23:33:24 -07:00
vh f76429c457 memory: snapshot — 2026-10-04 2300 (PVE 9 + PBS 4 fleet-wide, webhost destroyed, phasefinal on Pages, sudo-io archiver; 7 settled entries archived) 2026-10-04 23:03:58 -07:00
vh d9d1dd9325 feat(sudo-io): stop recording bulk data (sudoers BULK_DATA !log_output fleet-wide), drop >64 MiB streams from archives, prune existing bulk recordings 2026-10-04 22:55:49 -07:00
vh 008bd685aa feat(sudo-io-archive): weekly fleet archive of sudo session recordings (pack, verify, then clear); fv-ml1 HF_HOME on tank and root-pool cleanup 2026-10-04 22:48:05 -07:00
vh 626e0c7f94 memory: dragonfireacoustics-site now on gitea; fv-ml1 unused vLLM images removed 2026-10-04 17:59:21 -07:00
vh 905862a58e ops(phasefinal-web): dedicated infra-ops-pages token; apex->www as a Cloudflare Redirect Rule 2026-10-04 17:37:32 -07:00
vh 513f6ef38f ops: phasefinal.com moved to Cloudflare Pages; pfi-ana-webhost destroyed and removed from inventory, DNS and the FortiGate 2026-10-04 17:28:29 -07:00
vh 55e4d01b49 ops(pfi-ana-webhost): retired (VM 110 stopped, FortiGate policy 18 disabled); dragonfireacoustics.com located on Cloudflare Pages 2026-10-04 16:57:49 -07:00
vh 0e3ab447af ops: retire VM 101 (DC) and VM 104 (nh3-laser) from running + backups; infra-ops login on pbs-ana/pbs-nh3; nh3-pve-2 follow-ups verified done 2026-10-04 16:37:15 -07:00
vh d0c1bc628a ops(pbs): pbs-nh3 + pbs-ana upgraded to PBS 4.2.7 / Debian 13; guest-agent driver script, plan and log 2026-10-04 11:44:57 -07:00
vh 73129a0564 ops(esh-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20 (one reboot, Prime on site); ESH cluster uniform on 9 2026-10-04 07:45:05 -07:00
vh b67ab48833 ops(pfi-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20; reboot test + post-check recorded; pre-reboot checks corrected for PVE 9 2026-10-04 07:16:21 -07:00
vh a4b7626a13 memory: snapshot — 2026-10-04 ~0546 (albok 0.1.6; 8 settled 09-19 entries archived) 2026-10-04 05:46:09 -07:00
vh 53b3428d13 memory: snapshot — 2026-10-03 ~2045 (esh-nas-pve PVE 9, albok 0.1.5, QNAP LCD, open calls) 2026-10-03 20:40:50 -07:00
vh d4c5ae8d00 memory: NH3 Mac mini asleep; fix tomorrow on site 2026-10-03 16:23:15 -07:00
vh 9ba8962885 ops(esh-nas-pve): upgraded to PVE 9.2.21; /boot-LV-full trap and fixes recorded 2026-10-03 16:06:33 -07:00
vh 48f28cfe93 ops(esh-nas-pve): PVE 8 reboot test done; two restart foot-guns recorded 2026-10-03 15:46:37 -07:00
vh 5f8a1311a5 memory: snapshot — 2026-10-03 ~1440 (nh3-pve clean on 9, RustDesk 1.1.16, esh-nas-pve PVE 9 assessed) 2026-10-03 15:23:19 -07:00
vh d9e09fed14 ops(nh3-pve): PVE 9 post-upgrade fixes (7.0 headers + nvidia DKMS, microcode, enterprise repo off); old VM102 snapshot removed 2026-10-03 14:09:41 -07:00
vh 08c3271084 memory: nh3-pve on PVE 9; post-check findings (nh3-ml1 down: no 7.0 headers) 2026-10-03 14:06:01 -07:00
vh 5abd43e1c8 memory: snapshot — 2026-10-03 in-flight (nh3-pve 8→9 under way), two-tier split of 27 entries, 7 archived 2026-10-03 13:36:42 -07:00
vh 22224b5335 ops(nh3-pve-2): register host, network/AMT/disk layout and the re-IP lesson 2026-10-03 13:17:46 -07:00
vh 00a6fbd849 feat(cira-tunnel-watchdog): auto-drop stale AMT CIRA tunnels on MeshCentral's MPS 2026-10-03 12:34:35 -07:00
vh 73da3175e2 docs(albok-service): pushed tag verified to match the deployed image 2026-10-03 10:00:03 -07:00
vh f8290f64bf memory: worldtree gate duty reframed as post-deletion regression check (infra-ops 01M413W4) 2026-10-03 07:50:47 -07:00
vh 3a6d997c99 ops(esh-nas-pve): tank verified healthy after two scrubs; disk kept; PVE 9 blocker cleared 2026-10-03 06:28:48 -07:00
vh 70fa0fbd53 memory: Nemi timer installed 2026-10-03 05:58:12 -07:00
vh 8a4dbf2341 fix(albok-service): 0.1.2 on nh3-docker (fd leak); add restricted wing personal/agent-feedback; Nemi credentials 2026-10-03 02:54:18 -07:00
vh 1fb36b32d2 memory: ESH tank first scrub done, second scrub running 2026-10-03 01:18:24 -07:00
vh 3a91eff2e6 memory: ESH tank scrub repairing the Aug 20 replacement's unrepaired blocks 2026-10-02 23:33:07 -07:00
vh 55004090d1 ops(esh-pve-2): register host, AMT phoning home to MeshCentral; note MeshCentral first-CIRA crash race 2026-10-02 22:25:37 -07:00
vh 6315ce38d5 memory: ESH tank verified residual, cleared, full scrub running 2026-10-02 19:26:30 -07:00
vh 62254023f3 docs: esh-pve-cluster PVE 8->9 upgrade plan (not executed); ESH tank DEGRADED finding 2026-10-02 19:20:28 -07:00
vh 5439d5fdc1 albok-service: 0.1.1 (health fix) on nh3-docker 2026-10-02 18:04:57 -07:00
vh d3958655c1 feat(albok-service): deploy the fleet knowledgebase service on nh3-docker
albok-service 0.1.0 (vh/albok 0b37431), image pfi/albok-service pinned by
digest, published on host port 8392 because 8390 is the post office.
Host prep playbook creates the fixed ids (albok 1500, albok-read 1510,
albok-personal 1511) and the local store/private roots; the container
gets a mounted /etc/group and group_add so the service can resolve and
chgrp its wing dirs. The config carries a LiteLLM key scoped to
qwen3-embedding and lives outside the deploy-synced conf dir. DNS name
albok.nh3.internal.
2026-10-02 17:56:34 -07:00
vh 9c233ca255 memory: nh3-pve-2 end-of-day state and on-site reinstall checklist 2026-10-02 17:40:50 -07:00
vh 34659ae9e7 ops(nh3-pve-2): AMT 21 configured (KVM, no-consent, listener) and phoning home to MeshCentral 2026-10-02 14:54:56 -07:00
vh e73adfe7ae dns+memory: nh3-pve-2-amt 10.100.250.63 (UDM port 5 on nh3-mgmt, reservation) 2026-10-02 14:43:16 -07:00
vh 59c9c0715a memory: MS-03 roles — esh-dev inherits nh3-dev's sessions; nh3-pve-2 purpose TBD by design 2026-10-02 09:17:39 -07:00
vh 536230170e memory: one MS-03 deploys as nh3-pve-2 (AMT on DHCP for phone-home; addressing plan) 2026-10-02 09:10:31 -07:00