vh
3429b5c55d
ops(pfi-pteradactyl): docker.io/containerd upgraded (manual dockerd + wings restart); docs
2026-10-05 14:18:57 -07:00
vh
58bd802872
ops: docker engine 29.8.2 on esh-ml1, nh3-ml1, nh3-dev, corviduo-dev; docs
2026-10-05 14:15:12 -07:00
vh
5301ede9d2
ops(ana-docker): docker-ce from bookworm repo (29.8.2); live-restore stale-socket lesson; docs
2026-10-05 13:31:30 -07:00
vh
258a3eea68
docs: pfi-gx10 update recorded; held-engine list and Waiting-on-Prime refreshed
2026-10-05 11:34:00 -07:00
vh
f6fe6f67f8
runbook(park 100): pfi-tacticalrmm rebooted + TRMM 1.5.2 (Prime)
2026-10-05 11:13:34 -07:00
vh
912d932917
runbook(park 100): 4 kernel reboots done (nh3-extdev, corviduo-dev, esh-vm-db, pfi-pteradactyl)
2026-10-05 11:09:17 -07:00
vh
a808b349b8
runbook(park 100): pfi-tacticalrmm patched; reboot list now 8 hosts
2026-10-05 10:55:27 -07:00
vh
aa86c37954
ops(pfi-pteradactyl): mask leftover apache2, patch (MariaDB 11.8), engine held; docs
2026-10-05 10:41:23 -07:00
vh
ac068dc2bd
ops(pfi-tacticalrmm): *.phasefinal.com renews via dns-cloudflare (was manual; 16 days from expiry)
2026-10-05 08:43:15 -07:00
vh
67c72ed228
docs(ana-docker): crowdsec sqlite WAL fix for the bouncer fail-closed 403s
2026-10-05 04:19:52 -07:00
vh
241b14bc66
runbook(park 100): results — 15 patched, parks and deferrals; memory
2026-10-05 03:55:58 -07:00
vh
23371954f8
docs(fv-ml1): docker on tank zvol, held packages, kernel metas removed; memory
2026-10-05 02:43:19 -07:00
vh
7f34fe5a7c
docs(irv-ml1, ana-docker): storage cleanup, text encoders on worktank, Traefik readTimeout 10m
2026-10-05 01:19:36 -07:00
vh
afb9d3eef1
memory: irv-ml1 storetank 85% alert is the planned foley stage (200 GiB reserve guard)
2026-10-05 00:36:19 -07:00
vh
c0d3384c68
feat(litellm): embedder primary/failover split — qwen3-embedding on esh-ml1, new qwen3-embedding-bulk on nh3-ml1
...
Prime's ruling 2026-10-05 (relayed by albok-dev, confirmed directly) replaces
the 2026-09-26 simple-shuffle load-share for embeddings only; the reranker
groups are unchanged. Routing uses LiteLLM per-deployment `order` (semantics
read at source in v1.97.0): lowest order among healthy deployments, and a
failed request retries at the next order.
Verified after the restart (43 s gateway outage): 6/6 calls per alias landed
on its primary. With nh3-ml1 tei-embed stopped, qwen3-embedding-bulk served 6/6
from esh-ml1 with 200s; once nh3-ml1 was back, bulk returned 6/6 to nh3-ml1.
The albok-service key now covers qwen3-embedding-bulk (200); the Nemi key does
not (403).
2026-10-05 00:32:42 -07:00
vh
a7a068f56f
albok: move closed — acceptance passed, old nh3-docker store deleted, first VM 110 backup taken
2026-10-05 00:21:15 -07:00
vh
94ca792a97
memory: albok old-store deletion approved by Prime, conditional on acceptance
2026-10-04 23:49:48 -07:00
vh
d3f59aa2c5
feat(albok): move albok-service to a dedicated VM, nh3-albok (VM 110 on nh3-pve-2)
...
albok-service leaves nh3-docker before unit 4 puts the personal vault and
licensed books into restricted wings. On nh3-docker, portainer, portainer_agent,
dockge (socket rw) and dockerd on plaintext tcp/2375 were each root over
/srv/albok. The new host runs only albok-service: no socket containers and no
TCP Docker API, so its Homepage tile is a static services.yaml entry.
- nh3-pve-2: vmbr0 made VLAN-aware (applied by reboot, no guests running);
pbs-ana storage in the new namespace nh3-pve-2, plus a 21:30 vzdump job
(pbs-nh3's recursive sync mirrors it)
- VM 110 nh3-albok, 10.100.50.44 on VLAN 50, Debian 13 cloud-init, 4 vCPU
x86-64-v3 / 12 GiB / 100 GB; built to fleet conventions (vh 1000,
infra-ops 850, docker 851)
- compose: mem_limit 4g -> 8g, homepage labels removed; host port 8392 kept
- cutover by tar stream, verified by digest (contents plus owner, group, mode,
mtime and type identical); albok.nh3.internal now points at nh3-albok
- old /srv/albok stays on nh3-docker (container removed, compose renamed)
until acceptance and the operator's approval to delete it
- sudo-io archive hosts and ops-log STACK_HOSTS include nh3-albok
2026-10-04 23:33:24 -07:00
vh
f76429c457
memory: snapshot — 2026-10-04 2300 (PVE 9 + PBS 4 fleet-wide, webhost destroyed, phasefinal on Pages, sudo-io archiver; 7 settled entries archived)
2026-10-04 23:03:58 -07:00
vh
d9d1dd9325
feat(sudo-io): stop recording bulk data (sudoers BULK_DATA !log_output fleet-wide), drop >64 MiB streams from archives, prune existing bulk recordings
2026-10-04 22:55:49 -07:00
vh
008bd685aa
feat(sudo-io-archive): weekly fleet archive of sudo session recordings (pack, verify, then clear); fv-ml1 HF_HOME on tank and root-pool cleanup
2026-10-04 22:48:05 -07:00
vh
626e0c7f94
memory: dragonfireacoustics-site now on gitea; fv-ml1 unused vLLM images removed
2026-10-04 17:59:21 -07:00
vh
905862a58e
ops(phasefinal-web): dedicated infra-ops-pages token; apex->www as a Cloudflare Redirect Rule
2026-10-04 17:37:32 -07:00
vh
513f6ef38f
ops: phasefinal.com moved to Cloudflare Pages; pfi-ana-webhost destroyed and removed from inventory, DNS and the FortiGate
2026-10-04 17:28:29 -07:00
vh
55e4d01b49
ops(pfi-ana-webhost): retired (VM 110 stopped, FortiGate policy 18 disabled); dragonfireacoustics.com located on Cloudflare Pages
2026-10-04 16:57:49 -07:00
vh
0e3ab447af
ops: retire VM 101 (DC) and VM 104 (nh3-laser) from running + backups; infra-ops login on pbs-ana/pbs-nh3; nh3-pve-2 follow-ups verified done
2026-10-04 16:37:15 -07:00
vh
d0c1bc628a
ops(pbs): pbs-nh3 + pbs-ana upgraded to PBS 4.2.7 / Debian 13; guest-agent driver script, plan and log
2026-10-04 11:44:57 -07:00
vh
73129a0564
ops(esh-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20 (one reboot, Prime on site); ESH cluster uniform on 9
2026-10-04 07:45:05 -07:00
vh
b67ab48833
ops(pfi-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20; reboot test + post-check recorded; pre-reboot checks corrected for PVE 9
2026-10-04 07:16:21 -07:00
vh
a4b7626a13
memory: snapshot — 2026-10-04 ~0546 (albok 0.1.6; 8 settled 09-19 entries archived)
2026-10-04 05:46:09 -07:00
vh
53b3428d13
memory: snapshot — 2026-10-03 ~2045 (esh-nas-pve PVE 9, albok 0.1.5, QNAP LCD, open calls)
2026-10-03 20:40:50 -07:00
vh
d4c5ae8d00
memory: NH3 Mac mini asleep; fix tomorrow on site
2026-10-03 16:23:15 -07:00
vh
9ba8962885
ops(esh-nas-pve): upgraded to PVE 9.2.21; /boot-LV-full trap and fixes recorded
2026-10-03 16:06:33 -07:00
vh
48f28cfe93
ops(esh-nas-pve): PVE 8 reboot test done; two restart foot-guns recorded
2026-10-03 15:46:37 -07:00
vh
5f8a1311a5
memory: snapshot — 2026-10-03 ~1440 (nh3-pve clean on 9, RustDesk 1.1.16, esh-nas-pve PVE 9 assessed)
2026-10-03 15:23:19 -07:00
vh
d9e09fed14
ops(nh3-pve): PVE 9 post-upgrade fixes (7.0 headers + nvidia DKMS, microcode, enterprise repo off); old VM102 snapshot removed
2026-10-03 14:09:41 -07:00
vh
08c3271084
memory: nh3-pve on PVE 9; post-check findings (nh3-ml1 down: no 7.0 headers)
2026-10-03 14:06:01 -07:00
vh
5abd43e1c8
memory: snapshot — 2026-10-03 in-flight (nh3-pve 8→9 under way), two-tier split of 27 entries, 7 archived
2026-10-03 13:36:42 -07:00
vh
22224b5335
ops(nh3-pve-2): register host, network/AMT/disk layout and the re-IP lesson
2026-10-03 13:17:46 -07:00
vh
00a6fbd849
feat(cira-tunnel-watchdog): auto-drop stale AMT CIRA tunnels on MeshCentral's MPS
2026-10-03 12:34:35 -07:00
vh
73da3175e2
docs(albok-service): pushed tag verified to match the deployed image
2026-10-03 10:00:03 -07:00
vh
f8290f64bf
memory: worldtree gate duty reframed as post-deletion regression check (infra-ops 01M413W4)
2026-10-03 07:50:47 -07:00
vh
3a6d997c99
ops(esh-nas-pve): tank verified healthy after two scrubs; disk kept; PVE 9 blocker cleared
2026-10-03 06:28:48 -07:00
vh
70fa0fbd53
memory: Nemi timer installed
2026-10-03 05:58:12 -07:00
vh
8a4dbf2341
fix(albok-service): 0.1.2 on nh3-docker (fd leak); add restricted wing personal/agent-feedback; Nemi credentials
2026-10-03 02:54:18 -07:00
vh
1fb36b32d2
memory: ESH tank first scrub done, second scrub running
2026-10-03 01:18:24 -07:00
vh
3a91eff2e6
memory: ESH tank scrub repairing the Aug 20 replacement's unrepaired blocks
2026-10-02 23:33:07 -07:00
vh
55004090d1
ops(esh-pve-2): register host, AMT phoning home to MeshCentral; note MeshCentral first-CIRA crash race
2026-10-02 22:25:37 -07:00
vh
6315ce38d5
memory: ESH tank verified residual, cleared, full scrub running
2026-10-02 19:26:30 -07:00
vh
62254023f3
docs: esh-pve-cluster PVE 8->9 upgrade plan (not executed); ESH tank DEGRADED finding
2026-10-02 19:20:28 -07:00
vh