Commit Graph
170 Commits
Author SHA1 Message Date
vh 77b75aa172 feat(svos): v0.0.14 live on nh3-svos + nh3-dev (cards approved 0909); hermes task runtime on the query-file route (Prime) 2026-10-17 09:11:10 -07:00
vh 39458e013a fix(svos v0.0.14 playbook): terminal verify reads the ScanError line (svos-dev: scan:on is only on the signed /health) 2026-10-17 07:53:30 -07:00
vh b72783cb3c feat(svos): v0.0.14 (3726d73) release playbooks — staged on nh3-svos and built on nh3-dev; cutover in three phases held for Prime's card window 2026-10-17 07:50:44 -07:00
vh eb2a0a532c feat(nh3-dev): /srv/nemi/books book drop (fiction/ nonfiction/ rejected/) for albok unit 8 2026-10-16 11:09:13 -07:00
vh dd816662dc ops(svos): main 33a7077 (v0.0.13+2) on nh3-svos + nh3-dev — kernel 0.0.13, interface 0.0.5 (briefs), postoffice 0.0.3, UI 0.0.18
On svos-dev's ask, under Prime's 0955 go. The interface unit gets
SVOS_ARTIFACT_HOST=nh3-svos via a drop-in, and the briefs artifact mapping uses
the same host. Both changed manifests were held for the owner, and each app
read down (wrong-revision) until Prime approved its card: interface at
11:09:45, postoffice at 11:11:17. The postoffice was rolled only once he was
at the board, since a held bridge carries no Miranda mail. Health: schema 13,
config_gen 33f001f4e2dbc7dc, apps up 7.
2026-10-14 11:11:59 -07:00
vh a84f6b00df ops(svos): kernel 0.0.12 (d95078c) on nh3-svos; hermes runtime task-capable (Prime via svos-dev)
Kernel-only release from a clean archive of v0.0.12; everything else is
carried from 452a592. Offline backup first; READY in 0.74 s, schema 13, apps
up 7. The existing hermes runtime gained process = python3 and the one-off
task_argv (hermes -z, --accept-hooks, no --yolo), and its argv moved to the
launcher's chat form. check-config valid; config_gen 21489c1f0fff3c10.
2026-10-14 04:17:35 -07:00
vh 5e8307ed7a ops(svos): v0.0.11 (452a592) on nh3-svos + nh3-dev — kernel 0.0.11, schema 13, U15 [tasks] config; terminal 0.0.4
Built in-session from a clean archive of the tag (kernel, connector base and
terminal wheels; everything else carried from 7f95e54). Ran after the 21:30
vzdump of VM 111, with an offline backup of the schema 12 database first.
check-config accepted the new [tasks] table and the tasks artifact mapping
(host nh3-svos, the terminal's artifact_host). Kernel READY in 1.17 s,
schema 13, apps up 7. nh3-dev's current moved to 452a592; svos-terminal 0.0.4
registered manifest revision 4, which was pinned without a card.
2026-10-13 21:33:16 -07:00
vh 7ca2f76d21 feat(albok): personal/inbox wing on nh3-albok + /srv/nemi/inbox capture root on nh3-dev (unit 4.4 slice C, gate 2)
Prime ruled albok subsumes note intake (1530). The wing is like personal/notes
(albok-personal 1511, not service-restricted); health reads 7 wings ready.
Nemi's token was re-minted with read+write on the new wing and vaulted over
albok/token-nemi. /srv/nemi/inbox is lkraven:nemi-drop 2770 with a default ACL,
plus u:lkraven:rx on /srv/nemi, so Nemi reaches it from lkraven's user manager,
whose groups predate nemi-drop. Both paths are verified by the playbook.
2026-10-13 17:31:49 -07:00
vh eda7025ed1 ops(svos): A171 — miranda's app-postoffice write grant autonomous (Prime 2026-10-13 1400) 2026-10-13 13:59:30 -07:00
vh 3e634bb3d0 ops(infra-ops): per-host satellite keys; satellite-1 authorized on esh-nas
Satellite agents get their OWN key under the infra-ops identity instead of a
copy of the fleet key, so one can be revoked by deleting a line. The list is
playbooks/files/infra-ops-satellite-keys (target short hostname + public key).
bootstrap-infra-ops-user.yaml now writes the fleet key plus that host's
satellite keys, exactly, so a re-run keeps listed keys and removes unlisted
ones. satellite-1 (SHA256:KEFUjceF...) is authorized on esh-nas only (Prime,
2026-10-13); the playbook re-run there installed both keys, and the fleet key
still logs in with sudo.
2026-10-13 13:33:04 -07:00
vh 30a7220b2c ops(esh-nas): infra-ops identity bootstrapped in CT 103 via pct from esh-pve-nas
Prime's ruling 2026-10-13. esh-nas (10.0.50.50) had no infra-ops user, so key
login was refused. The same steps as bootstrap-infra-ops-user.yaml (uid/gid
850, fleet key, NOPASSWD sudoers with log_output and the BULK_DATA carve-out,
visudo-validated) ran inside the container from its host. Verified with a
direct ssh infra-ops@10.0.50.50 and sudo -n whoami = root.
2026-10-13 13:25:21 -07:00
vh cd3c4b8be9 feat(gitea-runner): opt-in fast runner on nh3-docker (label nh3-docker), no docker socket in jobs
Prime's CI fix A. ana-docker's runner sits on HDD mirrors with no SLOG; synced
8 KiB writes measured ~34 ms there vs ~2.7 ms on nh3-docker (same probe, one run,
n=3 each). Repos opt in with runs-on: nh3-docker; the runner carries no
pfi-fleet label. Its config is tighter because nh3-docker also runs NH3 DNS and
the post office: docker_host '-' (verified absent in the job, present on the
ana-docker baseline), capacity 1, jobs capped at 4 GiB / 6 CPUs, a network per
job. The deploy playbook takes runner_config; each runner's homepage name is
its runner name.
2026-10-12 17:27:46 -07:00
vh 70b7281182 ops(albok): vh joins every albok read group; macOS withdrawal-test run staged
Prime, 2026-10-09: "vh gets albok-everything". playbooks/nh3-albok-vh-viewer.yaml
joins vh to albok-read, albok-personal and albok-feedback on nh3-albok and
authorizes his four Macs' public keys for vh; verified at a fresh login that
every wing's .browse/ lists and reads.

The withdrawal harness gains a macOS driver (run-mac.sh: every mountable FUSE
backend, a read-only smoke of the real wings, results copied back), env
overrides for a non-container client, and summarize.py. Sanity-checked as vh
from the Linux container before staging.
2026-10-09 22:45:52 -07:00
vh c35f6510bf feat(fv-ml1): rebalance GPUs 0/1 by usage; contexts to published maximums; gen-small default output cap
Layout: GPU 0 = cyberprev (KV 16 -> 15 GiB, 1.29x), erp-seat (moved from
GPU 1, KV 8.49 -> 5.5 GiB, 1.32x), voices (ctx 8,192 -> 40,960, its native
maximum, KV 6 GiB, 1.07x), coder (moved from GPU 1, ctx 16,384 -> 32,768,
the card maximum, 2.35x), parakeet-nemo. GPU 1 = gen-small (moved from
GPU 0, 5.5 GiB, 1.76x), meromero-rp (22 GiB, 1.32x), intern-decision.

Sizing rule: KV >= 1.25x max context rounded up to 0.5 GiB, checked against
7 days of LiteLLM traffic and the engines' own KV-usage logs. Each card keeps
spare memory for its bursty tenant (parakeet's cap on GPU 0, intern-decision's
32k call on GPU 1). YaRN extensions are not enabled: they do not fit, and the
Qwen cards warn static YaRN degrades short prompts.

gen-small gets --override-generation-config max_new_tokens 32768. It applies
only to requests that omit max_tokens (vLLM get_max_tokens) and answers 9
runaway ~260k-token replies in 7 days; the longest normal reply was 6,357.

Applied with playbooks/fv-ml1-rebalance-20261005.yaml (boot-check guard and
health/KV verification per seat); all nine seats healthy, gateway smoke
tests pass.
2026-10-05 22:51:17 -07:00
vh 6851ecbd0b fix(fv-ml1): pin vLLM KV in bytes on meromero (22 GiB) and coder; gen-small KV 8 -> 5.5 GiB
meromero-rp sized its KV from --gpu-memory-utilization, and vLLM's profile
counts co-booting neighbours as its own non-torch memory. Three boots gave
20.37 / 26.74 / 29.04 GiB of KV, and the 2026-10-05 Docker-move restart cut
GPU 1 free memory to 3,981 MiB, below intern-decision's 32k-token budget.
Coder read negative KV on every co-boot. Both are now pinned, so util only
gates the boot check (meromero util 0.52 -> 0.47, just above its footprint).

GPU 0 plateaus at ~60 MiB free 1-2 days after a boot (Beszel), so
gen-small's KV drops to 5.5 GiB (461,215 tokens, 1.76x) and parakeet-nemo
stays on GPU 0 with room for its 3,840 MiB cap. GPU 3 stays clear.

Applied with playbooks/fv-ml1-kv-pin-20261005.yaml: GPU 1 free 3,981 ->
11,801 MiB, GPU 0 495 -> 5,039 MiB right after boot; all nine seats healthy.
2026-10-05 21:40:28 -07:00
vh e369b1c769 feat(nh3-svos): credential-free SVOS offline shell before Basic auth; contract static routes; backup drill results
svos-dev: the service worker fetches its hash-pinned INVENTORY with credentials omitted. Exactly those files plus /sw.js are now served before basic_auth (GET/HEAD, exact files). The list is generated from the release's sw.js by svos-front-inventory, which hash-checks each file and validates before reload. Gated static routes follow the web/README hosting contract: missing assets 404 instead of index.html, other document requests get index.html, everything else 404, unversioned files no-cache. acceptance-matrix.sh: 35/35 live; 16 FAIL against the previous config (positive control). git added to the nh3-svos playbook (kernel's remote check). Live backup drill: quiesced backup ~1-2 s outage; a refused freeze does NOT abort vzdump (unfrozen backup, job reported successful); both images restore-mounted read-only and verified.
2026-10-05 19:58:19 -07:00
vh d7e845eb64 feat(nh3-svos): VM 111 on nh3-pve-2 for the new SVOS (svos-dev, Prime); generic cloud-init VM playbook 2026-10-05 18:59:53 -07:00
vh d3f59aa2c5 feat(albok): move albok-service to a dedicated VM, nh3-albok (VM 110 on nh3-pve-2)
albok-service leaves nh3-docker before unit 4 puts the personal vault and
licensed books into restricted wings. On nh3-docker, portainer, portainer_agent,
dockge (socket rw) and dockerd on plaintext tcp/2375 were each root over
/srv/albok. The new host runs only albok-service: no socket containers and no
TCP Docker API, so its Homepage tile is a static services.yaml entry.

- nh3-pve-2: vmbr0 made VLAN-aware (applied by reboot, no guests running);
  pbs-ana storage in the new namespace nh3-pve-2, plus a 21:30 vzdump job
  (pbs-nh3's recursive sync mirrors it)
- VM 110 nh3-albok, 10.100.50.44 on VLAN 50, Debian 13 cloud-init, 4 vCPU
  x86-64-v3 / 12 GiB / 100 GB; built to fleet conventions (vh 1000,
  infra-ops 850, docker 851)
- compose: mem_limit 4g -> 8g, homepage labels removed; host port 8392 kept
- cutover by tar stream, verified by digest (contents plus owner, group, mode,
  mtime and type identical); albok.nh3.internal now points at nh3-albok
- old /srv/albok stays on nh3-docker (container removed, compose renamed)
  until acceptance and the operator's approval to delete it
- sudo-io archive hosts and ops-log STACK_HOSTS include nh3-albok
2026-10-04 23:33:24 -07:00
vh d9d1dd9325 feat(sudo-io): stop recording bulk data (sudoers BULK_DATA !log_output fleet-wide), drop >64 MiB streams from archives, prune existing bulk recordings 2026-10-04 22:55:49 -07:00
vh 73129a0564 ops(esh-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20 (one reboot, Prime on site); ESH cluster uniform on 9 2026-10-04 07:45:05 -07:00
vh 9a6b23581e ops(esh-pve): PVE 9 prep done (8.4.21, microcode, systemd-boot out, removable shim, NIC pins, vmbr0 bridge-ports fixed) 2026-10-04 07:32:13 -07:00
vh b67ab48833 ops(pfi-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20; reboot test + post-check recorded; pre-reboot checks corrected for PVE 9 2026-10-04 07:16:21 -07:00
vh 075ba0930e ops(pfi-pve): PVE 9 prep done (8.4.21, grub-efi-amd64 on the UEFI/LVM box, microcode, NIC pins); upgrade plan + playbooks 2026-10-04 06:11:13 -07:00
vh 453c927ace feat(qnap-lcd): Proxmox-correct front-panel LCD menu for esh-nas-pve; albok redeploy note 2026-10-03 16:14:43 -07:00
vh 241a668f3c ops(esh-nas-pve): PVE 9 upgrade playbook (snapshot, trixie sources, detached dist-upgrade) 2026-10-03 15:52:44 -07:00
vh f648e18b1a ops(esh-nas-pve): PVE 9 no-downtime prep (8.4.21, microcode, systemd-boot out, NIC pins) 2026-10-03 15:31:00 -07:00
vh d9e09fed14 ops(nh3-pve): PVE 9 post-upgrade fixes (7.0 headers + nvidia DKMS, microcode, enterprise repo off); old VM102 snapshot removed 2026-10-03 14:09:41 -07:00
vh 4b2a81bf39 feat(pve-nag-patch): remove the subscription popup on the PVE hosts, re-applied by an apt hook 2026-10-03 13:27:11 -07:00
vh 9b1cfa7344 feat(nh3-pve-2): drop stale 1 TB boot entry, wipe it into LVM-thin storage vmstore 2026-10-03 13:17:28 -07:00
vh 00a6fbd849 feat(cira-tunnel-watchdog): auto-drop stale AMT CIRA tunnels on MeshCentral's MPS 2026-10-03 12:34:35 -07:00
vh 8a4dbf2341 fix(albok-service): 0.1.2 on nh3-docker (fd leak); add restricted wing personal/agent-feedback; Nemi credentials 2026-10-03 02:54:18 -07:00
vh e707d87713 feat(esh-pve-2): drop stale 1 TB boot entry, wipe it into LVM-thin storage vmstore 2026-10-02 22:08:19 -07:00
vh d3958655c1 feat(albok-service): deploy the fleet knowledgebase service on nh3-docker
albok-service 0.1.0 (vh/albok 0b37431), image pfi/albok-service pinned by
digest, published on host port 8392 because 8390 is the post office.
Host prep playbook creates the fixed ids (albok 1500, albok-read 1510,
albok-personal 1511) and the local store/private roots; the container
gets a mounted /etc/group and group_add so the service can resolve and
chgrp its wing dirs. The config carries a LiteLLM key scoped to
qwen3-embedding and lives outside the deploy-synced conf dir. DNS name
albok.nh3.internal.
2026-10-02 17:56:34 -07:00
vh 6ec76bf35c ops(nh3-dev): grow root into the 378 GB disk online; swap moved to /swapfile
Prime grew VM 102's scsi0 from 250 to 378 GB after the root alert hit 85%
twice in 14 h. The swap partition sat right after sda1 and blocked growth,
so the playbook moves swap to a 4 GB /swapfile, deletes sda5/sda2, grows
sda1 in place (start sector unchanged) and ext4 online, and sets initramfs
RESUME=none so boots don't wait for the vanished swap. Root is 372 GB, 58%.
2026-10-02 07:33:36 -07:00
vh abcf4c3166 fix(scriberr): serve over HTTPS via the fleet TLS caddy so the browser recorder works
The in-browser recorder calls getUserMedia, which browsers refuse on http://
origins, so it sat at "Initializing recorder...". scriberr.nh3.phasefinal.com
is now fronted by the fleet TLS caddy on nh3-dev (wildcard cert) and added to
Scriberr's ALLOWED_ORIGINS; the Homepage link points at it. The plain
http://10.251.50.54:8080 URL keeps working except for recording.
2026-10-01 13:07:18 -07:00
vh 0b8632a7ed fix(esh-docker-vm): /32 route to Home Assistant over macvlan-shim
The shim holds 10.0.50.47/24, which gives two equal connected 10.0.50.0/24 routes,
and ens18's wins. Host-to-HA traffic therefore left via the macvlan parent and was
dropped. HA lost MQTT to the broker on this host on 2026-08-19, 2026-09-21 and
2026-09-25 (the last lasted two days). This adds an ifupdown if-up.d hook that
routes 10.0.50.46/32 via macvlan-shim; /etc/network/interfaces is not edited.
Verified: the route resolves via the shim, the host pings HA, HA reaches :1883,
and HA reconnected to the broker. Diagnosis by ha-dev.

Also corrects the zigbee2mqtt acceptance note, which had wrongly reported HA as
connected.
2026-09-27 12:50:05 -07:00
vh d775a01856 feat(bootstrap-infra-ops): create new accounts at the fleet-pinned uid/gid 850
docs/pfi/fleet-conventions.md § 3 pins infra-ops to 850, but the bootstrap
playbook let the OS allocate, so every host drifted (1001-2001). New
accounts now get 850 when uid and gid 850 are both free, and fall back to
OS allocation (with a note) when either is taken. The home is set to 0700
per § 1.2. Existing accounts are untouched (the step is gated on id).

Scope note: vm-esh-nas is added to the operator-granted ESH exceptions
(Prime, 2026-09-27).
2026-09-27 01:53:22 -07:00
vh 6e203dcb99 fix(restic): stop publishing rest-server passwords in systemd units
resticprofile schedule copies env-file values into the generated units, which
are 0644, so RESTIC_REPOSITORY (the rest-server basic-auth password included)
was readable by every local user on every restic host.

New playbooks/restic-repository-file.yaml:
- derives /etc/restic/repository (root 0400) from restic.env;
- uploads the profile switched to repository-file, but only when the live
  profile's sha matches the repo copy it was edited from (drift guard);
- checks the repository is reachable through the new profile (cat config);
- regenerates the units and verifies they exist and contain no rest:http.

Applied to ana-docker, fv-ml1 (configs/restic/ana-ml2), esh-docker-vm,
esh-vm-db, irv-ml1, nh3-dev and nh3-docker. An independent check across all
eight restic hosts (these seven plus esh-ml1) found 0 leaking units. nh3-docker's
scheduled unit ran a real backup afterwards (snapshot a29b889d). Each host's
URL and passphrase are vaulted as <host>/etc/restic/{repository,password}.

restic.env is kept (root 0600) because the per-host READMEs and the freshness
probe source it. A rotation must update the vault, restic.env and repository.

vm-esh-nas has no infra-ops account. Its in-place migration script is staged
for Prime to run with sudo, and its repo profile is pre-edited to match.

Also mirrors augaman-dev's 401df2d (compose header only). The config hash on
esh-ml1 is unchanged.
2026-09-27 01:51:05 -07:00
vh d8f59a15d9 feat(esh-ml1): restic backup of augaman's gallery; augaman v0.1.2
esh-ml1 is outside vzdump, so augaman's face gallery reaches backup only
through restic. New playbooks/esh-ml1-restic.yaml installs restic 0.14.0 (the
same Debian package as the other ESH hosts) and resticprofile 0.33.1 (pinned,
sha256-checked). It uploads configs/restic/esh-ml1/ and schedules a daily
0100 PT backup plus a Sunday 0500 PT check to rest-server-ana. The CT runs UTC,
so both schedules name the zone explicitly.

pre-backup.sh is fail-closed: it runs augaman's own backup CLI, and any failure,
including a stopped container, aborts the run. Tested with a stub docker that
exits 1: the run returned 1, and neither the snapshot count nor last-success
moved. The restore was verified at identity level against augaman-dev's
public-domain canary (snapshot fd3061a1: the restored copy's digest over
identities and samples matches the live gallery). That meets the operator gate
for real enrollments.

The repository URL is read through repository-file rather than restic.env.
resticprofile schedule copies env-file values into world-readable systemd
units, which publishes the rest-server password on the env-file hosts
(observed on esh-docker-vm). This is recorded in the backups runbook under
Known gaps, and the playbook verifies no generated unit contains the URL.

esh-ml1 is added to the freshness check's expected ana-side repos and to the
runbook tables.

augaman moves to v0.1.2 (dependency layer keyed on the lock without the
project; per-crop embedding). pytest -m gpu tests/vision passes 3/3 on the
card, and the canary survived the container recreate.
2026-09-27 00:06:33 -07:00
vh 8e7ae0675d feat(esh-matter): Matter server (matter.js 1.4.0) on a VLAN-90-only LXC for Home Assistant
For ha-dev (operator-approved 2026-09-26). CT 111 on esh-pve at 10.0.90.20:
Matter/Thread IPv6 (Echo ULA + RA route-information) is link-only, so the
server sits on esh-iot and HA reaches it over routed IPv4 ws :5580.
- playbooks/esh-matter-lxc.yaml: kernel RA (accept_ra=1,
  rt_info_max_plen=64), forwarding off, Docker ip-forward/iptables off;
  nftables admits 5580 from HA 10.0.50.46 only and SSH from mgmt ranges;
  the CT is added to esh-pve's vzdump job (fabric credentials).
- stacks/matter-server: ghcr.io/matter-js/matterjs-server:1.4.0 (digest),
  host networking, /data on the CT.
- Acceptance: fdad:: SLAAC, ping6 thermostat, 2 Thread RIO routes learned, ws
  server_info from inside the HA container; 5580 refused from 10.0.50.45,
  nh3-dev and a temporary VLAN 90 netns vantage.
2026-09-26 13:07:22 -07:00
vh 5960526c3f feat(nh3-ml1): second TEI embed/rerank backend live on nh3-pve; parity-verified vs esh-ml1
NH3 site visit done: Secure Boot off, iGPU restored as boot VGA, AMT port cabled.

- nh3-pve: NVIDIA 580.178.04 (DKMS, open modules) via pve-nvidia-host.yaml.
- nh3-ml1 = CT 109 @ 10.100.50.80 via gpu-lxc.yaml; embed-rerank (TEI 1.9.4)
  deployed with HOST_NAME/HOST_IP labels.
- Parity vs esh-ml1 (1,126 texts, 2 runs/host, controls): embed cosine min
  0.999993 = own noise floor; overlap@10 1.000 vs MRL-256 positive control
  0.684; rerank top-1 1.00, max diff 0.0014 vs floor 0.0020. On-box speed
  identical within rep spread.
- gpu-lxc.yaml: first step upgrades lxc-pve to >= 6.0.0-2 (Proxmox fix #7006).
  With 6.0.0-1 every docker run in a nesting CT failed on runc 1.5's sysctl
  reopen; applied on nh3-pve (one package).
- pve-nvidia-host.yaml: document that the headers meta drags in the newest
  kernel (nh3-pve went 6.8.12-11 -> -43 at the next reboot).
- Monitoring: Beszel NVIDIA agent + 5 alerts, Kuma #29/#30, Homepage
  nh3-ml1-docker, Dozzle agent (hub 8 clients). DNS nh3-ml1.nh3.internal.
- nh3-pve README: SB/IGFX/driver/kernel state, btmtk oops on -4x kernels,
  AMT cabled but unreachable on the network.

Gateway routing to nh3-ml1 is not changed.
2026-09-25 15:55:33 -07:00
vh 7ddd116e29 fix(pve-nvidia-host): refuse early on Secure Boot without an enrolled DKMS key; record nh3-pve SB=on 2026-09-25 14:22:47 -07:00
vh bc278d4ba8 refactor(playbooks): host-generic GPU host + GPU LXC playbooks for nh3-ml1
- esh-pve-nvidia-host -> pve-nvidia-host: headers/dkms/build-essential step,
  nouveau blacklist + guarded unload (refuses if nouveau bound a device)
- esh-ml1-lxc -> gpu-lxc: host vars have no defaults (elway aborts on undefined),
  rootfs storage/startup order parameterized, CT kept out of all-guests vzdump jobs
- embed-rerank: Homepage labels take HOST_NAME/HOST_IP, defaults = esh-ml1
2026-09-25 14:17:31 -07:00
vh 2118449881 feat(nh3-pve): prepare for GPU install — pin NIC names by MAC, pull AMT port from vmbr0
nh3-pve and esh-pve are the same Minisforum MS-01 (BIOS AHWSA.1.17). With a
card in the x16 slot its root port takes bus 01 and every NIC moves down a
bus (measured on esh-pve), so predictable names change (enp2s0f0np0 ->
enp3s0f0np0 etc.) and vmbr0 would boot with no uplink. systemd .link files
now pin all NICs by MAC, baked into every initramfs and synced to the ESP;
udev confirms the files apply. The AMT-capable I226-LM (enp88s0) leaves
vmbr0's bridge-ports in the file (next boot), so cabling it for AMT cannot
loop the STP-less bridge.

Also: documented the NanoKVM (https://10.100.250.171) as nh3-pve's console
OOB and that AMT is not wired; nh3-dev's Beszel agent no longer binds NAS
shares (it died on the last NH3 cold start); post-boot checklist in
persistent-memory.
2026-09-25 10:56:17 -07:00
vh 65dc586497 feat(esh-ml1): wire telemetry and monitoring; fix Dozzle's stale agent list
- Beszel: NVIDIA agent (stacks/beszel hosts/esh-ml1.yaml), hub system
  registered with Status/Disk/CPU/Memory/Temperature alerts; GPU samples
  verified (RTX 2000E Ada util, VRAM, power, temp).
- Uptime Kuma: health monitors for the TEI embed (:8001) and rerank (:8013)
  services — the sole backends behind the gateway, so the one exception to
  "seats are out of Kuma's lane". Reward seat excluded (no consumer).
- Homepage: dockerd on tcp/2375 bound to 10.0.50.80 (playbook step);
  esh-ml1-docker added to docker.yaml.
- Dozzle: agent v10.4.1 on esh-ml1. The hub's DOZZLE_REMOTE_AGENT still
  named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3; repointed
  to 10.251.50.54 / 10.6.110.50 (fv-ml1 and irv-ml1 logs were missing).
- Documented nh3-dev's Beszel agent failing on NH3 cold start (bind of an
  automounted NAS share); revived by hand, fix still open.
2026-09-25 09:20:36 -07:00
vh 7bdac80878 feat(embed-rerank): TEI is the fleet embed/rerank engine; esh-ml1 sole backend; retire fv-ml1 seats
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet
from now on; fv-ml1 retires both once esh-ml1 is up.

- stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports
  8001/8013, fail-closed truncation (--auto-truncate false; embed
  --max-batch-tokens 32768).
- litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address);
  reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's
  `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target.
- Verified via the gateway against the retiring fv-ml1 seats: embed cosine
  median 0.999927 (n=203); rerank top-1/top-3 29/30.
- stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on
  fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged.
- Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md;
  CLAUDE.md gains the TEI convention.
2026-09-25 08:30:53 -07:00
vh 5402568b76 feat(esh-ml1): RTX 2000E Ada on esh-pve serves embed + rerank as a LiteLLM failover
esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from
NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the
device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia`
retired. playbooks/esh-pve-nvidia-host.yaml.

esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN,
NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce +
nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the
vzdump job on purpose. DNS esh-ml1.esh.internal.

stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on
the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed
cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise
floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking.

litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2
behind fv-ml1 (order 1). Order fallback proven with throwaway groups:
refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500.

Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the
fv-ml1 relocation (still named 10.250.50.54); documented the third
unkillable homepage wedge on esh-docker-vm.
2026-09-24 22:38:55 -07:00
vh 1cbde502f4 fix(nh3): NFS clients mount nh3-nas shares on first access; record guest power-loss expectations
After the 2026-09-24 NH3 power outage every NFS line from nh3-nas failed at
boot: nh3-docker's /mnt/compose and /mnt/backup, and nh3-dev's /mnt/backup.
The NAS is the slowest box to serve NFS, and the plain fstab mounts stayed
down until remounted by hand. nh3-dev's /mnt/smithy already used
x-systemd.automount, and the same boot shows it self-healing on its next
access. playbooks/nh3-nfs-automount.yaml brings the other lines to that
shape (_netdev,nofail,x-systemd.automount,x-systemd.mount-timeout=30).
Hard-mount semantics are unchanged; only WHEN the mount happens moves.

Applied and read back on nh3-docker and nh3-dev: every nh3-nas line carries
automount, each automount unit is active, and each share is NFS-mounted after
access. A second run on nh3-docker is a no-op. The fstab check judges only
errors the rewrite ADDS, because nh3-dev's installer cdrom line has a
pre-existing verify error. The first run caught that and aborted without
writing.

servers/nh3-pve/README.md now lists each guest's expected state after a
power loss: nh3-laser is on-demand and stays off (Prime), pbs-nh3 now has
onboot=1 (it had none and stayed down), and pfi-gx10 is bare metal that
does not auto-power-on.
2026-09-24 13:16:18 -07:00
vh 466f7aa4e6 fix(elway): sudo uploads land root:root, validated and staged; add fleet ownership audit
elway's sudo upload did scp-as-user then `sudo mv`, and mv keeps the owner,
so every file it installed "as root" (systemd units, /etc configs, root-run
scripts) ended up owned by the SSH user. A sudoers drop-in installed that
way would be rejected by sudo outright.

elway:
- Sudo uploads now chown to root:root by default. Playbooks can override
  with `upload.owner:` and ad-hoc runs with `--owner`. An owner is refused
  on a non-sudo upload, and `--owner` outside an ad-hoc `--upload` is an
  error rather than silently ignored.
- Ownership and mode are applied to the STAGED file, then a single mv
  publishes it, so a failed chown can no longer leave the live path owned
  by the SSH user. chown runs before chmod so setuid bits survive. A trap
  removes the staged file on every exit path.
- A directory dest is refused before anything moves. mv would otherwise
  drop the file inside the directory under a staging name.
- `mode` was spliced unquoted into the remote root shell line. It is now
  validated as octal (^[0-7]{3,4}$) and shell-quoted, on both the sudo
  and non-sudo paths.
- `mode` and `owner` must be quoted YAML strings. YAML 1.1 turns a bare
  0644 into 420, a bare 1000:0 into 60000 and a bare yes into True, and
  str() of each is a different value that looks valid. All 85 playbooks
  already quote them.
- New preflight(): every step is resolved before any remote action. An
  undefined var or a templated owner that resolves badly is now refused
  up front, not after earlier steps have already changed the host.

scripts/fleet-ownership-audit.sh (new, read-only) finds files in root's
territory owned by a normal user. Tier A lists /etc, /usr/local, /root and
cron, judging symlinks by their target. Tier X covers files that root-run
units exec from /opt, /srv, /home or /volume1, reading drop-ins and quoted
paths. Tier B only summarises /opt app trees. A host that is unprivileged,
whose find fails, or whose run dies is reported INCOMPLETE (exit 5), never
clean. Completion is marked with a per-run nonce.

Verification: 24 unit tests in scripts/test_elway.py. Live checks on
esh-docker-vm: default lands root:root; the override lands infra-ops:adm at
4755 with setuid intact; a bad group fails with dest untouched and no
staging left; a directory dest is refused; a bad mode is refused before
any copy. Audit positive controls on nh3-dev (a drop-in-only root Exec, a
quoted path containing a space, a symlink to a user-owned target) were all
flagged; esh-docker-vm negative control was 0. Probes removed. Cross-model
bug-hunt (heid, Gróa arm + seat) findings folded.
2026-09-23 09:59:15 -07:00
vh 6e8da46a28 fix(restic/esh-docker-vm): stop dumping paperless's Postgres from this host
Operator decision. paperless-ngx's database lives on esh-vm-db and is backed
up at the source by that host's fail-closed pg_dumpall. esh-docker-vm's
second copy had failed auth every night since 2026-04-24 behind a WARN. Its
`> paperless.pg_dump` redirect left a 0-byte file in every snapshot
(confirmed in snapshot 6ec9f74f), which looked like a dump but held nothing.

The block was the only consumer of /etc/restic/dbcreds.env, so the creds
loader is gone and the template dbcreds.env.example is deleted. The host
file was moved (not deleted) to /var/lib/restic/repair-20260923/.
Paperless's media volumes are still captured under /var/lib/docker/volumes.

Also fixes ownership. elway's sudo upload does scp-as-user then `sudo mv`,
so the hook deployed at 08:04 landed infra-ops:infra-ops even though root
executes it. Both esh-docker-vm playbooks now chown it back to root and
verify root:root 700.

Verified: the live hook hash matches canonical (e0d3ddcef1bddf43), and
the manual backup saved snapshot decfae71 with 3 staged dumps and no WARN
lines.
2026-09-23 09:18:03 -07:00