From 10c0e342eec94cdc5b8cbb3d28e13cc7e4fc70c7 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Thu, 8 Oct 2026 07:35:01 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=202026-10-08=20p?= =?UTF-8?q?atch=20night,=20nh3-svos=20a737a06=20+=20post=20office=20live,?= =?UTF-8?q?=20albok=200.1.13/0.1.14,=20gateway=20glm-5.3,=20heid=20sweep?= =?UTF-8?q?=20incident;=2016=20entries=20archived?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- archival-memory.md | 272 ++++++++++++++++++ ...-on-nh3-dev-and-delegated-fleet-wide-to.md | 3 - ...6-09-23-elway-ownership-fix-fleet-audit.md | 33 --- ...tic-was-skipped-09-22-23-by-my-own-kuma.md | 3 - .../2026-09-23-headscale-ddns-hardened.md | 3 - ...art-exit-1-is-a-hermes-race-not-a-crash.md | 3 - .../2026-09-24-esh-ml1-embed-rerank.md | 80 ------ ...26-09-24-esh-pve-vm102-retired-gpu-swap.md | 40 --- .../2026-09-24-gx10-ac-restore-patch.md | 32 --- ...ing-order-is-a-repo-claude-md-operating.md | 3 - .../2026-09-24-nh3-power-outage-recovery.md | 31 -- ...7-albok-service-0-1-13-sensitivity-gate.md | 1 + .../2026-10-07-albok-service-0-1-14.md | 1 + ...2026-10-07-gateway-glm-5-3-gen-frontier.md | 1 + ...026-10-07-nh3-svos-refresh-a737a06-apps.md | 1 + .../2026-10-08-heid-runaway-sweep-killed.md | 16 ++ ...t-fv-ml1-kernel-nvidia-zfs-docker-hosts.md | 1 + persistent-memory.md | 61 ++-- 18 files changed, 322 insertions(+), 263 deletions(-) delete mode 100644 persistent-memory.d/2026-09-22-safe-rm-installed-on-nh3-dev-and-delegated-fleet-wide-to.md delete mode 100644 persistent-memory.d/2026-09-23-elway-ownership-fix-fleet-audit.md delete mode 100644 persistent-memory.d/2026-09-23-esh-docker-vm-restic-was-skipped-09-22-23-by-my-own-kuma.md delete mode 100644 persistent-memory.d/2026-09-23-headscale-ddns-hardened.md delete mode 100644 persistent-memory.d/2026-09-23-hermes-gateway-restart-exit-1-is-a-hermes-race-not-a-crash.md delete mode 100644 persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md delete mode 100644 persistent-memory.d/2026-09-24-esh-pve-vm102-retired-gpu-swap.md delete mode 100644 persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md delete mode 100644 persistent-memory.d/2026-09-24-miranda-standing-order-is-a-repo-claude-md-operating.md delete mode 100644 persistent-memory.d/2026-09-24-nh3-power-outage-recovery.md create mode 100644 persistent-memory.d/2026-10-07-albok-service-0-1-13-sensitivity-gate.md create mode 100644 persistent-memory.d/2026-10-07-albok-service-0-1-14.md create mode 100644 persistent-memory.d/2026-10-07-gateway-glm-5-3-gen-frontier.md create mode 100644 persistent-memory.d/2026-10-07-nh3-svos-refresh-a737a06-apps.md create mode 100644 persistent-memory.d/2026-10-08-heid-runaway-sweep-killed.md create mode 100644 persistent-memory.d/2026-10-08-patch-night-fv-ml1-kernel-nvidia-zfs-docker-hosts.md diff --git a/archival-memory.md b/archival-memory.md index 7944fb3..3b536b2 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -14735,3 +14735,275 @@ longer takes a full-card seat. That is the trade; it is the operator's call, and gateway should move with it either way. _Archived 2026-10-06._ +## Recent decisions (archived 2026-10-08 batch) + +# `[2026-09-24]` esh-ml1 built: RTX 2000E Ada serving embed + rerank, LiteLLM order-2 failover behind fv-ml1 + +Executes Prime's 2026-09-24 decision (LXC + host driver, not a VFIO VM). Done +2204–2240 PT the same night, **with no esh-pve reboot**. + +**Host (esh-pve).** NVIDIA **580.178.04** open modules via DKMS from NVIDIA's +`-no-compat32.run` (sha256 checked against NVIDIA's published sum), applied +live by `playbooks/pve-nvidia-host.yaml`: nouveau was never loaded and the +T400's vfio ids matched nothing, so nothing held the card. Our own +`nvidia-persistenced.service` creates `/dev/nvidia{0,ctl,-uvm,-uvm-tools}` +**before `pve-guests`** (else `pct start` refuses the `devN:` entries). Retired +`vfio.conf` (moved to `/root/nvidia/`) and `blacklist nvidia`. Initramfs NOT +rebuilt on purpose (boot path untouched). 580 because the fleet's vLLM +`v0.24.0` image is CUDA 13.0 (driver ≥ 580); fv-ml1 runs 580.65.06. `.run` +not an apt repo: one file on both sides makes the host-module/container-lib +version match true by construction; apt would let an upgrade move one side. + +**CT 110 `esh-ml1`** (`playbooks/gpu-lxc.yaml`): unprivileged Debian 12, +nesting+keyctl, 6c/16G/80G `local-lvm`, `10.0.50.80` VLAN 50 (static; UDM pool +is `.150–.250`), `startup order=30` after esh-scale/VMs. NVIDIA userspace from +the same `.run` with `--no-kernel-modules`; docker-ce; nvidia-container-toolkit +with `no-cgroups=true`. **Docker-in-unprivileged-LXC with the GPU worked first +try on lxc-pve 6.0.0-2** (no AppArmor workaround needed). Fleet ids per +`docs/pfi/fleet-conventions.md` (infra-ops 850, docker 851, vh 1000 — vh has +no key/password yet). **Not in the vzdump job, on purpose** (rebuildable). +DNS `esh-ml1.esh.internal` synced to all three resolvers. + +**Stack `stacks/embed-rerank`:** `vllm-embed` :8001 + `vllm-rerank-bge` :8013, +same models/flags/vLLM digest (`251eba5cc7c1`) as fv-ml1, 0.20 mem-util each → +4,823 MiB used of 16,380. + +**Parity (the load-bearing measurement; embeddings are model-specific).** 11 +texts × 2 runs per site. Embed cosine FV-vs-ESH median 0.999908 / min 0.999772, +inside both self-noise floors (FV-vs-FV 0.999927 / 0.999791; ESH-vs-ESH +0.999911 / 0.999809). Negative control (different texts) 0.07–0.38. Sensitivity +floor ~2×10⁻⁴ cosine. Rerank |score| max 0.000145 vs FV-FV floor 0.000181, +identical ranking. + +**Gateway.** `qwen3-embedding` and `reranker` each got a second deployment +(`order: 2`, esh-ml1); fv-ml1 is `order: 1`. LiteLLM v1.97 order-fallback +proven with throwaway groups (created + deleted): primary refusing → ESH in ++0.15 s (3/3); primary host-down (no ARP) → ESH but **~18.7 s on every call, +no cooldown** (3/3); dead-only negative control → 500 (3/3). Post-restart +production traffic is served by fv-ml1 (header `x-litellm-model-api-base`). +`/health?model=` returned 503 with empty lists for both groups — not +investigated. Gateway restart: liveliness back in ~52 s. + +**Found + fixed:** DB-only alias `reranker-a3-bge-v2-m3` pointed at ana-ml2's +pre-relocation IP `10.250.50.54` — dead since 2026-09-12 (500 after ~23 s), no +callers in 7-day spend logs. PATCHed to `10.251.50.54:8013`. Sweep of +`/model/info` found no other `10.250.50.54` targets. + +**Who actually uses these names (spend logs, 7 days to 2026-09-25 05:35Z):** +`qwen3-embedding` — worldtree-gateway 481 (nh3-dev + corviduo-dev), nevermore +20; `reranker` — nevermore 13. **No ESH-side consumer called either in 7 days** +(Open WebUI's RAG embeds only on document upload). So today esh-ml1's value is +an FV-outage failover for Worldtree + nevermore, not local service for ESH. + +**Open:** a direct ESH→esh-ml1 path (survives a mesh outage) would help only +ESH consumers, which are idle — recommended parked. esh-ml1 not yet in +Homepage docker.yaml or Beszel. The ~18.7 s host-down failover penalty is +untuned (router retries/cooldown). + +**Speed A/B vs fv-ml1 (2026-09-25 0628–0640 PT, table in `servers/esh-ml1/README.md`).** +Single queries: a wash (on-box ESH ~9 ms vs FV ~12 ms; from ana-docker ESH is +faster because it is network-closer). Bulk: ESH 3–10× slower (bulk embed ~50 vs +~403 passages/s; rerank-20 ~170 vs ~51 ms p50, ~5.7 vs ~55 req/s at conc 8). +Noise floor 3–5% on throughput. ⇒ failover role confirmed; load-sharing would +roughly double rerank latency for half the calls. + +**TEI bake-off (2026-09-25, `stacks/tei-bakeoff/README.md`).** Prime asked to +evaluate a purpose-built engine instead of vLLM → HF Text Embeddings Inference +1.9.4 (Infinity ruled out: last release 2025-08-22, dropped by us in May for no +Qwen3). Verdict: TEI **matches** (embed cosine vs vLLM-FV median 0.999925, inside +noise; TEI queries against the OLD vLLM index overlap@10 0.988 vs 0.986 self-noise +→ no re-embed needed; rerank decisions identical, tail scores move ≤0.019) but is +**not faster** on the Ada (novel 40 s vs 31 s; tiny queries ~25% faster). It is +**much lighter**: ~2.6 GB VRAM for both vs vLLM's 4.8 GB budget, 8 GB image vs 30, +~4 s restart vs ~24 s. Gateway: embed drop-in via `hosted_vllm/`; rerank needs the +`huggingface/` provider. Both engines left running side by side pending Prime's call. + _Archived 2026-10-08._ + +# `[2026-09-24]` esh-pve: VM 102 retired, a 14-day hung VFIO process found, T400 → RTX 2000 Ada + +**VM 102 `esh-vm-workstation` retired permanently** (Prime: "not the machine I +want running a virtual windows machine"). Final backup +`pbs-ana:backup/vm/102/2026-09-10T10:32:20Z` (verification ok) set +**protected** so prune cannot take it; `qm destroy 102 --purge`. + +⚠⚠ **Found: `kvm -id 102` had been hung in D state since the 2026-09-10 0332 +vzdump.** vzdump starts a stopped VM (here with the T400 passed through, +`x-vga=1`) to read its disks; that process never exited. It held **16 GB of +locked RAM**, the GPU, and two LVs, while `qm status 102` said `stopped` the +whole time. Same class as the 2026-08-19 hard freeze: VFIO + this box. Only a +reboot cleared it. + +**GPU swap, Prime on site, 2119–2137.** Guests shut down cleanly; host powered +off; the T400 was removed and an **NVIDIA RTX 2000 / 2000E Ada** installed +(`01:00.0`, `10de:28b0`, subsystem `10de:1871`). Host back 2136:49; leftover +LVs removed; all guests auto-started; esh-docker-vm NFS, traefik (0×403) and +AdGuard verified. The new card has **no driver bound**; `/etc/modprobe.d` still +lists the T400's vfio ids (`10de:1ff2,10de:10fa`), now matching nothing. +Commits `719e3fa`, `7cbd4c3`. + +**Decision (Prime, 2026-09-24): the card serves embedding + reranking as an +LXC with the NVIDIA driver on the host — NOT a VFIO VM.** Rationale: both +esh-pve hangs this year came from VFIO passthrough; LXC avoids it and pins no +VM RAM. Accepted cost: a driver on the hypervisor, rebuilt on PVE kernel +updates, host/container versions must match. **Implementation deferred to the +next session** (tracking: this entry + `servers/esh-pve/README.md`). + +**ESH single-route observation:** esh-scale (CT 108, ESH's only mesh subnet +router) lives on esh-pve, so while esh-pve was down *all* of ESH looked dark +from outside, including esh-pve-nas and vm-esh-nas, which were up. Untracked by +operator choice so far — a second ESH route is an idea, not a decision. + +**vPro:** the copper cable is in the AMT-capable port (**I226-LM**, `enp89s0`, +MAC `58:47:ca:76:99:32`), verified at 2500 Mb/s; the I226-V (`enp88s0`) is +not AMT-capable. Both links set back admin-down on the host. **MEBx +provisioning deferred by Prime** ("some other time"): Ctrl+P at boot, enable +manageability, static IP, KVM on with User Opt-in None, activate network +access; MeshCentral should live on ana-docker/nh3-docker, never on esh-pve. + _Archived 2026-10-08._ + +# `[2026-09-24]` pfi-gx10 "power on after AC loss" — UEFI patch applied, NOT validated + +pfi-gx10 (ASUS Ascent GX10, BIOS `GX10DGX.0105.2026.0505.1153`, no BMC) stayed +off after the NH3 outage. With Prime's approval (after the permission +classifier first refused it as externally-sourced firmware code), bytes 3–4 +of the 81-byte UEFI var `Setup-ec87d643-eba4-4bb5-a1e5-3f3e36b20da9` were +changed `0x1d,0x00 → 0x1c,0x01` via `efivar -w`; readback shows exactly those +two bytes changed. Recipe: community tool `Fheodoor01/DGX-Spark---Power-on-boot-` +(diff-derived from a power-on unit vs a stay-off unit; author-verified on +GX10DGX.0103/.0104). **Byte 3's meaning is unknown**; the resulting pair +matches factory power-on units. + +**Original saved twice:** `/var/lib/efi-setup-backup/Setup-ec87d643-20260924-pre-poweron.bin` +on the box and `servers/pfi-gx10/efi-Setup-ec87d643-20260924-pre-poweron.bin` +(sha256 `6d6f6f77…`, raw incl. the 4-byte attr header). Revert procedure in +`servers/pfi-gx10/README.md`. Commit `e5197a3`. + +⚠⚠ **An OS shutdown is NOT a test of this setting.** It means "start when +power RETURNS after a cut"; a `poweroff` is supposed to stay off by design. +The tool's README claims the two are indistinguishable — false here. I +trusted it and ran two poweroff "tests" (1421, and 1435 after the reboot the +README says the firmware needs); both stayed off, each costing Prime a trip to +the button, and both prove nothing. Prime called it out directly. + +**Status:** the box is OFF (the 1435 shutdown) with the patch in place, and +Prime left NH3. **He will pull and restore AC on 2026-09-25.** Comes up alone +→ works; drop "NOT YET VALIDATED" from the README. Stays off → he presses the +button, revert from the backup, and set it in firmware via a PiKVM (he intends +to add one with ATX control, which also covers remote power-on regardless). + +**Validated 2026-09-25 (Prime, relayed by Miranda 1333):** Prime replugged AC and +the box came up by itself at 1108:54 PT, with no button press. The patch works. + _Archived 2026-10-08._ + +# `[2026-09-24]` NH3 power outage — recovered; three latent gaps closed, one needs a hand + +Power at NH3 dropped and returned; every NH3 box booted at 1233–1234 PT. The +core recovered unaided (post office, headscale, egress proxy, DNS, WAN IP +unchanged at `70.230.226.88`). Three things did not, each a latent config gap +that only a site-wide power event exposes: + +1. **pbs-nh3 (VM 105) had no `onboot` flag** and stayed down. Set `onboot=1` + and started it; datastore NFS mounted (its line uses `bg`). `qm guest exec + 105` works (guest agent on) — the root path, since infra-ops is not + provisioned there. +2. **NFS boot race.** nh3-nas is the slowest box to serve NFS, so every plain + fstab NFS line failed at boot (nh3-docker `/mnt/compose` `/mnt/backup`, + nh3-dev `/mnt/backup`). nh3-dev's `/mnt/smithy` already had + `x-systemd.automount` and self-healed on its next access (1234 failed, 1309 + mounted, no hand) — the existence proof. `playbooks/nh3-nfs-automount.yaml` + brought the rest to that shape (`_netdev,nofail,x-systemd.automount, + x-systemd.mount-timeout=30`), commit `1cbde50`. ⚠ nh3-dev's installer + cdrom fstab line is a pre-existing `findmnt --verify` error; the playbook + judges only errors a rewrite ADDS, after its first run aborted on it. +3. **pfi-gx10 (bare metal, no BMC) did not power on** — ASUS ships "Restore AC + Power Loss" = stay off. See `[[2026-09-24-gx10-ac-restore-patch]]`. + +Also: **every Claude session on nh3-dev died** with the reboot except +infra-ops, infra-hermes, jekyll. Relaunch is Prime's call (dev-launch). +`nh3-laser` (VM 104) is on-demand and **stays off** by Prime's ruling; +`servers/nh3-pve/README.md` now carries an expected-after-power-loss column per +guest so the next triage does not flag it. + +The headscale-ddns failures 1141–1202 were the WAN being down, correctly +refused (no public v4); every run since 1247 reads `unchanged`. + _Archived 2026-10-08._ + +# `[2026-09-24]` Miranda standing order is a repo CLAUDE.md operating parameter + +**Miranda standing order is a repo CLAUDE.md operating parameter** (`4b29492`, aligned to the global send protocol in `bcf3342`): high-urgency matters go to her, fixed or not; URGENT only when it cannot wait (she phones Prime). Channel verified end to end (thread `01M3A0RP4Q8T0KNGH8TMFSNDA6`); it depends on svos + hermes-gateway. Prime's callsign **PRiMe / papa romeo mike** is a name, not an authenticator (`617b759`, `62817a2`). + _Archived 2026-10-08._ + +- `[2026-09-24]` **task-board mothballed** (Prime): container removed on ana-docker, data/image/compose kept, Kuma monitor deleted, task_* instructions removed from CLAUDE.md and the fork template (`e6da607`). Its hooks had sent no traffic in 30 days. + _Archived 2026-10-08._ + +- `[2026-09-24]` **Military 24-hour Pacific clock times** carried into the Codex/Grok shared bootstrap `docs/fleettools/AGENT-BOOTSTRAP.md` (`ad2b4d9`); Claude seats get it from the global CLAUDE.md. + _Archived 2026-10-08._ + +- `[2026-09-24]` **A git checkout under `root:docker` needs `safe.directory` for its deploy user** — the 09-14 normalization (`826a63b`) silently broke yt-voice-clipper's webhook deploy until v0.3.13; fixed on irv-ml1 and recorded in fleet conventions (`eea9eb2`). Sweep found no other case. + _Archived 2026-10-08._ + +# `[2026-09-23]` elway sudo uploads land root:root; fleet ownership audit; 33 files fixed + +elway's sudo upload was scp-as-user then `sudo mv`, and mv keeps the owner, so +every file it installed as root (systemd units, `/etc` configs, root-run +scripts) landed owned by infra-ops or lkraven. A sudoers drop-in installed that +way would break sudo outright. Commit `466f7aa`: + +- sudo uploads chown to `root:root` by default; `upload.owner:` / `--owner` + override; owner refused on non-sudo uploads. +- chown+chmod act on the STAGED file, then one `mv` publishes it, so a failed + chown cannot leave the live path mis-owned. A trap removes the staged file on + every exit. A directory dest is refused before anything moves. +- `mode` was spliced unquoted into the remote root shell; now octal-validated + and quoted. `mode`/`owner` must be quoted YAML strings (bare `0644` → 420). +- `preflight()` resolves every step before any remote action. +- 24 unit tests; heid bug-hunt "Puck" (Gróa + seat) folded. + +`scripts/fleet-ownership-audit.sh` (read-only; exit 0/4/5): tier A root-parsed +paths (symlinks judged by target), tier X root-run unit Exec paths incl. +drop-ins, tier B /opt summary. Unprivileged or partial hosts report INCOMPLETE, +never clean; completion is nonce-marked. Positive controls on nh3-dev fired for +each tier. **Six hosts have no infra-ops identity** (ana-wg, nh3-nas, pbs-ana, +pbs-nh3, pfi-postgres, vm-esh-nas) and audit unprivileged. + +**33 files chowned to root on 10 hosts** with Prime's approval (the first +attempt was blocked by the permission classifier): 31 elway-placed (beszel +agent trio on 6 hosts, esh-vm-db/ana-docker restic hooks and retry drop-ins, +irv-ml1 units, nh3-dev alert bridge, fv-ml1 retired file) plus 2 non-elway +root-exec holes (ana-docker `/usr/local/bin/resticprofile` owned by llmuser; +esh-pve-nas `/usr/local/lib/libigdgmm.so.11.3.1343` uid 1000). Re-audit: 0 of +the 33 remain; all affected services still active. Left as low-risk and +reported: ana-docker `/root/.nvm` (lkraven), irv-ml1 `uv`/`uvx`, fv-ml1 +`btop` and a root uv-cache python. + _Archived 2026-10-08._ + +# `[2026-09-23]` headscale-ddns hardened + +**headscale-ddns hardened** (`fedd4b6`): Cloudflare calls retry and validate every body (`pick()`), no write without both IDs, the run ends on a confirmation, only a global v4 is published, `curl -q`. Three of four weekly failures were empty zone lookups. heid bug-hunt "Talus" folded (one finding was my own regression). + _Archived 2026-10-08._ + +# `[2026-09-23]` esh-docker-vm restic was skipped 09-22..23 by my own Kuma move + +**esh-docker-vm restic was skipped 09-22..23 by my own Kuma move** — a dead uptime-kuma lookup aborted `pre-backup.sh` under `set -e` (`25e41d2`). Then, by Prime's decision, the redundant Paperless pg_dump went too: it had failed auth every night since 2026-04-24 and left a 0-byte dump in every snapshot; the DB is covered at source by esh-vm-db's pg_dumpall (`6e8da46`). + _Archived 2026-10-08._ + +# `[2026-09-23]` hermes-gateway restart exit-1 is a Hermes race, not a crash + +**hermes-gateway restart exit-1 is a Hermes race, not a crash** — the planned-stop watcher consumes the marker before SIGTERM re-runs the handler; drop-in `SuccessExitStatus=1` stops the false OnFailure page at zero coverage cost (`c2b0a05`). svos had its own stop-timeout (an open board SSE tab), fixed by svos-dev with `timeout_graceful_shutdown=5`. + _Archived 2026-10-08._ + +- `[2026-09-23]` **Booth link board cleared to 14 durable links** (208 removed: drops, release posts, research links, token-bearing URLs; Scriberr reposted at `scriberr.fv.internal:8080`). Prime's rule: durable debug links only. + _Archived 2026-10-08._ + +# `[2026-09-22]` safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes. + +⭐ **safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes.** ⚠ The package installs INERT and looks fine — Debian's `/etc/zsh/zprofile` has 0 non-comment lines so the shipped `profile.d` hook never fires under zsh; verify with `command -v rm`, never `dpkg -l`. ⚠ And it does NOT cover the habit that prompted it: measured, it refuses `rm -rf /home` and deletes an unset-variable path without complaint. Blacklist, not heuristic; `set -u` is the actual cover. Wiring ruled: `/etc/bash.bashrc` above the `case $-` guard, not `/etc/environment` (an rc self-guards, smaller blast radius, covers bash-over-ssh). + _Archived 2026-10-08._ + +## Tried and abandoned (archived 2026-10-08 batch) + +- `[2026-09-24]` **Testing "Restore AC Power Loss" with an OS shutdown** — a shutdown stays off BY DESIGN; only pulling and restoring AC tests it. A community README claimed the two are indistinguishable; trusting it cost Prime two trips to the pfi-gx10 power button. + _Archived 2026-10-08._ + +- `[2026-09-23]` **`booth link --help`** — there is no help flag; it posts `--help` to the operator's link board as a link. Read `booth` with no args for usage. + _Archived 2026-10-08._ diff --git a/persistent-memory.d/2026-09-22-safe-rm-installed-on-nh3-dev-and-delegated-fleet-wide-to.md b/persistent-memory.d/2026-09-22-safe-rm-installed-on-nh3-dev-and-delegated-fleet-wide-to.md deleted file mode 100644 index 092f311..0000000 --- a/persistent-memory.d/2026-09-22-safe-rm-installed-on-nh3-dev-and-delegated-fleet-wide-to.md +++ /dev/null @@ -1,3 +0,0 @@ -# `[2026-09-22]` safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes. - -⭐ **safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes.** ⚠ The package installs INERT and looks fine — Debian's `/etc/zsh/zprofile` has 0 non-comment lines so the shipped `profile.d` hook never fires under zsh; verify with `command -v rm`, never `dpkg -l`. ⚠ And it does NOT cover the habit that prompted it: measured, it refuses `rm -rf /home` and deletes an unset-variable path without complaint. Blacklist, not heuristic; `set -u` is the actual cover. Wiring ruled: `/etc/bash.bashrc` above the `case $-` guard, not `/etc/environment` (an rc self-guards, smaller blast radius, covers bash-over-ssh). diff --git a/persistent-memory.d/2026-09-23-elway-ownership-fix-fleet-audit.md b/persistent-memory.d/2026-09-23-elway-ownership-fix-fleet-audit.md deleted file mode 100644 index 18dffdf..0000000 --- a/persistent-memory.d/2026-09-23-elway-ownership-fix-fleet-audit.md +++ /dev/null @@ -1,33 +0,0 @@ -# `[2026-09-23]` elway sudo uploads land root:root; fleet ownership audit; 33 files fixed - -elway's sudo upload was scp-as-user then `sudo mv`, and mv keeps the owner, so -every file it installed as root (systemd units, `/etc` configs, root-run -scripts) landed owned by infra-ops or lkraven. A sudoers drop-in installed that -way would break sudo outright. Commit `466f7aa`: - -- sudo uploads chown to `root:root` by default; `upload.owner:` / `--owner` - override; owner refused on non-sudo uploads. -- chown+chmod act on the STAGED file, then one `mv` publishes it, so a failed - chown cannot leave the live path mis-owned. A trap removes the staged file on - every exit. A directory dest is refused before anything moves. -- `mode` was spliced unquoted into the remote root shell; now octal-validated - and quoted. `mode`/`owner` must be quoted YAML strings (bare `0644` → 420). -- `preflight()` resolves every step before any remote action. -- 24 unit tests; heid bug-hunt "Puck" (Gróa + seat) folded. - -`scripts/fleet-ownership-audit.sh` (read-only; exit 0/4/5): tier A root-parsed -paths (symlinks judged by target), tier X root-run unit Exec paths incl. -drop-ins, tier B /opt summary. Unprivileged or partial hosts report INCOMPLETE, -never clean; completion is nonce-marked. Positive controls on nh3-dev fired for -each tier. **Six hosts have no infra-ops identity** (ana-wg, nh3-nas, pbs-ana, -pbs-nh3, pfi-postgres, vm-esh-nas) and audit unprivileged. - -**33 files chowned to root on 10 hosts** with Prime's approval (the first -attempt was blocked by the permission classifier): 31 elway-placed (beszel -agent trio on 6 hosts, esh-vm-db/ana-docker restic hooks and retry drop-ins, -irv-ml1 units, nh3-dev alert bridge, fv-ml1 retired file) plus 2 non-elway -root-exec holes (ana-docker `/usr/local/bin/resticprofile` owned by llmuser; -esh-pve-nas `/usr/local/lib/libigdgmm.so.11.3.1343` uid 1000). Re-audit: 0 of -the 33 remain; all affected services still active. Left as low-risk and -reported: ana-docker `/root/.nvm` (lkraven), irv-ml1 `uv`/`uvx`, fv-ml1 -`btop` and a root uv-cache python. diff --git a/persistent-memory.d/2026-09-23-esh-docker-vm-restic-was-skipped-09-22-23-by-my-own-kuma.md b/persistent-memory.d/2026-09-23-esh-docker-vm-restic-was-skipped-09-22-23-by-my-own-kuma.md deleted file mode 100644 index b9f3c66..0000000 --- a/persistent-memory.d/2026-09-23-esh-docker-vm-restic-was-skipped-09-22-23-by-my-own-kuma.md +++ /dev/null @@ -1,3 +0,0 @@ -# `[2026-09-23]` esh-docker-vm restic was skipped 09-22..23 by my own Kuma move - -**esh-docker-vm restic was skipped 09-22..23 by my own Kuma move** — a dead uptime-kuma lookup aborted `pre-backup.sh` under `set -e` (`25e41d2`). Then, by Prime's decision, the redundant Paperless pg_dump went too: it had failed auth every night since 2026-04-24 and left a 0-byte dump in every snapshot; the DB is covered at source by esh-vm-db's pg_dumpall (`6e8da46`). diff --git a/persistent-memory.d/2026-09-23-headscale-ddns-hardened.md b/persistent-memory.d/2026-09-23-headscale-ddns-hardened.md deleted file mode 100644 index f5e8eb0..0000000 --- a/persistent-memory.d/2026-09-23-headscale-ddns-hardened.md +++ /dev/null @@ -1,3 +0,0 @@ -# `[2026-09-23]` headscale-ddns hardened - -**headscale-ddns hardened** (`fedd4b6`): Cloudflare calls retry and validate every body (`pick()`), no write without both IDs, the run ends on a confirmation, only a global v4 is published, `curl -q`. Three of four weekly failures were empty zone lookups. heid bug-hunt "Talus" folded (one finding was my own regression). diff --git a/persistent-memory.d/2026-09-23-hermes-gateway-restart-exit-1-is-a-hermes-race-not-a-crash.md b/persistent-memory.d/2026-09-23-hermes-gateway-restart-exit-1-is-a-hermes-race-not-a-crash.md deleted file mode 100644 index 93b1754..0000000 --- a/persistent-memory.d/2026-09-23-hermes-gateway-restart-exit-1-is-a-hermes-race-not-a-crash.md +++ /dev/null @@ -1,3 +0,0 @@ -# `[2026-09-23]` hermes-gateway restart exit-1 is a Hermes race, not a crash - -**hermes-gateway restart exit-1 is a Hermes race, not a crash** — the planned-stop watcher consumes the marker before SIGTERM re-runs the handler; drop-in `SuccessExitStatus=1` stops the false OnFailure page at zero coverage cost (`c2b0a05`). svos had its own stop-timeout (an open board SSE tab), fixed by svos-dev with `timeout_graceful_shutdown=5`. diff --git a/persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md b/persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md deleted file mode 100644 index 34de46b..0000000 --- a/persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md +++ /dev/null @@ -1,80 +0,0 @@ -# `[2026-09-24]` esh-ml1 built: RTX 2000E Ada serving embed + rerank, LiteLLM order-2 failover behind fv-ml1 - -Executes Prime's 2026-09-24 decision (LXC + host driver, not a VFIO VM). Done -2204–2240 PT the same night, **with no esh-pve reboot**. - -**Host (esh-pve).** NVIDIA **580.178.04** open modules via DKMS from NVIDIA's -`-no-compat32.run` (sha256 checked against NVIDIA's published sum), applied -live by `playbooks/pve-nvidia-host.yaml`: nouveau was never loaded and the -T400's vfio ids matched nothing, so nothing held the card. Our own -`nvidia-persistenced.service` creates `/dev/nvidia{0,ctl,-uvm,-uvm-tools}` -**before `pve-guests`** (else `pct start` refuses the `devN:` entries). Retired -`vfio.conf` (moved to `/root/nvidia/`) and `blacklist nvidia`. Initramfs NOT -rebuilt on purpose (boot path untouched). 580 because the fleet's vLLM -`v0.24.0` image is CUDA 13.0 (driver ≥ 580); fv-ml1 runs 580.65.06. `.run` -not an apt repo: one file on both sides makes the host-module/container-lib -version match true by construction; apt would let an upgrade move one side. - -**CT 110 `esh-ml1`** (`playbooks/gpu-lxc.yaml`): unprivileged Debian 12, -nesting+keyctl, 6c/16G/80G `local-lvm`, `10.0.50.80` VLAN 50 (static; UDM pool -is `.150–.250`), `startup order=30` after esh-scale/VMs. NVIDIA userspace from -the same `.run` with `--no-kernel-modules`; docker-ce; nvidia-container-toolkit -with `no-cgroups=true`. **Docker-in-unprivileged-LXC with the GPU worked first -try on lxc-pve 6.0.0-2** (no AppArmor workaround needed). Fleet ids per -`docs/pfi/fleet-conventions.md` (infra-ops 850, docker 851, vh 1000 — vh has -no key/password yet). **Not in the vzdump job, on purpose** (rebuildable). -DNS `esh-ml1.esh.internal` synced to all three resolvers. - -**Stack `stacks/embed-rerank`:** `vllm-embed` :8001 + `vllm-rerank-bge` :8013, -same models/flags/vLLM digest (`251eba5cc7c1`) as fv-ml1, 0.20 mem-util each → -4,823 MiB used of 16,380. - -**Parity (the load-bearing measurement; embeddings are model-specific).** 11 -texts × 2 runs per site. Embed cosine FV-vs-ESH median 0.999908 / min 0.999772, -inside both self-noise floors (FV-vs-FV 0.999927 / 0.999791; ESH-vs-ESH -0.999911 / 0.999809). Negative control (different texts) 0.07–0.38. Sensitivity -floor ~2×10⁻⁴ cosine. Rerank |score| max 0.000145 vs FV-FV floor 0.000181, -identical ranking. - -**Gateway.** `qwen3-embedding` and `reranker` each got a second deployment -(`order: 2`, esh-ml1); fv-ml1 is `order: 1`. LiteLLM v1.97 order-fallback -proven with throwaway groups (created + deleted): primary refusing → ESH in -+0.15 s (3/3); primary host-down (no ARP) → ESH but **~18.7 s on every call, -no cooldown** (3/3); dead-only negative control → 500 (3/3). Post-restart -production traffic is served by fv-ml1 (header `x-litellm-model-api-base`). -`/health?model=` returned 503 with empty lists for both groups — not -investigated. Gateway restart: liveliness back in ~52 s. - -**Found + fixed:** DB-only alias `reranker-a3-bge-v2-m3` pointed at ana-ml2's -pre-relocation IP `10.250.50.54` — dead since 2026-09-12 (500 after ~23 s), no -callers in 7-day spend logs. PATCHed to `10.251.50.54:8013`. Sweep of -`/model/info` found no other `10.250.50.54` targets. - -**Who actually uses these names (spend logs, 7 days to 2026-09-25 05:35Z):** -`qwen3-embedding` — worldtree-gateway 481 (nh3-dev + corviduo-dev), nevermore -20; `reranker` — nevermore 13. **No ESH-side consumer called either in 7 days** -(Open WebUI's RAG embeds only on document upload). So today esh-ml1's value is -an FV-outage failover for Worldtree + nevermore, not local service for ESH. - -**Open:** a direct ESH→esh-ml1 path (survives a mesh outage) would help only -ESH consumers, which are idle — recommended parked. esh-ml1 not yet in -Homepage docker.yaml or Beszel. The ~18.7 s host-down failover penalty is -untuned (router retries/cooldown). - -**Speed A/B vs fv-ml1 (2026-09-25 0628–0640 PT, table in `servers/esh-ml1/README.md`).** -Single queries: a wash (on-box ESH ~9 ms vs FV ~12 ms; from ana-docker ESH is -faster because it is network-closer). Bulk: ESH 3–10× slower (bulk embed ~50 vs -~403 passages/s; rerank-20 ~170 vs ~51 ms p50, ~5.7 vs ~55 req/s at conc 8). -Noise floor 3–5% on throughput. ⇒ failover role confirmed; load-sharing would -roughly double rerank latency for half the calls. - -**TEI bake-off (2026-09-25, `stacks/tei-bakeoff/README.md`).** Prime asked to -evaluate a purpose-built engine instead of vLLM → HF Text Embeddings Inference -1.9.4 (Infinity ruled out: last release 2025-08-22, dropped by us in May for no -Qwen3). Verdict: TEI **matches** (embed cosine vs vLLM-FV median 0.999925, inside -noise; TEI queries against the OLD vLLM index overlap@10 0.988 vs 0.986 self-noise -→ no re-embed needed; rerank decisions identical, tail scores move ≤0.019) but is -**not faster** on the Ada (novel 40 s vs 31 s; tiny queries ~25% faster). It is -**much lighter**: ~2.6 GB VRAM for both vs vLLM's 4.8 GB budget, 8 GB image vs 30, -~4 s restart vs ~24 s. Gateway: embed drop-in via `hosted_vllm/`; rerank needs the -`huggingface/` provider. Both engines left running side by side pending Prime's call. diff --git a/persistent-memory.d/2026-09-24-esh-pve-vm102-retired-gpu-swap.md b/persistent-memory.d/2026-09-24-esh-pve-vm102-retired-gpu-swap.md deleted file mode 100644 index fe4b5c3..0000000 --- a/persistent-memory.d/2026-09-24-esh-pve-vm102-retired-gpu-swap.md +++ /dev/null @@ -1,40 +0,0 @@ -# `[2026-09-24]` esh-pve: VM 102 retired, a 14-day hung VFIO process found, T400 → RTX 2000 Ada - -**VM 102 `esh-vm-workstation` retired permanently** (Prime: "not the machine I -want running a virtual windows machine"). Final backup -`pbs-ana:backup/vm/102/2026-09-10T10:32:20Z` (verification ok) set -**protected** so prune cannot take it; `qm destroy 102 --purge`. - -⚠⚠ **Found: `kvm -id 102` had been hung in D state since the 2026-09-10 0332 -vzdump.** vzdump starts a stopped VM (here with the T400 passed through, -`x-vga=1`) to read its disks; that process never exited. It held **16 GB of -locked RAM**, the GPU, and two LVs, while `qm status 102` said `stopped` the -whole time. Same class as the 2026-08-19 hard freeze: VFIO + this box. Only a -reboot cleared it. - -**GPU swap, Prime on site, 2119–2137.** Guests shut down cleanly; host powered -off; the T400 was removed and an **NVIDIA RTX 2000 / 2000E Ada** installed -(`01:00.0`, `10de:28b0`, subsystem `10de:1871`). Host back 2136:49; leftover -LVs removed; all guests auto-started; esh-docker-vm NFS, traefik (0×403) and -AdGuard verified. The new card has **no driver bound**; `/etc/modprobe.d` still -lists the T400's vfio ids (`10de:1ff2,10de:10fa`), now matching nothing. -Commits `719e3fa`, `7cbd4c3`. - -**Decision (Prime, 2026-09-24): the card serves embedding + reranking as an -LXC with the NVIDIA driver on the host — NOT a VFIO VM.** Rationale: both -esh-pve hangs this year came from VFIO passthrough; LXC avoids it and pins no -VM RAM. Accepted cost: a driver on the hypervisor, rebuilt on PVE kernel -updates, host/container versions must match. **Implementation deferred to the -next session** (tracking: this entry + `servers/esh-pve/README.md`). - -**ESH single-route observation:** esh-scale (CT 108, ESH's only mesh subnet -router) lives on esh-pve, so while esh-pve was down *all* of ESH looked dark -from outside, including esh-pve-nas and vm-esh-nas, which were up. Untracked by -operator choice so far — a second ESH route is an idea, not a decision. - -**vPro:** the copper cable is in the AMT-capable port (**I226-LM**, `enp89s0`, -MAC `58:47:ca:76:99:32`), verified at 2500 Mb/s; the I226-V (`enp88s0`) is -not AMT-capable. Both links set back admin-down on the host. **MEBx -provisioning deferred by Prime** ("some other time"): Ctrl+P at boot, enable -manageability, static IP, KVM on with User Opt-in None, activate network -access; MeshCentral should live on ana-docker/nh3-docker, never on esh-pve. diff --git a/persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md b/persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md deleted file mode 100644 index 69aeeda..0000000 --- a/persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md +++ /dev/null @@ -1,32 +0,0 @@ -# `[2026-09-24]` pfi-gx10 "power on after AC loss" — UEFI patch applied, NOT validated - -pfi-gx10 (ASUS Ascent GX10, BIOS `GX10DGX.0105.2026.0505.1153`, no BMC) stayed -off after the NH3 outage. With Prime's approval (after the permission -classifier first refused it as externally-sourced firmware code), bytes 3–4 -of the 81-byte UEFI var `Setup-ec87d643-eba4-4bb5-a1e5-3f3e36b20da9` were -changed `0x1d,0x00 → 0x1c,0x01` via `efivar -w`; readback shows exactly those -two bytes changed. Recipe: community tool `Fheodoor01/DGX-Spark---Power-on-boot-` -(diff-derived from a power-on unit vs a stay-off unit; author-verified on -GX10DGX.0103/.0104). **Byte 3's meaning is unknown**; the resulting pair -matches factory power-on units. - -**Original saved twice:** `/var/lib/efi-setup-backup/Setup-ec87d643-20260924-pre-poweron.bin` -on the box and `servers/pfi-gx10/efi-Setup-ec87d643-20260924-pre-poweron.bin` -(sha256 `6d6f6f77…`, raw incl. the 4-byte attr header). Revert procedure in -`servers/pfi-gx10/README.md`. Commit `e5197a3`. - -⚠⚠ **An OS shutdown is NOT a test of this setting.** It means "start when -power RETURNS after a cut"; a `poweroff` is supposed to stay off by design. -The tool's README claims the two are indistinguishable — false here. I -trusted it and ran two poweroff "tests" (1421, and 1435 after the reboot the -README says the firmware needs); both stayed off, each costing Prime a trip to -the button, and both prove nothing. Prime called it out directly. - -**Status:** the box is OFF (the 1435 shutdown) with the patch in place, and -Prime left NH3. **He will pull and restore AC on 2026-09-25.** Comes up alone -→ works; drop "NOT YET VALIDATED" from the README. Stays off → he presses the -button, revert from the backup, and set it in firmware via a PiKVM (he intends -to add one with ATX control, which also covers remote power-on regardless). - -**Validated 2026-09-25 (Prime, relayed by Miranda 1333):** Prime replugged AC and -the box came up by itself at 1108:54 PT, with no button press. The patch works. diff --git a/persistent-memory.d/2026-09-24-miranda-standing-order-is-a-repo-claude-md-operating.md b/persistent-memory.d/2026-09-24-miranda-standing-order-is-a-repo-claude-md-operating.md deleted file mode 100644 index 39f6179..0000000 --- a/persistent-memory.d/2026-09-24-miranda-standing-order-is-a-repo-claude-md-operating.md +++ /dev/null @@ -1,3 +0,0 @@ -# `[2026-09-24]` Miranda standing order is a repo CLAUDE.md operating parameter - -**Miranda standing order is a repo CLAUDE.md operating parameter** (`4b29492`, aligned to the global send protocol in `bcf3342`): high-urgency matters go to her, fixed or not; URGENT only when it cannot wait (she phones Prime). Channel verified end to end (thread `01M3A0RP4Q8T0KNGH8TMFSNDA6`); it depends on svos + hermes-gateway. Prime's callsign **PRiMe / papa romeo mike** is a name, not an authenticator (`617b759`, `62817a2`). diff --git a/persistent-memory.d/2026-09-24-nh3-power-outage-recovery.md b/persistent-memory.d/2026-09-24-nh3-power-outage-recovery.md deleted file mode 100644 index 9a456aa..0000000 --- a/persistent-memory.d/2026-09-24-nh3-power-outage-recovery.md +++ /dev/null @@ -1,31 +0,0 @@ -# `[2026-09-24]` NH3 power outage — recovered; three latent gaps closed, one needs a hand - -Power at NH3 dropped and returned; every NH3 box booted at 1233–1234 PT. The -core recovered unaided (post office, headscale, egress proxy, DNS, WAN IP -unchanged at `70.230.226.88`). Three things did not, each a latent config gap -that only a site-wide power event exposes: - -1. **pbs-nh3 (VM 105) had no `onboot` flag** and stayed down. Set `onboot=1` - and started it; datastore NFS mounted (its line uses `bg`). `qm guest exec - 105` works (guest agent on) — the root path, since infra-ops is not - provisioned there. -2. **NFS boot race.** nh3-nas is the slowest box to serve NFS, so every plain - fstab NFS line failed at boot (nh3-docker `/mnt/compose` `/mnt/backup`, - nh3-dev `/mnt/backup`). nh3-dev's `/mnt/smithy` already had - `x-systemd.automount` and self-healed on its next access (1234 failed, 1309 - mounted, no hand) — the existence proof. `playbooks/nh3-nfs-automount.yaml` - brought the rest to that shape (`_netdev,nofail,x-systemd.automount, - x-systemd.mount-timeout=30`), commit `1cbde50`. ⚠ nh3-dev's installer - cdrom fstab line is a pre-existing `findmnt --verify` error; the playbook - judges only errors a rewrite ADDS, after its first run aborted on it. -3. **pfi-gx10 (bare metal, no BMC) did not power on** — ASUS ships "Restore AC - Power Loss" = stay off. See `[[2026-09-24-gx10-ac-restore-patch]]`. - -Also: **every Claude session on nh3-dev died** with the reboot except -infra-ops, infra-hermes, jekyll. Relaunch is Prime's call (dev-launch). -`nh3-laser` (VM 104) is on-demand and **stays off** by Prime's ruling; -`servers/nh3-pve/README.md` now carries an expected-after-power-loss column per -guest so the next triage does not flag it. - -The headscale-ddns failures 1141–1202 were the WAN being down, correctly -refused (no public v4); every run since 1247 reads `unchanged`. diff --git a/persistent-memory.d/2026-10-07-albok-service-0-1-13-sensitivity-gate.md b/persistent-memory.d/2026-10-07-albok-service-0-1-13-sensitivity-gate.md new file mode 100644 index 0000000..bc45e4b --- /dev/null +++ b/persistent-memory.d/2026-10-07-albok-service-0-1-13-sensitivity-gate.md @@ -0,0 +1 @@ +- `[2026-10-07]` **albok-service 0.1.13 (unit 4.2, the sensitivity gate) deployed on nh3-albok at 1135 at albok-dev's ask 8413.** Rollback is NOT free: the first-start scan writes `sensitivity` sidecars, which 0.1.12 cannot load, so rolling back needs the strip snippet first. The census found `leaks` 8: six hold the shared all-agents key, two are false positives on a doc placeholder, nothing to rotate. → `stacks/albok-service/README.md` diff --git a/persistent-memory.d/2026-10-07-albok-service-0-1-14.md b/persistent-memory.d/2026-10-07-albok-service-0-1-14.md new file mode 100644 index 0000000..a03e570 --- /dev/null +++ b/persistent-memory.d/2026-10-07-albok-service-0-1-14.md @@ -0,0 +1 @@ +- `[2026-10-07]` **albok-service 0.1.14 (unit 4.3: sidecar listing, byte-less ingest, a `.browse/` view; detector 2) deployed at 1804 at albok-dev's ask 8524.** `leaks` went from 8 to 6 when the placeholder false positives cleared. Reindexes ran one per wing afterwards to render the view. ⚠ **OWED: the `.browse/` exposure test.** Pick a share surface (sshfs `follow_symlinks` is the lead), rename `.browse/` on the server, and check that a fresh open through the client's cached listing FAILS. Nothing is shared until it passes; report to albok-dev. → `stacks/albok-service/README.md` diff --git a/persistent-memory.d/2026-10-07-gateway-glm-5-3-gen-frontier.md b/persistent-memory.d/2026-10-07-gateway-glm-5-3-gen-frontier.md new file mode 100644 index 0000000..b2e255e --- /dev/null +++ b/persistent-memory.d/2026-10-07-gateway-glm-5-3-gen-frontier.md @@ -0,0 +1 @@ +- `[2026-10-07]` **Gateway: glm-5.3 added; gen-frontier / gen-frontier-reasoning moved 5.2 → 5.3 (Prime); every GLM 5.2/5.3 entry advertises 1,048,576 in / 131,072 out with max_tokens 131,072 as the default.** 5.3 cannot turn reasoning off, so the gen-frontier pair is now effort low / high (measured ~260 vs ~1,490 reasoning tokens). ⚠ z.ai's `thinking: disabled` is light thinking, not zero, on 5.1/5.2/5.3 alike (220–340 reasoning tokens on a hard prompt). Restarted 1725 (61 s), held 2 h for albok-dev's load A/B. The gate script's first two runs could never pass: `grep -c` exits 1 on a zero count. Images: glm-5.x is text only (z.ai rejects image parts); glm-4.6v and glm-5.3-flash read images on our plan, but neither is on the gateway. → `stacks/litellm/conf/config.yaml` diff --git a/persistent-memory.d/2026-10-07-nh3-svos-refresh-a737a06-apps.md b/persistent-memory.d/2026-10-07-nh3-svos-refresh-a737a06-apps.md new file mode 100644 index 0000000..21a235b --- /dev/null +++ b/persistent-memory.d/2026-10-07-nh3-svos-refresh-a737a06-apps.md @@ -0,0 +1 @@ +- `[2026-10-07]` **nh3-svos refreshed to SVOS a737a06 (schema 11) at svos-dev's ask (Prime's 2240 decision).** Live: app-knowledge (Albok token per A111), plus app-terminal, degraded by design (no seat on the box), on its own origin `svos-terminal.phasefinal.com` (no Basic auth, mesh-only). app-postoffice is declared and stopped (⚠ the bridge handle is Prime's to declare); Miranda's packages are only staged, for U13. Acceptance: 17/17 HTTPS checks and the Chromium phone/desk checks passed. ⚠ The interface fails once on thaw (it starts before the kernel listens); the restart recovers it, but the hook's single status check can race it. Reported to svos-dev. → `servers/nh3-svos/README.md` diff --git a/persistent-memory.d/2026-10-08-heid-runaway-sweep-killed.md b/persistent-memory.d/2026-10-08-heid-runaway-sweep-killed.md new file mode 100644 index 0000000..7c4fadd --- /dev/null +++ b/persistent-memory.d/2026-10-08-heid-runaway-sweep-killed.md @@ -0,0 +1,16 @@ +# 2026-10-08 — heid's runaway sweep killed (Prime: "kill sweep immediately") + +**Symptom.** nh3-dev's root disk flapped on the Beszel 85% line (2300 and 2329 10-07). The uv-cache prune (−6.7 GiB) bought minutes. `du` could not see the growth. + +**Cause (found by infra-hermes, confirmed by infra-ops).** `heidnotes/sweep.py` under `heid/dispatch-log/work/01M4CC74W1HK46D1ZP9J26YSZK` was left running by a consult agent of svos-dev's U11 bug hunt, which had been relayed and closed at 1748. It swept the dispatch work dir recursively, and its output `heidnotes/final_sweep.txt` sat INSIDE that dir. So it read its own output and wrote it back out, forever: the tail was lines quoting `final_sweep.txt:66200017 …`. The work dir was then cleaned while the file was still open, so ~76 GB lived in a deleted-but-open file that only `/proc//fd` showed. + +**Two holders, not one.** infra-hermes saw PID 2332755; a second copy, PID 2368442 (its zsh wrapper 2368332, launched from the `heid` Claude session, pid 35804), held the same inode on fd 3. Killing the first freed nothing. Only when the second died did root go 86% → 66% (52 → 123 GB free). + +**Lessons.** +- A deleted-but-open file IS readable through `/proc//fd/`. That is how the self-feeding loop was confirmed before the kill, turning "runaway, kill it" from a guess into a verdict. +- Before claiming that space will free, enumerate EVERY holder of the inode (walk `/proc/*/fd`), not just the first PID found. +- Kill by explicit PID. Matching `heidnotes/sweep.py` also caught the wrapper shell, which was harmless here, and `pgrep -f` self-matched the checking command. + +**Fix at the source (heid `8bf2fe57`).** The sweep tool skips `heidnotes/` and any file over 20 MB. The consult-agent brief forbids leaving a process running after the report. heid is adding a check for lingering processes before each workdir clean. The same bug had hit a second agent at ~3.4 GB earlier that day, and that agent caught it itself. + +Ops-log 2026-10-08T06:37:07Z `nh3-dev kill heid-sweep`. diff --git a/persistent-memory.d/2026-10-08-patch-night-fv-ml1-kernel-nvidia-zfs-docker-hosts.md b/persistent-memory.d/2026-10-08-patch-night-fv-ml1-kernel-nvidia-zfs-docker-hosts.md new file mode 100644 index 0000000..5d949fe --- /dev/null +++ b/persistent-memory.d/2026-10-08-patch-night-fv-ml1-kernel-nvidia-zfs-docker-hosts.md @@ -0,0 +1 @@ +- `[2026-10-08]` **Patch night (Prime's go): fv-ml1 → kernel 6.12.111, ZFS 2.3.9, NVIDIA 580.178.04, toolkit 1.20.1 (one reboot, 13/13 seats healthy in ~7 min, every alias answered); Docker 29.4.1 → 29.8.2 on esh-docker-vm and nh3-docker (post office ~25 s blip), all containers back.** ⚠ fv-ml1 boots ZFSBootMenu, not GRUB; trixie's apt rejects NVIDIA's debian12 repo key (SHA1 binding), so driver debs are gpgv/sha256-verified by hand. Holds re-applied on all three. Rollback snapshot `zroot/ROOT/debian@pre-patch-20261008`, destroy ~10-15. SVOS post office is running; its inbox grant is a card for Prime. → `servers/fv-ml1/README.md` diff --git a/persistent-memory.md b/persistent-memory.md index 6eb9037..0f3d394 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-10-06 ~2145 PT (fv-ml1 KV pins + GPU 0/1 rebalance by usage with contexts at published maximums, gen-small output cap; backup checker triple fixed, destroyed-guest residue pruned, pbs-nh3 mirror retention matched; Scriberr diarization fixed with caches on /tank and no real token; albok-service 0.1.11 → 0.1.12; gateway keeps no content (arbo incognito) + irv-ml1 restic exclude; jekyll moved to sec-reasoning.)_ +_Last updated: 2026-10-08 ~0740 PT (patch night: fv-ml1 kernel 6.12.111 + NVIDIA 580.178.04 + ZFS 2.3.9, Docker 29.8.2 on nh3-docker + esh-docker-vm; nh3-svos on SVOS a737a06 with knowledge/terminal/post office live; albok-service 0.1.13 → 0.1.14; gateway glm-5.3 + gen-frontier on 5.3; heid's runaway 76 GB sweep killed; Hermes launchers repaired.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -113,24 +113,30 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-10-06 ~2145 PT. The newest subsection is first; older subsections carry their own dates. What was DONE on 10-05/06 lives in Recent decisions; this section keeps only what is open, scheduled or being watched._ +_As of 2026-10-08 ~0740 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched._ -### 2026-10-06: live now (as of 2026-10-06 ~2145 PT) +### 2026-10-08: live now (as of 2026-10-08 ~0740 PT) -- **Nothing is mid-action.** No claims held (`scripts/ops-log claims`). eshpfi is pushed through `0f6c7a0` (origin/main == HEAD at ~2140). ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- ` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose (auto-regenerated). -- **Verified 2026-10-07/08 (closed):** the 10-07 0800 backup check read all fresh with no DESTROYED/DEGRADED lines; the pbs-nh3 mirror prune ran OK at 00:00 on 10-07 and 10-08 (nh3-nas still 79% at 2350, since GC frees chunks only after the 24 h atime cutoff; re-read after the 10-08 GC); `pre-pve9` snapshots are gone on pfi-pve and esh-pve. **Still owed:** the fv-ml1 GPU 0/1 plateau read (Beszel `stats.g..mu` × 1.024); sfsrv-ana `pre-pbs4` delete unverified (infra-ops has no key there; ask infra-hermes). -- **Prime 2026-10-07 late:** SVOS post office bridge handle = `svos` (he declares it on the fleet page; then enable `svos-postoffice` and request its `ns:notifications` write grant as a card for him). **No z.ai vision alias** on the gateway. +- **Nothing is mid-action.** No claims held (`scripts/ops-log claims`). eshpfi is pushed through this snapshot's commit. ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- ` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose (auto-regenerated). +- **Owed by me (no input needed):** + - albok `.browse/` exposure test before ANY share: pick a surface (sshfs `follow_symlinks` is the lead), rename `.browse/` on nh3-albok, and a fresh open through the client's cached listing must FAIL; also record the failed-rename case. Report on albok-dev's thread; the operator's user needs the restricted wings' groups on the share host. → `stacks/albok-service/README.md` + - fv-ml1 GPU 0/1 plateau read (Beszel `stats.g..mu` × 1.024), now ~10-10, since every seat restarted at 0034 10-08. + - Destroy `zroot/ROOT/debian@pre-patch-20261008` on fv-ml1 ~10-15 if nothing regressed. + - sfsrv-ana `pre-pbs4` snapshot delete is unverified (infra-ops has no key there; ask infra-hermes). + - Re-read nh3-nas usage after the 10-08 GC (79% at 10-07 2350; the mirror prune ran OK). + - Install SVOS kernel 0.0.7 (`Type=notify`, `NotifyAccess=main`: the thaw-race fix Prime accepted) with the next nh3-svos refresh, when svos-dev says it is on main. +- **Verified 2026-10-07/08 (closed):** the 10-07 0800 backup check read all fresh with no DESTROYED/DEGRADED lines; the pbs-nh3 mirror prune ran OK at 00:00 on 10-07 and 10-08; `pre-pve9` snapshots are gone on pfi-pve and esh-pve. - **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens. jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it, as gen-large did (preemptions; decode down to 2–16 tok/s). jekyll's context grows ~1k/call toward the 262k ceiling. It runs on the shared all-agents-local key; a key of its own would make its traffic attributable. - **Waiting on Prime:** - albok's 200-object load check, 2.01× against a 2× bar (albok-dev: the ruling is the operator's). - The cleanup bundle offered 10-05: close stale parks 49/46/40; refresh the fv-ml1 README stack tables plus the serial-keyed NVMe table and `zpool replace` line (the 2084-media-error drive is now `/dev/nvme6`, not nvme7); wire ZED pool alerts into althing-alert-bridge; prune the exited one-shot containers on fv-ml1 (keep the `parakeet` rollback). - Push Booth commit `ce095c1`. A trusted HTTPS name for ESH Homepage. nh3-dev reboot (kernel 6.1.0-53 installed). - ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope. -- **Scheduled / unchecked:** the 10-05 0815 + 0900 infra-hermes jobs (sfsrv-ana `pre-pbs4` snapshot delete; drop `pve/root-pre-pve9` on pfi-pve + esh-pve) left no ops-log lines; ask infra-hermes or read the hosts. esh-nas-pve `nvme/ROOT@pre-pve9` destroy ~10-10 (ours). 10-11 0540: first weekly sudo-io archiver run (check it). 10-11 0900: infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet. Next scheduled tank scrub on fv-ml1: Sun 10-11 0024 (the September slot was missed); check it ran. +- **Scheduled / unchecked:** esh-nas-pve `nvme/ROOT@pre-pve9` destroy ~10-10 (ours). 10-11 0540: first weekly sudo-io archiver run (check it). 10-11 0900: infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet. Next scheduled tank scrub on fv-ml1: Sun 10-11 0024 (the September slot was missed); check it ran. - **Worldtree memory gate:** every FAIL since 09-30 (0930T0809, 1004, 1007) is ONE fixture, `u-negative-ephemeral` FILED in one run — the #416 ephemeral residual, ~3 of 42 fixture-runs since bbcc1788 (~1 batch in 5 FAILs on it), NOT a recall regression; recall (filing/user median 0.83–0.92, opener/cross_session/inline_core 100%) has held throughout. Negative is a structural class, so one run's leak fails the aggregate (`_run_structural_failed`); filing alone is judged by median. 10-07: infra-hermes told to classify FAILs by failing row (msg to its skill's daily-duty section); worldtree-dev told #416 is not closed (pull-only, unread 4 days) — reopen is theirs/Prime's. -- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`. +- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`. The LiteLLM compose.yaml on ana-docker differs from `stacks/litellm/compose.yaml` (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%). - **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in. -- **Older open items, one line each:** esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (`stacks/blender/README.md`). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran `51168a2e` on 09-27). nh3-dev's root disk sits near the Beszel 85% line. ESH has a single outside route (esh-scale on esh-pve), noted and untracked. +- **Older open items, one line each:** esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (`stacks/blender/README.md`). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran `51168a2e` on 09-27). nh3-dev's root disk is at ~66% since heid's runaway sweep was killed 10-08 (it flapped on the Beszel 85% line before). ESH has a single outside route (esh-scale on esh-pve), noted and untracked. ### fv-ml1 GPU layout (as of 2026-10-05 2249) @@ -181,14 +187,17 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache- ## Recent decisions +- `[2026-10-08]` **heid's runaway sweep killed on Prime's order: two `heidnotes/sweep.py` processes had written ~76 GB into a deleted-but-open file; nh3-dev root 86% → 66%** — the sweep read its own output. heid fixed it at the source. → `persistent-memory.d/2026-10-08-heid-runaway-sweep-killed.md` +- `[2026-10-08]` **SVOS post office LIVE on nh3-svos** — Prime declared `svos`; its `ns:notifications` write grant was approved on his word at 0731 (card `card:4d3a8b26c17f472d828b06fd495192db`); a test message was ingested and delivered; `reach` ok, unread 0. Reported to svos-dev. +- `[2026-10-07]` **No z.ai vision alias on the gateway (Prime).** glm-5.x is text-only at z.ai; glm-4.6v and glm-5.3-flash read images on our plan but stay off the gateway. - `[2026-10-06]` **Scriberr diarization fixed: gated pyannote, a torchcodec/torch mismatch, and WhisperX's CPU defaults; model caches on /tank, HF_TOKEN a placeholder (Prime), GPU+speaker profiles.** Patch 0003 waits for the next rebuild (tracked at `stacks/scriberr/patches/README.md`). → `persistent-memory.d/2026-10-06-scriberr-diarization-fixed.md` - `[2026-10-06]` **Backup checker: destroyed guests reported as residue, an ssh stdin swallow fixed, snapshot age from the name; residue pruned (Prime); pbs-nh3 mirror keeps pbs-ana's history (Prime "match history"), first prune 10-07 00:00.** → `persistent-memory.d/2026-10-06-backup-checker-fixes-and-mirror-retention.md` -- `[2026-10-08]` **Patch night (Prime's go): fv-ml1 → kernel 6.12.111, ZFS 2.3.9, NVIDIA 580.178.04, toolkit 1.20.1 (one reboot, 13/13 seats healthy in ~7 min, every alias answered); Docker 29.4.1 → 29.8.2 on esh-docker-vm and nh3-docker (post office ~25 s blip), all containers back.** ⚠ fv-ml1 boots ZFSBootMenu, not GRUB; trixie's apt rejects NVIDIA's debian12 repo key (SHA1 binding), so driver debs are gpgv/sha256-verified by hand. Holds re-applied on all three. Rollback snapshot `zroot/ROOT/debian@pre-patch-20261008`, destroy ~10-15. SVOS post office is running; its inbox grant is a card for Prime. → `servers/fv-ml1/README.md` -- `[2026-10-07]` **nh3-svos refreshed to SVOS a737a06 (schema 11) at svos-dev's ask (Prime's 2240 decision).** Live: app-knowledge (Albok token per A111), plus app-terminal, degraded by design (no seat on the box), on its own origin `svos-terminal.phasefinal.com` (no Basic auth, mesh-only). app-postoffice is declared and stopped (⚠ the bridge handle is Prime's to declare); Miranda's packages are only staged, for U13. Acceptance: 17/17 HTTPS checks and the Chromium phone/desk checks passed. ⚠ The interface fails once on thaw (it starts before the kernel listens); the restart recovers it, but the hook's single status check can race it. Reported to svos-dev. → `servers/nh3-svos/README.md` -- `[2026-10-07]` **Gateway: glm-5.3 added; gen-frontier / gen-frontier-reasoning moved 5.2 → 5.3 (Prime); every GLM 5.2/5.3 entry advertises 1,048,576 in / 131,072 out with max_tokens 131,072 as the default.** 5.3 cannot turn reasoning off, so the gen-frontier pair is now effort low / high (measured ~260 vs ~1,490 reasoning tokens). ⚠ z.ai's `thinking: disabled` is light thinking, not zero, on 5.1/5.2/5.3 alike (220–340 reasoning tokens on a hard prompt). Restarted 1725 (61 s), held 2 h for albok-dev's load A/B. The gate script's first two runs could never pass: `grep -c` exits 1 on a zero count. Images: glm-5.x is text only (z.ai rejects image parts); glm-4.6v and glm-5.3-flash read images on our plan, but neither is on the gateway. → `stacks/litellm/conf/config.yaml` +- `[2026-10-08]` **Patch night (Prime's go): fv-ml1 kernel/NVIDIA/ZFS/toolkit + Docker 29.8.2 on esh-docker-vm and nh3-docker, all verified** — ZBM boot, the hand-verified NVIDIA driver path, holds re-applied. → `persistent-memory.d/2026-10-08-patch-night-fv-ml1-kernel-nvidia-zfs-docker-hosts.md` +- `[2026-10-07]` **nh3-svos refreshed to SVOS a737a06 (schema 11): knowledge, terminal (own origin) and post office apps** — acceptance 17/17 + Chromium; the thaw race. → `persistent-memory.d/2026-10-07-nh3-svos-refresh-a737a06-apps.md` +- `[2026-10-07]` **Gateway: glm-5.3 added, gen-frontier pair moved to 5.3 (effort low/high), 1M/128K advertised on GLM 5.2/5.3 (Prime)** — `thinking: disabled` is light thinking, not zero. → `persistent-memory.d/2026-10-07-gateway-glm-5-3-gen-frontier.md` - `[2026-10-07]` **Hermes launchers repaired (Miranda relay 8462):** svos-dev's scratch probe had rewritten `~/.hermes/hermes-agent/.hermes/bin/{hermes,hermes-acp}` to exec a /tmp scratch Python. Repointed to `~/.hermes/tools/python-3.14.7…`; the gateway was not restarted. ⚠ Don't run `git` in `~/.hermes/hermes-agent`: a `status` there looped on auto-gc. -- `[2026-10-07]` **albok-service 0.1.14 (unit 4.3: sidecar listing, byte-less ingest, a `.browse/` view; detector 2) deployed at 1804 at albok-dev's ask 8524.** `leaks` went from 8 to 6 when the placeholder false positives cleared. Reindexes ran one per wing afterwards to render the view. ⚠ **OWED: the `.browse/` exposure test.** Pick a share surface (sshfs `follow_symlinks` is the lead), rename `.browse/` on the server, and check that a fresh open through the client's cached listing FAILS. Nothing is shared until it passes; report to albok-dev. → `stacks/albok-service/README.md` -- `[2026-10-07]` **albok-service 0.1.13 (unit 4.2, the sensitivity gate) deployed on nh3-albok at 1135 at albok-dev's ask 8413.** Rollback is NOT free: the first-start scan writes `sensitivity` sidecars, which 0.1.12 cannot load, so rolling back needs the strip snippet first. The census found `leaks` 8: six hold the shared all-agents key, two are false positives on a doc placeholder, nothing to rotate. → `stacks/albok-service/README.md` +- `[2026-10-07]` **albok-service 0.1.14 (unit 4.3) deployed + all four wings reindexed** — the `.browse/` view exists on the box only; the share test is owed. → `persistent-memory.d/2026-10-07-albok-service-0-1-14.md` +- `[2026-10-07]` **albok-service 0.1.13 (unit 4.2, the sensitivity gate) deployed** — rollback to 0.1.12 needs the strip snippet first. → `persistent-memory.d/2026-10-07-albok-service-0-1-13-sensitivity-gate.md` - `[2026-10-06]` **albok-service 0.1.11 (0202) then 0.1.12 (1254) deployed on nh3-albok at albok-dev's asks 7991/8199**, built from clean archives, pinned by digest, single-file upload (deploy-stack's `--delete` would wipe the on-host rollback copies). Every wing is now migrated to generation collections, so a 0.1.10 rollback needs a full reindex. albok's 2.01×-vs-2× load check awaits Prime. → `stacks/albok-service/README.md` - `[2026-10-06]` **The LiteLLM gateway keeps no request/response content, and `store_prompts_in_spend_logs: false` is now a marked consumer guarantee (comfy-dev, arbo incognito).** Proved with a canary plus positive controls; per-request `no-log` does NOT cover the proxy's own spend-log callback (v1.97 source). irv-ml1 restic also excludes `input/incognito-*` (dry-run proved with a canary and a control). `847424c`, `0f6c7a0`. - `[2026-10-06]` **jekyll moved from gen-large to sec-reasoning (Prime)** after its ~180k-token agent loop and Prime's ~140k session overflowed flash-next's KV. → `persistent-memory.d/2026-10-06-gen-large-contention-jekyll.md` @@ -280,22 +289,8 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache- - `[2026-09-25]` Created empty private repo `corviduo/norn` (git@gitea.phasefinal.com:corviduo/norn.git) as claude-bot, at brokkr-smithy-dev's relay of the operator's Norn ruling; the `norn-dev` handle is the operator's to declare. - `[2026-09-25]` **Reward seat audited (Skywork-Reward-V2-Llama-3.1-8B still #1 of 188 on RewardBench 2; our AWQ ≈ bf16 within noise; double BOS costs ~2.7 pts) and moved fv-ml1 → esh-ml1; esh-ml1 monitoring wired (Beszel+GPU, Kuma, Homepage, Dozzle); Dozzle hub's stale agent IPs fixed; nh3-dev Beszel agent revived.** → `persistent-memory.d/2026-09-25-reward-move-and-esh-ml1-monitoring.md` - `[2026-09-25]` **TEI adopted as the fleet embed/rerank engine (Prime); esh-ml1 is the sole gateway backend for `qwen3-embedding` + `reranker`; fv-ml1's vLLM embed/rerank seats retired (~6.1 GB freed); parakeet stays on fv-ml1.** → `persistent-memory.d/2026-09-25-tei-fleet-embed-rerank.md` -- `[2026-09-24]` **esh-ml1 built — RTX 2000E Ada as CT 110 on esh-pve (host driver 580.178.04 DKMS, loaded live, no reboot), vLLM embed+rerank at measured parity with fv-ml1, wired as LiteLLM `order: 2` failover; dead DB alias `reranker-a3-bge-v2-m3` (pre-relocation IP) repaired.** → `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md` -- `[2026-09-24]` **esh-pve: VM 102 retired, a 14-day hung VFIO process found and cleared, T400 → RTX 2000 Ada; Prime chose LXC + host driver for embed/rerank** (implementation deferred to next session, tracked here + `servers/esh-pve/README.md`). → `persistent-memory.d/2026-09-24-esh-pve-vm102-retired-gpu-swap.md` -- `[2026-09-24]` **pfi-gx10 AC-restore UEFI patch applied but NOT validated; box OFF until Prime's AC pull 2026-09-25** (tracking: `servers/pfi-gx10/README.md`, `e5197a3`). — `[2026-09-25]` ✅ **VALIDATED**: Prime pulled and replugged AC, and it came up by itself at 1108:54 (`2b49be4`). → `persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md` -- `[2026-09-24]` **NH3 power outage recovered** — pbs-nh3 had no `onboot` (set), NFS boot race fixed with automount (`1cbde50`), every other Claude session on nh3-dev died. → `persistent-memory.d/2026-09-24-nh3-power-outage-recovery.md` -- `[2026-09-24]` **Miranda standing order is a repo CLAUDE.md operating parameter** — (`4b29492`, aligned to the global send protocol in `bcf3342`): high-urgency matters go to her, fixed or not… → `persistent-memory.d/2026-09-24-miranda-standing-order-is-a-repo-claude-md-operating.md` -- `[2026-09-24]` **task-board mothballed** (Prime): container removed on ana-docker, data/image/compose kept, Kuma monitor deleted, task_* instructions removed from CLAUDE.md and the fork template (`e6da607`). Its hooks had sent no traffic in 30 days. -- `[2026-09-24]` **Military 24-hour Pacific clock times** carried into the Codex/Grok shared bootstrap `docs/fleettools/AGENT-BOOTSTRAP.md` (`ad2b4d9`); Claude seats get it from the global CLAUDE.md. - `[2026-09-24]` **Worldtree `admin.memory.forget` stays OFF on demo/personal** until an instance needs it — a destructive erase; enabling it is a per-change operator yes via deploy-wt-config. -- `[2026-09-24]` **A git checkout under `root:docker` needs `safe.directory` for its deploy user** — the 09-14 normalization (`826a63b`) silently broke yt-voice-clipper's webhook deploy until v0.3.13; fixed on irv-ml1 and recorded in fleet conventions (`eea9eb2`). Sweep found no other case. -- `[2026-09-23]` **elway sudo uploads land root:root, validated and staged; fleet ownership audit built; 33 mis-owned root files fixed on 10 hosts.** → `persistent-memory.d/2026-09-23-elway-ownership-fix-fleet-audit.md` -- `[2026-09-23]` **headscale-ddns hardened** — (`fedd4b6`): Cloudflare calls retry and validate every body (`pick()`), no write without both IDs, the run… → `persistent-memory.d/2026-09-23-headscale-ddns-hardened.md` -- `[2026-09-23]` **esh-docker-vm restic was skipped 09-22..23 by my own Kuma move** — a dead uptime-kuma lookup aborted `pre-backup.sh` under `set -e` (`25e41d2`). → `persistent-memory.d/2026-09-23-esh-docker-vm-restic-was-skipped-09-22-23-by-my-own-kuma.md` -- `[2026-09-23]` **hermes-gateway restart exit-1 is a Hermes race, not a crash** — the planned-stop watcher consumes the marker before SIGTERM re-runs the handler; drop-in… → `persistent-memory.d/2026-09-23-hermes-gateway-restart-exit-1-is-a-hermes-race-not-a-crash.md` -- `[2026-09-23]` **Booth link board cleared to 14 durable links** (208 removed: drops, release posts, research links, token-bearing URLs; Scriberr reposted at `scriberr.fv.internal:8080`). Prime's rule: durable debug links only. -- `[2026-09-22]` **safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes.** — ⚠ The package installs INERT and looks fine — Debian's `/etc/zsh/zprofile` has 0 non-comment lines so the… → `persistent-memory.d/2026-09-22-safe-rm-installed-on-nh3-dev-and-delegated-fleet-wide-to.md` - `[2026-09-17]` **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the…** → `persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion.md` - `[2026-09-17]` **The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev** — `dev_launch.py` has zero occurrences of "route", no hook declares one, and every live route was hand-declared… → `persistent-memory.d/2026-09-17-the-althing-route-declaring-sessionstart-hook-is-documented.md` @@ -308,10 +303,14 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache- - `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md` -_214 older entries archived to archival-memory.md._ +_228 older entries archived to archival-memory.md._ ## Tried and abandoned +- `[2026-10-08]` **NVIDIA's network debian12 repo on trixie via `cuda-keyring`** — trixie's sqv rejects the key's SHA1 binding signature, so `apt update` errors until the list is gone (`dpkg -r` keeps the conffile; it took `dpkg -P`). Fetch the debs and verify InRelease (gpgv) → Packages.gz → sha256 instead. → `servers/fv-ml1/README.md` +- `[2026-10-08]` **`secret get | head` "to see its shape"** — a single-value secret IS its shape; it printed the fv-ml1 BMC password into the local transcript. Check with `secret list` or `wc -c`, never by printing. +- `[2026-10-07]` **An idle gate counting with `grep -c … || echo ERR`** — `grep -c` exits 1 on a ZERO count, so the value was "0\nERR" and never equalled "0"; two 75-min LiteLLM restart gates were blind to any quiet window. Use `|| true` and test the gate against a known-idle state first. +- `[2026-10-07]` **`pgrep -f ` to check that a process died** — it matched the checking shell's own command line and reported a survivor; check with `ps -eo … | grep '[x]…'`, as the same self-match bit `pkill -f` before. - `[2026-10-06]` **Restarting a user-facing service after an idle check that only PRINTS** — `nvidia-smi … | wc -l` said 1 and the same command restarted Scriberr anyway, killing Prime's transcript run. Make the check gate (`&&` / exit). auto-memory `feedback_idle_check_must_gate_the_restart`. - `[2026-10-06]` **Aging PBS snapshots by directory mtime** — verify jobs and notes edits rewrite `index.json.blob`, so a 10-04 snapshot read as 10-06. Age by the snapshot NAME. auto-memory `reference_pbs_snapshot_mtime_moves_on_verify`. - `[2026-10-06]` **`deploy-stack.sh` on a stack whose host dir holds rollback copies** (albok-service `compose.yaml.pre-*`) — its rsync `--delete` removes them. Upload the single changed file with elway instead. @@ -338,7 +337,5 @@ _214 older entries archived to archival-memory.md._ - `[2026-09-25]` **Concluding "not on the UDM" from a port table read 15 s after link-up** — UniFi polls (~60 s); AMT was on port 6 as Prime said. Wait a poll, and prefer a which-port-lists-the-MAC check. auto-memory `feedback_polled_stats_lag_the_event`. - `[2026-09-25]` **The esh-pve NVIDIA DKMS recipe on nh3-pve** — it built, then the kernel refused to load it: nh3-pve has SECURE BOOT ON (esh-pve, the same MS-01, has it off). The installer rolled back. The playbook now pre-flights `mokutil --sb-state` (`7ddd116`). - `[2026-09-25]` **elway against an unpinned host-key name** — every `when:` hit ssh rc 255, so all 9 steps showed SKIPPED, not failed. Only the verify phase went red. Pin the name (the IP key matched) or use the IP. Untracked elway defect: rc 255 in `when:` should fail. -- `[2026-09-24]` **Testing "Restore AC Power Loss" with an OS shutdown** — a shutdown stays off BY DESIGN; only pulling and restoring AC tests it. A community README claimed the two are indistinguishable; trusting it cost Prime two trips to the pfi-gx10 power button. -- `[2026-09-23]` **`booth link --help`** — there is no help flag; it posts `--help` to the operator's link board as a link. Read `booth` with no args for usage. -_121 older entries archived to archival-memory.md._ +_123 older entries archived to archival-memory.md._