memory: snapshot — 2026-10-08 patch night, nh3-svos a737a06 + post office live, albok 0.1.13/0.1.14, gateway glm-5.3, heid sweep incident; 16 entries archived
This commit is contained in:
+29
-32
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-10-06 ~2145 PT (fv-ml1 KV pins + GPU 0/1 rebalance by usage with contexts at published maximums, gen-small output cap; backup checker triple fixed, destroyed-guest residue pruned, pbs-nh3 mirror retention matched; Scriberr diarization fixed with caches on /tank and no real token; albok-service 0.1.11 → 0.1.12; gateway keeps no content (arbo incognito) + irv-ml1 restic exclude; jekyll moved to sec-reasoning.)_
|
||||
_Last updated: 2026-10-08 ~0740 PT (patch night: fv-ml1 kernel 6.12.111 + NVIDIA 580.178.04 + ZFS 2.3.9, Docker 29.8.2 on nh3-docker + esh-docker-vm; nh3-svos on SVOS a737a06 with knowledge/terminal/post office live; albok-service 0.1.13 → 0.1.14; gateway glm-5.3 + gen-frontier on 5.3; heid's runaway 76 GB sweep killed; Hermes launchers repaired.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -113,24 +113,30 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-10-06 ~2145 PT. The newest subsection is first; older subsections carry their own dates. What was DONE on 10-05/06 lives in Recent decisions; this section keeps only what is open, scheduled or being watched._
|
||||
_As of 2026-10-08 ~0740 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched._
|
||||
|
||||
### 2026-10-06: live now (as of 2026-10-06 ~2145 PT)
|
||||
### 2026-10-08: live now (as of 2026-10-08 ~0740 PT)
|
||||
|
||||
- **Nothing is mid-action.** No claims held (`scripts/ops-log claims`). eshpfi is pushed through `0f6c7a0` (origin/main == HEAD at ~2140). ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- <paths>` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose (auto-regenerated).
|
||||
- **Verified 2026-10-07/08 (closed):** the 10-07 0800 backup check read all fresh with no DESTROYED/DEGRADED lines; the pbs-nh3 mirror prune ran OK at 00:00 on 10-07 and 10-08 (nh3-nas still 79% at 2350, since GC frees chunks only after the 24 h atime cutoff; re-read after the 10-08 GC); `pre-pve9` snapshots are gone on pfi-pve and esh-pve. **Still owed:** the fv-ml1 GPU 0/1 plateau read (Beszel `stats.g.<n>.mu` × 1.024); sfsrv-ana `pre-pbs4` delete unverified (infra-ops has no key there; ask infra-hermes).
|
||||
- **Prime 2026-10-07 late:** SVOS post office bridge handle = `svos` (he declares it on the fleet page; then enable `svos-postoffice` and request its `ns:notifications` write grant as a card for him). **No z.ai vision alias** on the gateway.
|
||||
- **Nothing is mid-action.** No claims held (`scripts/ops-log claims`). eshpfi is pushed through this snapshot's commit. ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- <paths>` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose (auto-regenerated).
|
||||
- **Owed by me (no input needed):**
|
||||
- albok `.browse/` exposure test before ANY share: pick a surface (sshfs `follow_symlinks` is the lead), rename `.browse/` on nh3-albok, and a fresh open through the client's cached listing must FAIL; also record the failed-rename case. Report on albok-dev's thread; the operator's user needs the restricted wings' groups on the share host. → `stacks/albok-service/README.md`
|
||||
- fv-ml1 GPU 0/1 plateau read (Beszel `stats.g.<n>.mu` × 1.024), now ~10-10, since every seat restarted at 0034 10-08.
|
||||
- Destroy `zroot/ROOT/debian@pre-patch-20261008` on fv-ml1 ~10-15 if nothing regressed.
|
||||
- sfsrv-ana `pre-pbs4` snapshot delete is unverified (infra-ops has no key there; ask infra-hermes).
|
||||
- Re-read nh3-nas usage after the 10-08 GC (79% at 10-07 2350; the mirror prune ran OK).
|
||||
- Install SVOS kernel 0.0.7 (`Type=notify`, `NotifyAccess=main`: the thaw-race fix Prime accepted) with the next nh3-svos refresh, when svos-dev says it is on main.
|
||||
- **Verified 2026-10-07/08 (closed):** the 10-07 0800 backup check read all fresh with no DESTROYED/DEGRADED lines; the pbs-nh3 mirror prune ran OK at 00:00 on 10-07 and 10-08; `pre-pve9` snapshots are gone on pfi-pve and esh-pve.
|
||||
- **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens. jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it, as gen-large did (preemptions; decode down to 2–16 tok/s). jekyll's context grows ~1k/call toward the 262k ceiling. It runs on the shared all-agents-local key; a key of its own would make its traffic attributable.
|
||||
- **Waiting on Prime:**
|
||||
- albok's 200-object load check, 2.01× against a 2× bar (albok-dev: the ruling is the operator's).
|
||||
- The cleanup bundle offered 10-05: close stale parks 49/46/40; refresh the fv-ml1 README stack tables plus the serial-keyed NVMe table and `zpool replace` line (the 2084-media-error drive is now `/dev/nvme6`, not nvme7); wire ZED pool alerts into althing-alert-bridge; prune the exited one-shot containers on fv-ml1 (keep the `parakeet` rollback).
|
||||
- Push Booth commit `ce095c1`. A trusted HTTPS name for ESH Homepage. nh3-dev reboot (kernel 6.1.0-53 installed).
|
||||
- ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope.
|
||||
- **Scheduled / unchecked:** the 10-05 0815 + 0900 infra-hermes jobs (sfsrv-ana `pre-pbs4` snapshot delete; drop `pve/root-pre-pve9` on pfi-pve + esh-pve) left no ops-log lines; ask infra-hermes or read the hosts. esh-nas-pve `nvme/ROOT@pre-pve9` destroy ~10-10 (ours). 10-11 0540: first weekly sudo-io archiver run (check it). 10-11 0900: infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet. Next scheduled tank scrub on fv-ml1: Sun 10-11 0024 (the September slot was missed); check it ran.
|
||||
- **Scheduled / unchecked:** esh-nas-pve `nvme/ROOT@pre-pve9` destroy ~10-10 (ours). 10-11 0540: first weekly sudo-io archiver run (check it). 10-11 0900: infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet. Next scheduled tank scrub on fv-ml1: Sun 10-11 0024 (the September slot was missed); check it ran.
|
||||
- **Worldtree memory gate:** every FAIL since 09-30 (0930T0809, 1004, 1007) is ONE fixture, `u-negative-ephemeral` FILED in one run — the #416 ephemeral residual, ~3 of 42 fixture-runs since bbcc1788 (~1 batch in 5 FAILs on it), NOT a recall regression; recall (filing/user median 0.83–0.92, opener/cross_session/inline_core 100%) has held throughout. Negative is a structural class, so one run's leak fails the aggregate (`_run_structural_failed`); filing alone is judged by median. 10-07: infra-hermes told to classify FAILs by failing row (msg to its skill's daily-duty section); worldtree-dev told #416 is not closed (pull-only, unread 4 days) — reopen is theirs/Prime's.
|
||||
- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`.
|
||||
- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`. The LiteLLM compose.yaml on ana-docker differs from `stacks/litellm/compose.yaml` (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%).
|
||||
- **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in.
|
||||
- **Older open items, one line each:** esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (`stacks/blender/README.md`). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran `51168a2e` on 09-27). nh3-dev's root disk sits near the Beszel 85% line. ESH has a single outside route (esh-scale on esh-pve), noted and untracked.
|
||||
- **Older open items, one line each:** esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (`stacks/blender/README.md`). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran `51168a2e` on 09-27). nh3-dev's root disk is at ~66% since heid's runaway sweep was killed 10-08 (it flapped on the Beszel 85% line before). ESH has a single outside route (esh-scale on esh-pve), noted and untracked.
|
||||
|
||||
### fv-ml1 GPU layout (as of 2026-10-05 2249)
|
||||
|
||||
@@ -181,14 +187,17 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-10-08]` **heid's runaway sweep killed on Prime's order: two `heidnotes/sweep.py` processes had written ~76 GB into a deleted-but-open file; nh3-dev root 86% → 66%** — the sweep read its own output. heid fixed it at the source. → `persistent-memory.d/2026-10-08-heid-runaway-sweep-killed.md`
|
||||
- `[2026-10-08]` **SVOS post office LIVE on nh3-svos** — Prime declared `svos`; its `ns:notifications` write grant was approved on his word at 0731 (card `card:4d3a8b26c17f472d828b06fd495192db`); a test message was ingested and delivered; `reach` ok, unread 0. Reported to svos-dev.
|
||||
- `[2026-10-07]` **No z.ai vision alias on the gateway (Prime).** glm-5.x is text-only at z.ai; glm-4.6v and glm-5.3-flash read images on our plan but stay off the gateway.
|
||||
- `[2026-10-06]` **Scriberr diarization fixed: gated pyannote, a torchcodec/torch mismatch, and WhisperX's CPU defaults; model caches on /tank, HF_TOKEN a placeholder (Prime), GPU+speaker profiles.** Patch 0003 waits for the next rebuild (tracked at `stacks/scriberr/patches/README.md`). → `persistent-memory.d/2026-10-06-scriberr-diarization-fixed.md`
|
||||
- `[2026-10-06]` **Backup checker: destroyed guests reported as residue, an ssh stdin swallow fixed, snapshot age from the name; residue pruned (Prime); pbs-nh3 mirror keeps pbs-ana's history (Prime "match history"), first prune 10-07 00:00.** → `persistent-memory.d/2026-10-06-backup-checker-fixes-and-mirror-retention.md`
|
||||
- `[2026-10-08]` **Patch night (Prime's go): fv-ml1 → kernel 6.12.111, ZFS 2.3.9, NVIDIA 580.178.04, toolkit 1.20.1 (one reboot, 13/13 seats healthy in ~7 min, every alias answered); Docker 29.4.1 → 29.8.2 on esh-docker-vm and nh3-docker (post office ~25 s blip), all containers back.** ⚠ fv-ml1 boots ZFSBootMenu, not GRUB; trixie's apt rejects NVIDIA's debian12 repo key (SHA1 binding), so driver debs are gpgv/sha256-verified by hand. Holds re-applied on all three. Rollback snapshot `zroot/ROOT/debian@pre-patch-20261008`, destroy ~10-15. SVOS post office is running; its inbox grant is a card for Prime. → `servers/fv-ml1/README.md`
|
||||
- `[2026-10-07]` **nh3-svos refreshed to SVOS a737a06 (schema 11) at svos-dev's ask (Prime's 2240 decision).** Live: app-knowledge (Albok token per A111), plus app-terminal, degraded by design (no seat on the box), on its own origin `svos-terminal.phasefinal.com` (no Basic auth, mesh-only). app-postoffice is declared and stopped (⚠ the bridge handle is Prime's to declare); Miranda's packages are only staged, for U13. Acceptance: 17/17 HTTPS checks and the Chromium phone/desk checks passed. ⚠ The interface fails once on thaw (it starts before the kernel listens); the restart recovers it, but the hook's single status check can race it. Reported to svos-dev. → `servers/nh3-svos/README.md`
|
||||
- `[2026-10-07]` **Gateway: glm-5.3 added; gen-frontier / gen-frontier-reasoning moved 5.2 → 5.3 (Prime); every GLM 5.2/5.3 entry advertises 1,048,576 in / 131,072 out with max_tokens 131,072 as the default.** 5.3 cannot turn reasoning off, so the gen-frontier pair is now effort low / high (measured ~260 vs ~1,490 reasoning tokens). ⚠ z.ai's `thinking: disabled` is light thinking, not zero, on 5.1/5.2/5.3 alike (220–340 reasoning tokens on a hard prompt). Restarted 1725 (61 s), held 2 h for albok-dev's load A/B. The gate script's first two runs could never pass: `grep -c` exits 1 on a zero count. Images: glm-5.x is text only (z.ai rejects image parts); glm-4.6v and glm-5.3-flash read images on our plan, but neither is on the gateway. → `stacks/litellm/conf/config.yaml`
|
||||
- `[2026-10-08]` **Patch night (Prime's go): fv-ml1 kernel/NVIDIA/ZFS/toolkit + Docker 29.8.2 on esh-docker-vm and nh3-docker, all verified** — ZBM boot, the hand-verified NVIDIA driver path, holds re-applied. → `persistent-memory.d/2026-10-08-patch-night-fv-ml1-kernel-nvidia-zfs-docker-hosts.md`
|
||||
- `[2026-10-07]` **nh3-svos refreshed to SVOS a737a06 (schema 11): knowledge, terminal (own origin) and post office apps** — acceptance 17/17 + Chromium; the thaw race. → `persistent-memory.d/2026-10-07-nh3-svos-refresh-a737a06-apps.md`
|
||||
- `[2026-10-07]` **Gateway: glm-5.3 added, gen-frontier pair moved to 5.3 (effort low/high), 1M/128K advertised on GLM 5.2/5.3 (Prime)** — `thinking: disabled` is light thinking, not zero. → `persistent-memory.d/2026-10-07-gateway-glm-5-3-gen-frontier.md`
|
||||
- `[2026-10-07]` **Hermes launchers repaired (Miranda relay 8462):** svos-dev's scratch probe had rewritten `~/.hermes/hermes-agent/.hermes/bin/{hermes,hermes-acp}` to exec a /tmp scratch Python. Repointed to `~/.hermes/tools/python-3.14.7…`; the gateway was not restarted. ⚠ Don't run `git` in `~/.hermes/hermes-agent`: a `status` there looped on auto-gc.
|
||||
- `[2026-10-07]` **albok-service 0.1.14 (unit 4.3: sidecar listing, byte-less ingest, a `.browse/` view; detector 2) deployed at 1804 at albok-dev's ask 8524.** `leaks` went from 8 to 6 when the placeholder false positives cleared. Reindexes ran one per wing afterwards to render the view. ⚠ **OWED: the `.browse/` exposure test.** Pick a share surface (sshfs `follow_symlinks` is the lead), rename `.browse/` on the server, and check that a fresh open through the client's cached listing FAILS. Nothing is shared until it passes; report to albok-dev. → `stacks/albok-service/README.md`
|
||||
- `[2026-10-07]` **albok-service 0.1.13 (unit 4.2, the sensitivity gate) deployed on nh3-albok at 1135 at albok-dev's ask 8413.** Rollback is NOT free: the first-start scan writes `sensitivity` sidecars, which 0.1.12 cannot load, so rolling back needs the strip snippet first. The census found `leaks` 8: six hold the shared all-agents key, two are false positives on a doc placeholder, nothing to rotate. → `stacks/albok-service/README.md`
|
||||
- `[2026-10-07]` **albok-service 0.1.14 (unit 4.3) deployed + all four wings reindexed** — the `.browse/` view exists on the box only; the share test is owed. → `persistent-memory.d/2026-10-07-albok-service-0-1-14.md`
|
||||
- `[2026-10-07]` **albok-service 0.1.13 (unit 4.2, the sensitivity gate) deployed** — rollback to 0.1.12 needs the strip snippet first. → `persistent-memory.d/2026-10-07-albok-service-0-1-13-sensitivity-gate.md`
|
||||
- `[2026-10-06]` **albok-service 0.1.11 (0202) then 0.1.12 (1254) deployed on nh3-albok at albok-dev's asks 7991/8199**, built from clean archives, pinned by digest, single-file upload (deploy-stack's `--delete` would wipe the on-host rollback copies). Every wing is now migrated to generation collections, so a 0.1.10 rollback needs a full reindex. albok's 2.01×-vs-2× load check awaits Prime. → `stacks/albok-service/README.md`
|
||||
- `[2026-10-06]` **The LiteLLM gateway keeps no request/response content, and `store_prompts_in_spend_logs: false` is now a marked consumer guarantee (comfy-dev, arbo incognito).** Proved with a canary plus positive controls; per-request `no-log` does NOT cover the proxy's own spend-log callback (v1.97 source). irv-ml1 restic also excludes `input/incognito-*` (dry-run proved with a canary and a control). `847424c`, `0f6c7a0`.
|
||||
- `[2026-10-06]` **jekyll moved from gen-large to sec-reasoning (Prime)** after its ~180k-token agent loop and Prime's ~140k session overflowed flash-next's KV. → `persistent-memory.d/2026-10-06-gen-large-contention-jekyll.md`
|
||||
@@ -280,22 +289,8 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
- `[2026-09-25]` Created empty private repo `corviduo/norn` (git@gitea.phasefinal.com:corviduo/norn.git) as claude-bot, at brokkr-smithy-dev's relay of the operator's Norn ruling; the `norn-dev` handle is the operator's to declare.
|
||||
- `[2026-09-25]` **Reward seat audited (Skywork-Reward-V2-Llama-3.1-8B still #1 of 188 on RewardBench 2; our AWQ ≈ bf16 within noise; double BOS costs ~2.7 pts) and moved fv-ml1 → esh-ml1; esh-ml1 monitoring wired (Beszel+GPU, Kuma, Homepage, Dozzle); Dozzle hub's stale agent IPs fixed; nh3-dev Beszel agent revived.** → `persistent-memory.d/2026-09-25-reward-move-and-esh-ml1-monitoring.md`
|
||||
- `[2026-09-25]` **TEI adopted as the fleet embed/rerank engine (Prime); esh-ml1 is the sole gateway backend for `qwen3-embedding` + `reranker`; fv-ml1's vLLM embed/rerank seats retired (~6.1 GB freed); parakeet stays on fv-ml1.** → `persistent-memory.d/2026-09-25-tei-fleet-embed-rerank.md`
|
||||
- `[2026-09-24]` **esh-ml1 built — RTX 2000E Ada as CT 110 on esh-pve (host driver 580.178.04 DKMS, loaded live, no reboot), vLLM embed+rerank at measured parity with fv-ml1, wired as LiteLLM `order: 2` failover; dead DB alias `reranker-a3-bge-v2-m3` (pre-relocation IP) repaired.** → `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md`
|
||||
- `[2026-09-24]` **esh-pve: VM 102 retired, a 14-day hung VFIO process found and cleared, T400 → RTX 2000 Ada; Prime chose LXC + host driver for embed/rerank** (implementation deferred to next session, tracked here + `servers/esh-pve/README.md`). → `persistent-memory.d/2026-09-24-esh-pve-vm102-retired-gpu-swap.md`
|
||||
- `[2026-09-24]` **pfi-gx10 AC-restore UEFI patch applied but NOT validated; box OFF until Prime's AC pull 2026-09-25** (tracking: `servers/pfi-gx10/README.md`, `e5197a3`). — `[2026-09-25]` ✅ **VALIDATED**: Prime pulled and replugged AC, and it came up by itself at 1108:54 (`2b49be4`). → `persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md`
|
||||
- `[2026-09-24]` **NH3 power outage recovered** — pbs-nh3 had no `onboot` (set), NFS boot race fixed with automount (`1cbde50`), every other Claude session on nh3-dev died. → `persistent-memory.d/2026-09-24-nh3-power-outage-recovery.md`
|
||||
- `[2026-09-24]` **Miranda standing order is a repo CLAUDE.md operating parameter** — (`4b29492`, aligned to the global send protocol in `bcf3342`): high-urgency matters go to her, fixed or not… → `persistent-memory.d/2026-09-24-miranda-standing-order-is-a-repo-claude-md-operating.md`
|
||||
- `[2026-09-24]` **task-board mothballed** (Prime): container removed on ana-docker, data/image/compose kept, Kuma monitor deleted, task_* instructions removed from CLAUDE.md and the fork template (`e6da607`). Its hooks had sent no traffic in 30 days.
|
||||
- `[2026-09-24]` **Military 24-hour Pacific clock times** carried into the Codex/Grok shared bootstrap `docs/fleettools/AGENT-BOOTSTRAP.md` (`ad2b4d9`); Claude seats get it from the global CLAUDE.md.
|
||||
- `[2026-09-24]` **Worldtree `admin.memory.forget` stays OFF on demo/personal** until an instance needs it — a destructive erase; enabling it is a per-change operator yes via deploy-wt-config.
|
||||
- `[2026-09-24]` **A git checkout under `root:docker` needs `safe.directory` for its deploy user** — the 09-14 normalization (`826a63b`) silently broke yt-voice-clipper's webhook deploy until v0.3.13; fixed on irv-ml1 and recorded in fleet conventions (`eea9eb2`). Sweep found no other case.
|
||||
- `[2026-09-23]` **elway sudo uploads land root:root, validated and staged; fleet ownership audit built; 33 mis-owned root files fixed on 10 hosts.** → `persistent-memory.d/2026-09-23-elway-ownership-fix-fleet-audit.md`
|
||||
- `[2026-09-23]` **headscale-ddns hardened** — (`fedd4b6`): Cloudflare calls retry and validate every body (`pick()`), no write without both IDs, the run… → `persistent-memory.d/2026-09-23-headscale-ddns-hardened.md`
|
||||
- `[2026-09-23]` **esh-docker-vm restic was skipped 09-22..23 by my own Kuma move** — a dead uptime-kuma lookup aborted `pre-backup.sh` under `set -e` (`25e41d2`). → `persistent-memory.d/2026-09-23-esh-docker-vm-restic-was-skipped-09-22-23-by-my-own-kuma.md`
|
||||
- `[2026-09-23]` **hermes-gateway restart exit-1 is a Hermes race, not a crash** — the planned-stop watcher consumes the marker before SIGTERM re-runs the handler; drop-in… → `persistent-memory.d/2026-09-23-hermes-gateway-restart-exit-1-is-a-hermes-race-not-a-crash.md`
|
||||
- `[2026-09-23]` **Booth link board cleared to 14 durable links** (208 removed: drops, release posts, research links, token-bearing URLs; Scriberr reposted at `scriberr.fv.internal:8080`). Prime's rule: durable debug links only.
|
||||
|
||||
- `[2026-09-22]` **safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes.** — ⚠ The package installs INERT and looks fine — Debian's `/etc/zsh/zprofile` has 0 non-comment lines so the… → `persistent-memory.d/2026-09-22-safe-rm-installed-on-nh3-dev-and-delegated-fleet-wide-to.md`
|
||||
|
||||
- `[2026-09-17]` **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the…** → `persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion.md`
|
||||
- `[2026-09-17]` **The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev** — `dev_launch.py` has zero occurrences of "route", no hook declares one, and every live route was hand-declared… → `persistent-memory.d/2026-09-17-the-althing-route-declaring-sessionstart-hook-is-documented.md`
|
||||
@@ -308,10 +303,14 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
|
||||
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md`
|
||||
|
||||
_214 older entries archived to archival-memory.md._
|
||||
_228 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-10-08]` **NVIDIA's network debian12 repo on trixie via `cuda-keyring`** — trixie's sqv rejects the key's SHA1 binding signature, so `apt update` errors until the list is gone (`dpkg -r` keeps the conffile; it took `dpkg -P`). Fetch the debs and verify InRelease (gpgv) → Packages.gz → sha256 instead. → `servers/fv-ml1/README.md`
|
||||
- `[2026-10-08]` **`secret get <name> | head` "to see its shape"** — a single-value secret IS its shape; it printed the fv-ml1 BMC password into the local transcript. Check with `secret list` or `wc -c`, never by printing.
|
||||
- `[2026-10-07]` **An idle gate counting with `grep -c … || echo ERR`** — `grep -c` exits 1 on a ZERO count, so the value was "0\nERR" and never equalled "0"; two 75-min LiteLLM restart gates were blind to any quiet window. Use `|| true` and test the gate against a known-idle state first.
|
||||
- `[2026-10-07]` **`pgrep -f <pattern>` to check that a process died** — it matched the checking shell's own command line and reported a survivor; check with `ps -eo … | grep '[x]…'`, as the same self-match bit `pkill -f` before.
|
||||
- `[2026-10-06]` **Restarting a user-facing service after an idle check that only PRINTS** — `nvidia-smi … | wc -l` said 1 and the same command restarted Scriberr anyway, killing Prime's transcript run. Make the check gate (`&&` / exit). auto-memory `feedback_idle_check_must_gate_the_restart`.
|
||||
- `[2026-10-06]` **Aging PBS snapshots by directory mtime** — verify jobs and notes edits rewrite `index.json.blob`, so a 10-04 snapshot read as 10-06. Age by the snapshot NAME. auto-memory `reference_pbs_snapshot_mtime_moves_on_verify`.
|
||||
- `[2026-10-06]` **`deploy-stack.sh` on a stack whose host dir holds rollback copies** (albok-service `compose.yaml.pre-*`) — its rsync `--delete` removes them. Upload the single changed file with elway instead.
|
||||
@@ -338,7 +337,5 @@ _214 older entries archived to archival-memory.md._
|
||||
- `[2026-09-25]` **Concluding "not on the UDM" from a port table read 15 s after link-up** — UniFi polls (~60 s); AMT was on port 6 as Prime said. Wait a poll, and prefer a which-port-lists-the-MAC check. auto-memory `feedback_polled_stats_lag_the_event`.
|
||||
- `[2026-09-25]` **The esh-pve NVIDIA DKMS recipe on nh3-pve** — it built, then the kernel refused to load it: nh3-pve has SECURE BOOT ON (esh-pve, the same MS-01, has it off). The installer rolled back. The playbook now pre-flights `mokutil --sb-state` (`7ddd116`).
|
||||
- `[2026-09-25]` **elway against an unpinned host-key name** — every `when:` hit ssh rc 255, so all 9 steps showed SKIPPED, not failed. Only the verify phase went red. Pin the name (the IP key matched) or use the IP. Untracked elway defect: rc 255 in `when:` should fail.
|
||||
- `[2026-09-24]` **Testing "Restore AC Power Loss" with an OS shutdown** — a shutdown stays off BY DESIGN; only pulling and restoring AC tests it. A community README claimed the two are indistinguishable; trusting it cost Prime two trips to the pfi-gx10 power button.
|
||||
- `[2026-09-23]` **`booth link --help`** — there is no help flag; it posts `--help` to the operator's link board as a link. Read `booth` with no args for usage.
|
||||
|
||||
_121 older entries archived to archival-memory.md._
|
||||
_123 older entries archived to archival-memory.md._
|
||||
|
||||
Reference in New Issue
Block a user