memory: snapshot — 2026-10-06 fv-ml1 rebalance, backup checker, Scriberr diarization, albok 0.1.12, gateway retention; in-flight compacted; 17 entries archived
This commit is contained in:
+60
-181
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-10-05 ~2035 PT (new SVOS on nh3-svos deployed + accepted: mesh-only https://svos.phasefinal.com, Basic auth `Vuong`, offline shell, hosting bounds, backup drill; backup check now flags fs-freeze failures (DEGRADED, exit 4) and covers nh3-pve-2; fleet guest patching, reboots and Docker engine bumps done except the held list; fv-ml1 Docker on tank; irv-ml1 cleanup.)_
|
||||
_Last updated: 2026-10-06 ~2145 PT (fv-ml1 KV pins + GPU 0/1 rebalance by usage with contexts at published maximums, gen-small output cap; backup checker triple fixed, destroyed-guest residue pruned, pbs-nh3 mirror retention matched; Scriberr diarization fixed with caches on /tank and no real token; albok-service 0.1.11 → 0.1.12; gateway keeps no content (arbo incognito) + irv-ml1 restic exclude; jekyll moved to sec-reasoning.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -113,45 +113,27 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-10-05 ~2035 PT. The newest subsection is first; older subsections carry their own dates._
|
||||
_As of 2026-10-06 ~2145 PT. The newest subsection is first; older subsections carry their own dates. What was DONE on 10-05/06 lives in Recent decisions; this section keeps only what is open, scheduled or being watched._
|
||||
|
||||
### 2026-10-05: live now (as of 2026-10-05 ~2035 PT)
|
||||
### 2026-10-06: live now (as of 2026-10-06 ~2145 PT)
|
||||
|
||||
- **Nothing is mid-action.** No claims held (`scripts/ops-log claims`); eshpfi pushed through `1210b4a`.
|
||||
- **New SVOS on nh3-svos (VM 111, nh3-pve-2, 10.100.50.43): DEPLOYED and ACCEPTED by svos-dev 2026-10-05 ~2024** (thread `01M47ET6XH0M7JZSH98Z19SKR4` closed). Separate from the High Seat (`svos.service` :8770 on nh3-dev) and Miranda, which are NOT migrated. Front door `https://svos.phasefinal.com`, mesh only, Basic auth login `Vuong` (Prime has the password; vault `nh3-svos/basic-auth-vuong`). ⚠ **After every SVOS release** (`/opt/svos/current` moves): `sudo /usr/local/sbin/svos-front-inventory`, and keep the previous release dir while it is "previous" (svos-dev's runbook carries both). Only open item is Prime's: whether his iPhone home-screen app keeps the login. → `servers/nh3-svos/README.md`
|
||||
- **Backup check (2026-10-06): the 0800 run was clean apart from one false STALE, now FIXED.** nh3-pve-2 coverage read clean, svos drill B was ACKNOWLEDGED, nothing was DEGRADED. The false STALE was `pbs/ana-pve/vm/110`, pfi-ana-webhost, destroyed 10-04 with `qm destroy --purge`. Three fixes landed in `scripts/check-backup-freshness.sh`. infra-hermes, `0a348e5`: destroyed guests are now reported as policy residue (GUEST DESTROYED), and an ssh in the read loop now uses `-n`, because without it the whole PBS layer silently vanished. infra-ops, `d920cf3`: snapshot age now comes from the snapshot NAME, not the directory mtime, which PBS bumps at every verify and every 30-day re-verify. Verified exit 0; positive control at 5h flags 10. Residue PRUNED on Prime's word (2026-10-06 1307): on pbs-ana, ana-pve/vm/110 (8 snapshots) and esh-pve/vm/102 (7, including the protected final 09-10 image, unprotected first; Prime's 09-19 ruling said that VM had no needed state); on the pbs-nh3 mirror, 162 + 10. The checker now reads 0 GUEST DESTROYED, exit 0. **Mirror retention SET (Prime "match history", 2026-10-06 1310):** pbs-nh3 prune job `default-backups-mirror-1b7d4123-` now keeps last 3 / daily 14 / weekly 8 / monthly 12 / yearly 5, the same as pbs-ana. It was keep-all: 3,932 snapshots against 266, 5.49 TiB against 0.93 TiB, nh3-nas at 79%. Dry-run: keep 738, remove 3,194. First run at 00:00 on 10-07, then daily GC. remove-vanished stays false (deliberate). ⚠ **Check on 10-07** that the prune ran and GC freed the space. Old cfg: `/root/prune.cfg.bak-20261006` on pbs-nh3.
|
||||
- **Waiting on Prime (updated 2026-10-05 2035):** push Booth commit `ce095c1` (touch-layout fix, `~/development/booth`, browser test + mutation); a trusted HTTPS name for ESH Homepage (https://10.0.50.45 serves Traefik's default cert; the ESH ACME resolver, Route53, is dead); nh3-dev reboot (runs kernel 6.1.0-49, 6.1.0-53 installed; its engine is ALREADY 29.8.2 since 10-05 1413, so the old "reboot + engine bump" wording was stale, checked 2035); Docker engine bumps held on nh3-docker (carries the post office) and esh-docker-vm (NFS D-state history); fv-ml1's held ZFS 2.3.9, nvidia-container-toolkit 1.20, and a new kernel only together with a newer NVIDIA driver (6.12.111 broke the 580.65 DKMS build). ⚠ **irv-ml1: DO NOT PATCH (Prime).** ana-nas is out of scope.
|
||||
- **Scheduled / to verify:** 10-05 0815 + 0900 infra-hermes jobs (sfsrv-ana `pre-pbs4` snapshot delete after its 04:00 backup; drop `pve/root-pre-pve9` on pfi-pve + esh-pve) — outcome not checked in this session, read `scripts/ops-log tail`; esh-nas-pve `nvme/ROOT@pre-pve9` destroy ~10-10 (ours); 10-11 0540 first weekly sudo-io archiver run (check it), 10-11 0900 infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet.
|
||||
- **Worldtree memory gate: the 20261004T143011Z FAIL did not recur.** That run filed the forbidden `u-negative-ephemeral` fact once (run 1, negative 1/2), and an esh-pve reboot overlapped its run 1. The next two batches PASSED: 20261005T143010Z and 20261006T143010Z (user filing 11/12 = 0.92 in both runs, floor 0.80, negative 2/2). Read from `Worldtree/docs/eval/memory_acceptance_runs/<stamp>/summary.md` on 10-06 after infra-hermes' 0747 PASS report. That is 2 passes after 1 failure on a stochastic gate: a single filing slip is not ruled out, so keep reading the daily verdict rather than calling it fixed.
|
||||
- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm.
|
||||
- **fv-ml1 GPU 0/1 REBALANCED 2026-10-05 2226–2249 (Prime: "accept all that" + "context up to published maximums if at all possible"; `playbooks/fv-ml1-rebalance-20261005.yaml`, elway all green, gateway smoke tests pass).** It follows the 2131 KV pins (meromero 22 GiB, coder pinned, gen-small 5.5 GiB). Why: on a shared card util-sized KV is a co-boot lottery, and GPU 0 creeps to ~60 MiB free after a boot. Rule: KV ≥ 1.25× the seat's max context, rounded up to 0.5 GiB. Two exceptions: meromero and gen-small keep Prime's numbers, and voices gets 1.07× (short prose traffic). Usage basis: 7 days of LiteLLM logs (gen-small 22,396 calls; sec 400; char-rp-fast 55; char-rp 14) plus the engines' own KV logs. **Layout:** GPU 0 = cyberprev, erp-seat, voices, coder, parakeet-nemo (4,036 MiB free right after; ~2.3 GB spare at parakeet's cap; cyberprev may still creep). GPU 1 = gen-small, meromero-rp, intern-decision (10,864 free; ~3.3 GB spare at the 32k peak once gen-small settles). Contexts raised: coder 16,384 → **32,768** (card max), voices 8,192 → **40,960** (native max). YaRN extensions were NOT enabled, and **Prime confirmed 2026-10-06: leave YaRN off for gen-small** (gen-small/cyberprev to ~1M, voices to 131k). They don't fit, and Qwen's own cards warn static YaRN hurts short prompts. intern-decision stays at 32,768 (card default 8,192; base config 262k; ~0.21 MiB/token of VRAM). **gen-small default output cap 32,768** (`--override-generation-config`; an explicit max_tokens still wins, verified in vLLM `get_max_tokens`). It answers 9 runaway ~260k-token replies in 7 days (albok-nemi ×6, all-agents-local ×3, all from nh3-dev); albok-dev told 2250 and closed it 10-06 0000. Nemi now sends max_tokens 8192 (albok-nemi 0.1.2, `tagging.max_tokens`), and its nh3-dev harness configs moved off the shared all-agents-local key onto vault `albok/litellm-key-nemi` + `albok/litellm-key`. The loops came from gen-small at temperature 0 on some memory entries; a capped reply degrades to Nemi's rule-based tagger. ⚠ **Verify both cards' plateaus ~10-07/08** from Beszel (`stats.g.<n>.mu` × 1.024). .env backups `*.env.bak-20261005-rebal`. Earlier the same night: KV pins 2131–2139 (`playbooks/fv-ml1-kv-pin-20261005.yaml`; diagnosis in commit 269c80f). Park 75 dropped by Prime 2115.
|
||||
- **gen-large contention (2026-10-06 1432):** the `jekyll` Hermes seat (nh3-dev, shared all-agents-local key) was sending 165–183k-token prompts every few seconds while Prime ran ~140k from his MacBook. flash-next's KV (344k tokens) overflowed: 539 → 819 preemptions, decode 2–16 tok/s, calls of 221–244 s. **Prime moved jekyll to `sec-reasoning` (cyberprev, GPU 0) at ~1435.** At 1439 gen-large was back to 90–143 tok/s with 1 running; sec had KV at 34% and 0 preemptions. ⚠ Watch: sec's KV is 337k tokens, so jekyll plus a long sec prompt (up to 192k) can overflow it too. jekyll's context grows ~1k/call toward the 262k ceiling. Heavier long-context traffic may grow cyberprev's footprint and eat GPU 0's ~2.3 GB spare; check at the 10-07/08 plateau review.
|
||||
- **Nothing is mid-action.** No claims held (`scripts/ops-log claims`). eshpfi is pushed through `0f6c7a0` (origin/main == HEAD at ~2140). ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- <paths>` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose (auto-regenerated).
|
||||
- **Verify on 2026-10-07:**
|
||||
- **pbs-nh3 mirror prune**: its first run with the new keep rules is at 00:00. Confirm it removed ~3,194 snapshots (dry-run said keep 738), and that the daily GC freed space on nh3-nas (79% before). Old config: `/root/prune.cfg.bak-20261006` on pbs-nh3.
|
||||
- **fv-ml1 GPU 0/1 plateau**: read Beszel `stats.g.<n>.mu` × 1.024 over the 1–2 days since the 10-05 rebalance. Expected: GPU 0 ~2.3 GB spare at parakeet's 3,840 MiB cap, GPU 1 ~3.3 GB spare at intern-decision's 32k peak. GPU 0 now also carries jekyll's long-context load on cyberprev (next bullet), so it may come in lower.
|
||||
- **The 0800 backup check** should read all fresh, with 0 GUEST DESTROYED lines (the residue is pruned).
|
||||
- **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens. jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it, as gen-large did (preemptions; decode down to 2–16 tok/s). jekyll's context grows ~1k/call toward the 262k ceiling. It runs on the shared all-agents-local key; a key of its own would make its traffic attributable.
|
||||
- **Waiting on Prime:**
|
||||
- albok's 200-object load check, 2.01× against a 2× bar (albok-dev: the ruling is the operator's).
|
||||
- The fv-ml1 patch window: held ZFS 2.3.9 (Debian stable-security), nvidia-container-toolkit 1.20, and a new kernel only together with a newer NVIDIA driver (6.12.111 broke the 580.65 DKMS build; test-build the driver first, keep 6.12.74 bootable).
|
||||
- The cleanup bundle offered 10-05: close stale parks 49/46/40; refresh the fv-ml1 README stack tables plus the serial-keyed NVMe table and `zpool replace` line (the 2084-media-error drive is now `/dev/nvme6`, not nvme7); wire ZED pool alerts into althing-alert-bridge; prune the exited one-shot containers on fv-ml1 (keep the `parakeet` rollback).
|
||||
- Push Booth commit `ce095c1`. A trusted HTTPS name for ESH Homepage. nh3-dev reboot (kernel 6.1.0-53 installed). Docker engine bumps held on nh3-docker and esh-docker-vm.
|
||||
- ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope.
|
||||
- **Scheduled / unchecked:** the 10-05 0815 + 0900 infra-hermes jobs (sfsrv-ana `pre-pbs4` snapshot delete; drop `pve/root-pre-pve9` on pfi-pve + esh-pve) left no ops-log lines; ask infra-hermes or read the hosts. esh-nas-pve `nvme/ROOT@pre-pve9` destroy ~10-10 (ours). 10-11 0540: first weekly sudo-io archiver run (check it). 10-11 0900: infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet. Next scheduled tank scrub on fv-ml1: Sun 10-11 0024 (the September slot was missed); check it ran.
|
||||
- **Worldtree memory gate:** the 10-04 FAIL did not recur (10-05 and 10-06 PASS, filing 0.92, floor 0.80). It is a stochastic gate, so keep reading the daily verdict (infra-hermes reports it).
|
||||
- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`.
|
||||
- **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in.
|
||||
|
||||
### Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01)
|
||||
|
||||
- **INCIDENT 04:21 PT 2026-10-01: FIXED by nemo-0.1.1 (infra-hermes 963f9ed, live ~0441, infra-ops spot audit passed).**
|
||||
- Cause: gen-small hit CUDA OOM on a 394 MiB lazily allocated workspace while the seat was parked at its 3,582 MiB window cache (GPU 0 Free 388).
|
||||
- The fix: `empty_cache` around every window; `MEM_CAP_MIB=3840` (a hard process ceiling); `CUDA_GRAPHS=0`, because empty_cache poisons the graph pool (an illegal memory access on the next request).
|
||||
- Result: the seat rests at 2,108 MiB and peaks at 3,028 during a 12-min file. gen-small is steady at 36,116 (its runtime growth landed at init, with zero movement on later requests). GPU 0 Free is ~1.05 GB at rest.
|
||||
- Cost: graphs-off is ~2–8 ms slower at short clips and ~27 ms at 20–60 s (35 / 40 / 50 / 98 ms against 33 / 36 / 42 / 71), still 4–15× faster than the old seat.
|
||||
- **Lesson: a shared-card tenant must carry a HARD cap, whether or not it returns memory.**
|
||||
- **Prime ~0445 2026-10-01: leave both as is.** gen-small keeps its 8 GiB KV pin (no insurance trim), and CUDA graphs stay OFF (the few-ms cost is accepted; no graph-pool rework).
|
||||
- **Rollback:** `docker stop parakeet-nemo && docker start parakeet`. The old container and image are kept.
|
||||
- **GPU 0 is FULL:**
|
||||
- The seat's steady state is **3,582 MiB** (its cached window peak); Free is **385 MiB**.
|
||||
- `vllm-gen-small` runs at util 0.36, and its `.env` holds **0.33** for the next restart (~3 GiB of boot-check margin). Its KV is byte-pinned: 670,142 tokens / 2.56×.
|
||||
- Before restarting any vLLM seat on this card, check that util × 95.6 GiB ≤ measured Free + the seat's own resident memory. Do not trial-boot. A trial-boot sequence took gen-small down for 34 min on 2026-10-01.
|
||||
- Seat invariants (in its README): cast to bf16 AFTER change_attention_model; uvicorn pinned with `--http h11` (httptools 0.8.0 emits `HTTP/1.1 200\x00OK`, which LiteLLM/httpx rejects).
|
||||
- NVIDIA Open Model License accepted for internal use. → `persistent-memory.d/2026-09-30-parakeet-seat-switch-approved.md`
|
||||
|
||||
### Worldtree U11 memory cutover (demo + personal)
|
||||
|
||||
- **Legacy plane OFF since 0115/0120 PT 2026-09-30** (config repo 63cf268; personal /metrics b6fdd81; the repo is pushed). → `persistent-memory.d/2026-09-30-worldtree-u11a-off-u11b-gate.md`
|
||||
- **U11b STEP 5 DONE 2026-10-02 14:54:03Z on Prime's direct go** (in this channel, ~0750 PT). H2 re-check exit 0 on both (demo rows 4/12/5, personal 0/6/2200, agent_self 0); deleted `agents/{forseti,lofn,mimir}/memory/<agent>.chroma` + `memory/context_promotion` on demo and personal with literal paths; apis healthy, no restart; stamp sent to worldtree-dev (b193 is theirs to cut). Demo had already lost 22 context_promotion JSONL (2026-07-03) to ledger.py's 90-day `JSONL_RETENTION_DAYS` sweep on 10-01 07:10:30 — TTL, not an erasure; flagged to worldtree-dev as retirement owner.
|
||||
- **b193 is cut and pushed (c2d87263, worldtree-dev 2026-10-02); demo deploys via CI, personal on Prime's staging tag.** `scripts/wt-memory-gate-batch` now passes `--legacy-mode` only when the DEPLOYED sha's harness still has it (git grep at that sha; tested b192 → passed, b193 → omitted), so no timed edit is needed. ⚠ It will REFUSE (exit 3) while demo's deployed sha lags a tree HEAD whose pyproject/uv.lock/packages changed (b193, then the albok commit): by design. **Still mine after b193 deploys:** remove the residue `memory/context_promotion/ledger.db` a b192 restart may have re-created; then the parity config retirements (detail file). Verdict ccs come FROM `hermes-gateway` (never read); reply to `infra-hermes`.
|
||||
- **Legacy archive:** DESTROY it whole by **2026-10-30**, or at retirement-done, or on a subject-erasure request, whichever comes first. The runbook is in the detail file.
|
||||
- **TODO:** re-sweep both api logs after real traffic. After the b193 push, remove the retired config keys (list: detail file, "Post-b193 config retirements").
|
||||
- **Older open items, one line each:** esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (`stacks/blender/README.md`). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran `51168a2e` on 09-27). nh3-dev's root disk sits near the Beszel 85% line. ESH has a single outside route (esh-scale on esh-pve), noted and untracked.
|
||||
|
||||
### fv-ml1 GPU layout (as of 2026-10-05 2249)
|
||||
|
||||
@@ -167,136 +149,49 @@ _As of 2026-10-05 ~2035 PT. The newest subsection is first; older subsections ca
|
||||
| 1 | intern-decision | 32,768/call | cap 14.4 GiB, peak 15,220 | — | — |
|
||||
| 2 | flash-next (gen, gen-large, +6) | 262,144 | — | 344,155 | 1.31× |
|
||||
|
||||
Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-memory`), so util only gates the boot check (Free at start ≥ util × total). flash-next is alone on GPU 2 and stays util-sized (0.96).
|
||||
Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-memory`), so util only gates the boot check (Free at start ≥ util × total). flash-next is alone on GPU 2 and stays util-sized (0.96). ⚠ Before restarting any seat on a shared card, gate on the arithmetic (`scripts/oneoff/fv-ml1-gpucheck.sh`), never trial-boot. Before restarting a user-facing service, the idle check must STOP the restart, not just print (auto-memory `feedback_idle_check_must_gate_the_restart`).
|
||||
|
||||
- **GPU 3:** clear for ad-hoc work (Prime, 2026-10-05); no always-resident tenant goes here. On-demand tenants: Blender, Scriberr (0 idle, ~5.5 GB per job), and MIA auto-rig (one-shot `scripts/mia-run`, peak 3.4 GB, added 2026-10-01 for dread-dev). When a full-size seat claims GPU 3, Scriberr steps aside to **irv-ml1's A6000**, not back to GPU 1.
|
||||
|
||||
### Scriberr (fv-ml1 GPU 3)
|
||||
|
||||
- **LIVE `scriberr:local-blackwell-a353078-dropout2`:** upstream a353078 plus patch 0001 (overlap slicer) and patch 0002 (gap retry, `PARAKEET_MODEL_PATH`), carried LOCALLY ONLY (Prime 2026-10-01: no upstream). `patches/0003` (pyannote env pins) is committed but in no image yet. `scripts/scriberr-rebuild` re-applies all. → `persistent-memory.d/2026-09-30-scriberr-slicer-gap-retry-gpu3.md`
|
||||
- **Diarization works (fixed 2026-10-06):** model caches on /tank, `HF_TOKEN` a placeholder, saved profiles `WhisperX large-v3 · GPU · speakers` and `Parakeet · GPU · speakers`. A NEW gated model needs one download with vault `fv-ml1/hf-token`. → `persistent-memory.d/2026-10-06-scriberr-diarization-fixed.md`
|
||||
- **URL is https://scriberr.nh3.phasefinal.com since 2026-10-01** (fronted by the fleet TLS caddy on nh3-dev; Caddyfile backup `.bak-20261001-scriberr`). `getUserMedia` is refused on http://, so recording needs the HTTPS name, which is in `ALLOWED_ORIGINS` (`playbooks/scriberr-https-origin.yaml`). ⚠ `/opt/docker/compose/scriberr` is lkraven-owned, so `deploy-stack.sh` as infra-ops FAILS there; upload single files with `elway --upload --sudo --owner lkraven:lkraven`.
|
||||
|
||||
### Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01)
|
||||
|
||||
- **Incident 2026-10-01 04:21, fixed by nemo-0.1.1** (infra-hermes 963f9ed). gen-small OOM'd on a lazily allocated workspace while the seat held its window cache. The fix: `empty_cache` around every window, `MEM_CAP_MIB=3840` (a hard ceiling), `CUDA_GRAPHS=0` (empty_cache poisons the graph pool). The seat rests at 2,108 MiB and peaks at 3,028 on a 12-min file. **Lesson: a shared-card tenant must carry a HARD cap, whether or not it returns memory.** Prime kept graphs OFF. His 10-01 "no insurance trim" on gen-small was superseded on 10-05 (KV 8 → 5.5 GiB) once the GPU 0 plateau turned out to be ~60 MiB free, not the ~1.05 GB read just after a restart.
|
||||
- **Rollback:** `docker stop parakeet-nemo && docker start parakeet` (old container and image kept). Seat invariants (in its README): cast to bf16 AFTER change_attention_model; uvicorn pinned with `--http h11` (httptools 0.8.0 emits `HTTP/1.1 200\x00OK`, which LiteLLM/httpx rejects). NVIDIA Open Model License accepted for internal use. → `persistent-memory.d/2026-09-30-parakeet-seat-switch-approved.md`
|
||||
|
||||
### Worldtree U11 memory cutover (demo + personal)
|
||||
|
||||
- **Legacy plane OFF since 0115/0120 PT 2026-09-30** (config repo 63cf268; personal /metrics b6fdd81; the repo is pushed). → `persistent-memory.d/2026-09-30-worldtree-u11a-off-u11b-gate.md`
|
||||
- **U11b STEP 5 DONE 2026-10-02 14:54:03Z on Prime's direct go** (in this channel, ~0750 PT). H2 re-check exit 0 on both (demo rows 4/12/5, personal 0/6/2200, agent_self 0); deleted `agents/{forseti,lofn,mimir}/memory/<agent>.chroma` + `memory/context_promotion` on demo and personal with literal paths; apis healthy, no restart; stamp sent to worldtree-dev (b193 is theirs to cut). Demo had already lost 22 context_promotion JSONL (2026-07-03) to ledger.py's 90-day `JSONL_RETENTION_DAYS` sweep on 10-01 07:10:30 — TTL, not an erasure; flagged to worldtree-dev as retirement owner.
|
||||
- **b193 is cut and pushed (c2d87263, worldtree-dev 2026-10-02); demo deploys via CI, personal on Prime's staging tag.** `scripts/wt-memory-gate-batch` now passes `--legacy-mode` only when the DEPLOYED sha's harness still has it (git grep at that sha; tested b192 → passed, b193 → omitted), so no timed edit is needed. ⚠ It will REFUSE (exit 3) while demo's deployed sha lags a tree HEAD whose pyproject/uv.lock/packages changed (b193, then the albok commit): by design. **Still mine after b193 deploys:** remove the residue `memory/context_promotion/ledger.db` a b192 restart may have re-created; then the parity config retirements (detail file). Verdict ccs come FROM `hermes-gateway` (never read); reply to `infra-hermes`.
|
||||
- **Legacy archive:** DESTROY it whole by **2026-10-30**, or at retirement-done, or on a subject-erasure request, whichever comes first. The runbook is in the detail file.
|
||||
- **TODO:** re-sweep both api logs after real traffic. After the b193 push, remove the retired config keys (list: detail file, "Post-b193 config retirements").
|
||||
|
||||
### intern-decision (replaced SemIf on 2026-09-30)
|
||||
|
||||
- **LIVE 0.1.3** at `intern-decision.fv.internal:8033`: semif-compatible `/decide` plus Jev `/v1/systemone`, 32k tokens, a Triton cache volume. Run `scripts/intern-decision-warmup` after an IMAGE change. → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md`
|
||||
- **Open:** label ~50 real Wyrd/Cicada turns before trusting it in production; its card makes no contamination claim.
|
||||
|
||||
### Scriberr (fv-ml1 GPU 3)
|
||||
### nh3-pve + nh3-ml1 (2026-09-25/26)
|
||||
|
||||
- **Diarization FIXED 2026-10-06 ~1320 (Prime: "whisper doesn't work at all, need diarized transcripts").** Three faults: no `HF_TOKEN` (pyannote is gated). FINAL: the gated models were downloaded once, caches moved to /tank (`HF_HOME`/`TORCH_HOME`/`NLTK_DATA` under `/app/whisperx-env`), and `HF_TOKEN` is a placeholder (Prime 1330: "move the downloads and drop the token"). Cached gated models load with an invalid token; a NEW gated model needs one download with vault `fv-ml1/hf-token`. Saved profiles added: `WhisperX large-v3 · GPU · speakers` and `Parakeet · GPU · speakers`. Prime's 1 h transcript finished diarized at 1336 (3 speakers, 3m30s). ⚠ My 1330 recreate killed his 1326 run (an idle check that printed but did not gate); see auto-memory `feedback_idle_check_must_gate_the_restart`. torchcodec 0.16 in the pyannote env against torch 2.8 (`AudioDecoder` undefined; pinned to 0.7 by hand, durable as `patches/0003`, applied at the next rebuild); and the WhisperX adapter's CPU/small/float32 defaults. Both diarization paths verified on GPU 3. → `stacks/scriberr/README.md` § Diarization
|
||||
|
||||
- **LIVE `scriberr:local-blackwell-a353078-dropout2`:** upstream a353078 plus patch 0001 (overlap slicer) and patch 0002 (gap retry, `PARAKEET_MODEL_PATH`), carried LOCALLY ONLY (Prime 2026-10-01: no upstream). v3 stays. `scripts/scriberr-rebuild` re-applies both. → `persistent-memory.d/2026-09-30-scriberr-slicer-gap-retry-gpu3.md`
|
||||
- **URL is https://scriberr.nh3.phasefinal.com since 2026-10-01** (fronted by the fleet TLS caddy on nh3-dev; Caddyfile backup `.bak-20261001-scriberr`). Prime's "can't initialize recorder" was the SecureContext rule: `getUserMedia` is refused on http://. The HTTPS name is in `ALLOWED_ORIGINS` (`playbooks/scriberr-https-origin.yaml`); http://10.251.50.54:8080 still works except for recording. ⚠ `/opt/docker/compose/scriberr` is lkraven-owned, so `deploy-stack.sh` as infra-ops FAILS there (Permission denied); upload single files with `elway --upload --sudo --owner lkraven:lkraven`.
|
||||
|
||||
### nh3-pve + nh3-ml1: post-visit, all live (2026-09-25/26)
|
||||
|
||||
- **nh3-pve:** Secure Boot OFF; IGFX restored; NVIDIA 580.178.04 DKMS; kernel
|
||||
6.8.12-43 (benign btmtk oops every boot). **AMT live**: static
|
||||
`10.100.250.61` on nh3-mgmt (UDM port 6), KVM on, Opt-in None, password in the
|
||||
vault as `nh3-pve/amt-admin`. → `servers/nh3-pve/README.md`
|
||||
- **nh3-ml1** (CT 109 @ `10.100.50.80`):
|
||||
- TEI rerank is **load-shared with esh-ml1** through the gateway (`enable_weighted_failover`). Embeddings are
|
||||
**primary/failover since 2026-10-05**: `qwen3-embedding` esh-ml1 first, `qwen3-embedding-bulk` (albok writes) nh3-ml1 first.
|
||||
- brokkr's foundry seats:
|
||||
- `lfm-vl` `:8030` (gateway `lfm25-vl-3b`);
|
||||
- `lfm-vl-uncensored` `:8032` (direct; passed brokkr's eval);
|
||||
- `vibevoice-asr` `:8031` (audio.cpp, **Q8_0**).
|
||||
- GPU ~11.3/16 GB. → `servers/nh3-ml1/README.md`
|
||||
- Not yet proven: nh3-ml1 coming back by itself after an nh3-pve reboot (only the
|
||||
config was checked).
|
||||
|
||||
### esh-matter: Matter server for Home Assistant (2026-09-26)
|
||||
|
||||
- CT 111 @ `10.0.90.20`, on VLAN 90 (esh-iot) only. matter.js 1.4.0; `:5580` is
|
||||
firewalled to HA `10.0.50.46`. HA's matter integration is loaded (ha-dev).
|
||||
- **Next is Prime's:** share the Aqara W200 into HA via Matter multi-admin. If
|
||||
commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob
|
||||
(a Prime/infra-ops change). → `servers/esh-matter/README.md`
|
||||
|
||||
### Worldtree reward path: code + config live (2026-09-27, Prime)
|
||||
|
||||
- `d8726a13` (Domari's Skywork/Selene fix) was ALREADY on origin/main when Prime said to push it: it
|
||||
was pushed as part of `79cce92f` at 1221. Demo runs image `79cce92f`. **Personal still runs
|
||||
`51168a2e`**, which is older and lacks the fix; its image follows worldtree CI, never a manual up.
|
||||
- `deploy-wt-config` deployed demo and personal (1437): `providers.yaml` `reward_models.scalar-judge`
|
||||
(6d4ac44), plus a77639d's corrected selene description, which had never reached the hosts. Both
|
||||
are healthy, with 0 drift. It is inert on personal until that image picks up d8726a13.
|
||||
- worldtree-instance-configs: its 6 long-unpushed commits (53349f8…6d4ac44, Aug 2 to Sep 27) were
|
||||
**pushed on Prime's go-ahead** (a9d091e..6d4ac44). Origin, the repo and both hosts now agree.
|
||||
|
||||
### Blender on fv-ml1 GPU 3, agent-driven (2026-09-27, Prime)
|
||||
|
||||
- Prime: "go ahead with gpu 3, both", and he does not use Blender, so **agents drive it through MCP**.
|
||||
Blender 5.2.2 LTS (linuxserver/selkies image, digest-pinned) with a web desktop at
|
||||
`https://10.251.50.54:3001` (vault `fv-ml1/blender-web-password`). **On demand only:**
|
||||
`scripts/blender-mcp up|down|status`. It was left DOWN (GPU 3 back to 2 MiB).
|
||||
- MCP: `mcp-for-blender` 2.1.1 (Dvalin's research pick). The add-on is vendored at `41a18432`. The
|
||||
server runs INSIDE the container, and agents reach it as ssh+docker-exec stdio via
|
||||
`scripts/blender-mcp`. No port is published, and screenshots work because server and Blender
|
||||
share a filesystem. Telemetry is off and safe mode is on. Verified end to end (render, screenshot,
|
||||
safe-mode refusal). **Batch path: `scripts/blender-run`** (one-shot `docker run --rm`, `--job` staging; first user is draupnir). **Access: the shared fleet `infra-ops` login, with no render-only key (Prime, 2026-09-28).** **Registration + up/down: PER WORKING SESSION (Prime, 2026-09-28, relayed by draupnir; superseded per-task of 09-27)**: `blender-mcp up` + `claude mcp add` at session start, `claude mcp remove` + `down` at its end. Never always-on, never user- or project-wide config.
|
||||
→ `stacks/blender/README.md`
|
||||
- **Extensions (2026-09-28, draupnir; Prime ruled Blender a MANDATORY pipeline stage):** 8 pinned
|
||||
add-ons (`stacks/blender/extensions.lock`) built by `scripts/blender-extensions sync` into
|
||||
`fv-ml1:/tank/blender-extensions/5.2/system` (LIVE), mounted read-only as the System repo;
|
||||
`fleet_extensions.py` enables them (GUI startup timer; `blender-run --extensions`). Headless
|
||||
acceptance 8/9 (CAD Sketcher sketching is GUI-only); **MCP 9/9 after Prime ran the deploy himself
|
||||
(1505; the classifier had refused mine)**. SurfacePsycho's eval() is patched to literal_eval
|
||||
(held under MCP). Open: agent-drawn CAD Sketcher geometry (its stateful ops want point picks),
|
||||
and MeasureIt overlays not seen in MCP screenshots. GUI left DOWN.
|
||||
|
||||
### Zigbee2MQTT on esh-docker-vm (2026-09-27, Prime go-ahead; ha-dev request)
|
||||
|
||||
- **LIVE since 1240:** Z2M 2.14.1 (digest-pinned) on `http://10.0.50.45:8099` (auth token), radio
|
||||
SLZB-MR1U chip 0 `tcp://10.0.90.10:6638` (ember), channel 25, **PAN 0xCFF4**. State and the network
|
||||
key are in `/opt/docker/data/zigbee2mqtt` (root 0700, restic via /opt/docker, never in git). Vault:
|
||||
`esh-docker-vm/zigbee2mqtt-{network-key,frontend-token,mqtt-password}`. Broker user `zigbee2mqtt`
|
||||
added to mosquitto (passwd backup `.bak-20260927-z2m`).
|
||||
- **ha-dev deleted the ZHA entry**; pairing the Aqara T1 is theirs. ⚠ The "bridge in HA" acceptance first
|
||||
FAILED because HA had had no MQTT since 2026-09-25 06:13. Cause: the esh-docker-vm macvlan shim had no
|
||||
host route to HA (two equal /24s, ens18 wins). Fixed at 1249 with a /32 via the shim (if-up.d hook,
|
||||
`playbooks/esh-docker-vm-macvlan-shim-route.yaml`), and HA reconnected. My first report called HA
|
||||
"connected" from a client count that included my own probe; `$SYS` counts are not identities.
|
||||
- ⚠ The first credential-mint attempt was blocked by the auto-mode classifier on a peer-relayed
|
||||
approval. It went ahead only on Prime's own go-ahead in this session. Treat that as correct.
|
||||
→ `stacks/zigbee2mqtt/README.md`
|
||||
|
||||
### restic: credential leak fixed (2026-09-27, Prime)
|
||||
|
||||
- Seven hosts were moved from `env-file` to `repository-file` (`playbooks/restic-repository-file.yaml`),
|
||||
so the systemd units no longer carry the rest-server password. Secrets are vaulted as
|
||||
`<host>/etc/restic/{repository,password}`. `restic.env` is KEPT for manual snippets, so
|
||||
**a rotation must update the vault, `restic.env` and `repository`.**
|
||||
- **vm-esh-nas done too** (Prime bootstrapped infra-ops there at uid 850, NOPASSWD). **All 8 hosts are clean
|
||||
and vaulted.**
|
||||
- The passwords were readable until today, so the rotation (Prime's) remains the real fix.
|
||||
- 2026-09-27 0859 freshness check: all restic and PBS repos ✅. Those 0100 runs PRE-date the move
|
||||
(done ~0140–0200).
|
||||
- **VERIFIED 2026-09-28 (infra-hermes, read-only):** the first nightly on `repository-file` (0100 PT)
|
||||
succeeded on all 8 hosts (unit Result=success AND a today-dated snapshot in each repo), and the
|
||||
0800 freshness check was all-green, PBS included. nh3-dev also passed a content restore (`af5580c3`).
|
||||
|
||||
### augaman: face recognition for Cicada
|
||||
|
||||
- **v0.1.3 LIVE on esh-ml1:8040** (`stacks/augaman`), the only instance (fv-ml1's was benched and removed 2026-09-27). Gallery
|
||||
backup wired and identity-level restore verified (restic → rest-server-ana `esh-ml1/`), so real enrollments are allowed.
|
||||
48 ms per face. CPU-mode regression is augaman-dev's. Nothing owed by infra-ops.
|
||||
|
||||
### esh-ml1
|
||||
|
||||
- The reward seat's consumer is FIXED: Worldtree d8726a13 speaks vLLM /classify through the
|
||||
gateway `/scalar-judge` passthrough, which is key-gated per key via `allowed_passthrough_routes`,
|
||||
granted by infra-hermes on 2026-09-27. It is live on demo; personal waits on a newer image
|
||||
(see "Worldtree reward path").
|
||||
- The nh3-docker Dozzle agent has been stopped by hand since ~2026-04. Revive or
|
||||
drop.
|
||||
- The Beszel superuser password was echoed into a session transcript (local
|
||||
only, not in memory). Rotation offered to Prime.
|
||||
|
||||
### Live threads
|
||||
|
||||
- git: **eshpfi-management pushed through `1210b4a` (2026-10-05 ~2020); worldtree-instance-configs at `b6fdd81`.** ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- <paths>` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose: it is auto-regenerated.
|
||||
- nh3-dev root disk was cleaned 2026-09-30 1704 (uv prune, dangling images, old build cache): 86% → 82%. The Beszel 85% alert flaps near the line.
|
||||
- Booth submit-all fix (Prime's report) is LIVE since 2026-09-27 1705, via booth-dev (booth `50bfc7b`).
|
||||
Pushing it is booth's call, per Prime; it is not ours.
|
||||
- ESH has a single outside route (esh-scale on esh-pve). Noted, untracked.
|
||||
- **nh3-pve:** Secure Boot OFF; IGFX restored; NVIDIA 580.178.04 DKMS. **AMT live**: static `10.100.250.61` on nh3-mgmt (UDM port 6), KVM on, Opt-in None, password in the vault as `nh3-pve/amt-admin`. → `servers/nh3-pve/README.md`
|
||||
- **nh3-ml1** (CT 109 @ `10.100.50.80`): TEI rerank load-shared with esh-ml1 through the gateway; embeddings primary/failover since 2026-10-05 (`qwen3-embedding` esh-ml1 first, `qwen3-embedding-bulk` for albok writes nh3-ml1 first). brokkr's foundry seats: `lfm-vl` `:8030` (gateway `lfm25-vl-3b`), `lfm-vl-uncensored` `:8032` (direct), `vibevoice-asr` `:8031` (audio.cpp, Q8_0). GPU ~11.3/16 GB. → `servers/nh3-ml1/README.md`
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-10-06]` **Scriberr diarization fixed: gated pyannote, a torchcodec/torch mismatch, and WhisperX's CPU defaults; model caches on /tank, HF_TOKEN a placeholder (Prime), GPU+speaker profiles.** Patch 0003 waits for the next rebuild (tracked at `stacks/scriberr/patches/README.md`). → `persistent-memory.d/2026-10-06-scriberr-diarization-fixed.md`
|
||||
- `[2026-10-06]` **Backup checker: destroyed guests reported as residue, an ssh stdin swallow fixed, snapshot age from the name; residue pruned (Prime); pbs-nh3 mirror keeps pbs-ana's history (Prime "match history"), first prune 10-07 00:00.** → `persistent-memory.d/2026-10-06-backup-checker-fixes-and-mirror-retention.md`
|
||||
- `[2026-10-06]` **albok-service 0.1.11 (0202) then 0.1.12 (1254) deployed on nh3-albok at albok-dev's asks 7991/8199**, built from clean archives, pinned by digest, single-file upload (deploy-stack's `--delete` would wipe the on-host rollback copies). Every wing is now migrated to generation collections, so a 0.1.10 rollback needs a full reindex. albok's 2.01×-vs-2× load check awaits Prime. → `stacks/albok-service/README.md`
|
||||
- `[2026-10-06]` **The LiteLLM gateway keeps no request/response content, and `store_prompts_in_spend_logs: false` is now a marked consumer guarantee (comfy-dev, arbo incognito).** Proved with a canary plus positive controls; per-request `no-log` does NOT cover the proxy's own spend-log callback (v1.97 source). irv-ml1 restic also excludes `input/incognito-*` (dry-run proved with a canary and a control). `847424c`, `0f6c7a0`.
|
||||
- `[2026-10-06]` **jekyll moved from gen-large to sec-reasoning (Prime)** after its ~180k-token agent loop and Prime's ~140k session overflowed flash-next's KV. → `persistent-memory.d/2026-10-06-gen-large-contention-jekyll.md`
|
||||
- `[2026-10-06]` **vh/rea created on gitea as a private pull mirror of github.com/morluto/rea (8h, wiki + LFS), matching the vh/ mirror convention.** Content verified (main sha, 20 branches, 29 tags).
|
||||
- `[2026-10-05]` **fv-ml1: vLLM KV pinned in bytes on every shared-card seat, GPUs 0/1 rebalanced by measured usage, contexts at published (native) maximums, gen-small default output cap 32,768; YaRN stays OFF (Prime).** → `persistent-memory.d/2026-10-05-fv-ml1-kv-pins-and-rebalance.md`
|
||||
- `[2026-10-05]` **Park 75 (TTS stack → fv-ml1) DROPPED (Prime, 2115):** its trigger fired 09-25, but intern-decision used the room, and the cross-site-hop objection still stands. Henge comment 63.
|
||||
- `[2026-10-05]` **nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only `https://svos.phasefinal.com` with HTTP Basic auth, login `Vuong` (Prime, direct)** — WhoIs gate retired (all mesh nodes are login `pfi`; Prime declined a mesh restructure). Password vaulted `nh3-svos/basic-auth-vuong`. qemu-ga freeze hooks were never run on Debian; now `configs/qemu-ga/fsfreeze-dispatch` (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"); the backup check now flags that as DEGRADED (exit 4) and covers nh3-pve-2, which it had never read. Offline shell (+ previous release's assets) served before auth; rerun `svos-front-inventory` after every SVOS release. Hosting bounds 1 MiB / 15 s body / 5 s headers. → `servers/nh3-svos/README.md`.
|
||||
- `[2026-10-05]` **ESH Homepage also on https://10.0.50.45 (Prime)** — a Traefik file-provider router (`homepage-https.yml`, `tls: {}`) on esh-docker-vm serves Traefik's default cert; the ESH ACME resolver (Route53) is dead, so a trusted name is Prime's open call. (`2c4c7ca`)
|
||||
- `[2026-10-05]` **bench-jev servers killed and removed, their HF cache cleaned (Prime)**; pfi-pteradactyl fixed + patched + engine upgraded (`3429b5c`; manual dockerd + wings restart).
|
||||
@@ -349,9 +244,10 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
- `[2026-09-28]` **Blender extensions live in a read-only System repo built from a sha256 lock, enabled by a hook, opt-in for blender-run (`--extensions`).** SurfacePsycho's eval() is patched to literal_eval (a proven safe-mode escape). → `stacks/blender/README.md` § Extensions
|
||||
- `[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** → `persistent-memory.d/2026-09-27-hermes-gateway-restarted-0401-for-highseat-dev.md`
|
||||
- `[2026-09-27]` **SemIf LIVE on fv-ml1 GPU 1 (semif-serve 0.1.2, Prime):** wrapper + contract + 39 tests, 142/144 upstream parity, two card-only memory defects fixed. → `persistent-memory.d/2026-09-27-semif-live-on-fv-ml1-gpu1.md`
|
||||
- `[2026-09-27]` **Blender 5.2 on fv-ml1 GPU 3 (on demand), agent-driven via mcp-for-blender running in-container over ssh stdio; safe mode on, no published port.** → `stacks/blender/README.md`
|
||||
- `[2026-09-27]` **Blender 5.2 on fv-ml1 GPU 3 (on demand), agent-driven via mcp-for-blender running in-container over ssh stdio; safe mode on, no published port.** → `persistent-memory.d/2026-09-27-blender-on-fv-ml1-gpu3.md` (canonical: `stacks/blender/README.md`)
|
||||
- `[2026-09-27]` **Worldtree reward path: d8726a13 + `providers.yaml` `reward_models.scalar-judge` live on demo; personal waited on a newer image.** → `persistent-memory.d/2026-09-27-worldtree-reward-path-live.md`
|
||||
- `[2026-09-27]` **esh-docker-vm host→HA traffic needs a /32 over macvlan-shim (if-up.d hook); without it HA silently loses MQTT (3× since August).** → `servers/esh-docker-vm/README.md`
|
||||
- `[2026-09-27]` **Zigbee2MQTT live on esh-docker-vm :8099 (PAN 0xCFF4, ch 25) replacing ZHA; network key vaulted, data host-only.** Prime go-ahead in this session; a peer-relayed approval was blocked by the permission gate. → `stacks/zigbee2mqtt/README.md`
|
||||
- `[2026-09-27]` **Zigbee2MQTT live on esh-docker-vm :8099 (PAN 0xCFF4, ch 25) replacing ZHA; network key vaulted, data host-only.** Prime go-ahead in this session; a peer-relayed approval was blocked by the permission gate. → `persistent-memory.d/2026-09-27-zigbee2mqtt-on-esh-docker-vm.md` (canonical: `stacks/zigbee2mqtt/README.md`)
|
||||
- `[2026-09-27]` **semif-serve 0.1.4: object states ending in `)`, `;` or `}` no longer 422 (INV-7, a prefix wrapper proven at startup); numerics are deterministic within a process but a bf16 near-tie can flip across a restart.** Prime ruled; heid bug hunt folded. → `stacks/semif/README.md`
|
||||
- `[2026-09-27]` **SemIf as Cicada's mood source: slower (+32 ms async, +94 ms sequential) and worse (67% vs 92% apt; carry 7/15 vs 14/15); only the gesture restraint is a win.** Build nothing (Prime). Henge 88 carries it. → `persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md`
|
||||
- `[2026-09-27]` **SemIf consumer-fit spikes (Prime): Cicada affect gate 30/31 with descriptive wording and 19/31 terse; Wyrd "left this place?" 21/21 on the second wording, exit choice 18/21.** Recommendations await Prime. → `persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md`
|
||||
@@ -396,45 +292,28 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
- `[2026-09-23]` **hermes-gateway restart exit-1 is a Hermes race, not a crash** — the planned-stop watcher consumes the marker before SIGTERM re-runs the handler; drop-in… → `persistent-memory.d/2026-09-23-hermes-gateway-restart-exit-1-is-a-hermes-race-not-a-crash.md`
|
||||
- `[2026-09-23]` **Booth link board cleared to 14 durable links** (208 removed: drops, release posts, research links, token-bearing URLs; Scriberr reposted at `scriberr.fv.internal:8080`). Prime's rule: durable debug links only.
|
||||
|
||||
- `[2026-09-22]` **Both carried calls approved** — build the `NRestarts` flap sampler (`163bb97`); the restic content-assertion ruling is ratified and stays (`ba60fda`).
|
||||
- `[2026-09-22]` **safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes.** — ⚠ The package installs INERT and looks fine — Debian's `/etc/zsh/zprofile` has 0 non-comment lines so the… → `persistent-memory.d/2026-09-22-safe-rm-installed-on-nh3-dev-and-delegated-fleet-wide-to.md`
|
||||
- `[2026-09-22]` **The acceptance probe for a guard must not be able to destroy what it tests** — (infra-hermes). → `persistent-memory.d/2026-09-22-the-acceptance-probe-for-a-guard-must-not-be-able-to.md`
|
||||
- `[2026-09-22]` ⚠ **`{"sent": true}` is a claim about transmission, never about effect.** `pane_send` structurally cannot deliver a harness command (every relay is prefixed with the card id), so D-0010 promised an unachievable `/clear` and its receipt reported success. Consumed an operator approval. → `persistent-memory.d/2026-09-22-instrument-errors.md`
|
||||
- `[2026-09-22]` **D-0010/D-0011 were misrouted to this seat by `pane_find` matching a ROLLING PANE TITLE.** — Genuine and operator-approved, wrong seat; `fleet_telemetry` held the right mapping and carries the warning… → `persistent-memory.d/2026-09-22-d-0010-d-0011-were-misrouted-to-this-seat-by-panefind.md`
|
||||
- `[2026-09-22]` **`headscale-ddns` exited 1 silently and the alarm carried no cause** → `persistent-memory.d/2026-09-22-headscale-ddns-exited-1-silently-and-the-alarm-carried-no.md`
|
||||
|
||||
- `[2026-09-22]` ⭐⭐ **Five instrument errors in one day, and the shape is one thing: a tool that enumerates "things that are fine" has selected against its own subject.** `--state=running` skipped the units most needing hooks; `awk '{print $1}'` dropped systemd's `●`-decorated FAILED rows; `grep -ic restic` on the wrapper missed the check script; `restic ls`'s header line made an absent path read as "blobs gone"; and a cooldown test that invoked before failing the unit. **Every one reported cleanly while looking at the wrong thing.** The rule is not "verify" — it is *verify, then ask what the verification could not have seen*. → `persistent-memory.d/2026-09-22-instrument-errors.md`
|
||||
- `[2026-09-22]` **Fleet alert bridge generalized** — `beszel-althing` → `althing-alert-bridge`, route registry (`/beszel` + `/kuma`), each with its own… → `persistent-memory.d/2026-09-22-fleet-alert-bridge-generalized.md`
|
||||
- `[2026-09-22]` **Uptime Kuma rebuilt from scratch on 2.5.5, moved esh-docker-vm → ana-docker.** — ⚠ `:latest` is a TRAP — it tracks 1.x, so an Aug-2026 pull gave a Dec-2024 build. → `persistent-memory.d/2026-09-22-uptime-kuma-rebuilt-from-scratch-on-2-5-5-moved-esh-docker.md`
|
||||
- `[2026-09-22]` **Beszel and Uptime Kuma are DISJOINT, not redundant** — Beszel's alerts bind to a *system* with a threshold; there is no URL column, so it is structurally incapable… → `persistent-memory.d/2026-09-22-beszel-and-uptime-kuma-are-disjoint-not-redundant.md`
|
||||
- `[2026-09-22]` **Failed-START alarms on 23 nh3-dev units** — (`services/althing-notify-failure/`). → `persistent-memory.d/2026-09-22-failed-start-alarms-on-23-nh3-dev-units.md`
|
||||
- `[2026-09-22]` **Backup coverage is a property of the SYSTEM, never one job's scope** — establish it by querying the repo for the path in a real snapshot, never by reading a job's `SRC=`. → `persistent-memory.d/2026-09-22-backup-coverage-is-a-property-of-the-system-never-one-job-s.md`
|
||||
- `[2026-09-22]` **restic checks now assert CONTENT and are DISCOVERED not enumerated** — conjunctive (recent AND present AND restores non-zero bytes); repos discovered from each NAS, hand list… → `persistent-memory.d/2026-09-22-restic-checks-now-assert-content-and-are-discovered-not.md`
|
||||
- `[2026-09-22]` **irv-ml1: Irvine is a TENANCY behind a Fortinet PFI does not control** — its TLS inspection breaks Tailscale relay/control intermittently (41 cert warnings/week). → `persistent-memory.d/2026-09-22-irv-ml1-irvine-is-a-tenancy-behind-a-fortinet-pfi-does-not.md`
|
||||
|
||||
- `[2026-09-21]` ⭐⭐⭐ **lv-mccarthy GATED and NOT SHIPPED — voice passes, memorisation is the cleanest in the line, and the length discipline is gone.** 720 generations, 3 arms. Voice +0.152 at **2.9×** floor for ckpt450 (ckpt900 +0.172 at only 1.2×, its spread one outlier seed), and **~3/4 of the gain survives stripping every punctuation mark**, so it is not the cheap win. ⭐ Memorisation: **ckpt450 at 0.12 against the author's own held-out 0.12 — identical, longest match 11 words against the author's coincidental 12**, all 96 matches READ and every one stock grammar (`he looked at the wolf and he looked at him`); the name-shaped hits are the RENAMED inventions. Axis C is the blocker: **20% / 28% of generations overshoot the 90–140 band against base's 1%**, worst case a degenerate loop at 279 words. ⚠ My own prereg's axis C transcribed `score_beats.py`'s **v1** criteria including "in-band up on base", which the operator RETIRED 2026-09-15 because base maxes it — under the v2 (ran-on only) ckpt450 passes by **0.01 against a 0.200 floor**, but that reading was found AFTER the numbers and was not used. Fix the prereg prospectively. → `persistent-memory.d/2026-09-21-lv-mccarthy-gate.md`
|
||||
|
||||
- `[2026-09-21]` ⭐⭐⭐ **The ops log shipped — and then failed FOUR ways in its first hours, every one recording something unfindable.** Claim dropped by a sub-tool, hook behind graphify's eight `exit 0`s, no handle in the env, ssh-target written as a hostname. The general shape is **configured ≠ effective**; twelve instruments reported confidently and wrongly across three days, five of them mine. → `persistent-memory.d/2026-09-21-ops-log-and-the-instruments-that-lied.md`
|
||||
|
||||
- `[2026-09-21]` ⭐⭐ **The Booth gained blur + a closed keep round trip, after shipping TWO controls that did nothing** — a reveal handler Jinja discarded for sitting after `{% endblock %}`, and a `×` a sibling form covered by 30×22 px. Both operator-found, both from reading templates instead of rendering them. `scripts/layout-probe.py` took four iterations to become trustworthy. ⚠ Blur is COSMETIC and a test asserts the 200 on purpose. → `persistent-memory.d/2026-09-21-booth-two-dead-controls.md`
|
||||
|
||||
- `[2026-09-17]` **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the…** → `persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion.md`
|
||||
- `[2026-09-17]` **The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev** — `dev_launch.py` has zero occurrences of "route", no hook declares one, and every live route was hand-declared… → `persistent-memory.d/2026-09-17-the-althing-route-declaring-sessionstart-hook-is-documented.md`
|
||||
|
||||
- `[2026-09-17]` **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the…** → `persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion-2.md`
|
||||
|
||||
- `[2026-09-15]` **breeze-tts stays on irv-ml1; the TTS-stack move to fv-ml1 is PARKED (park id 75, `move-the-tts-stack-breeze-tts-bragi-tts-gateway`), triggered on evacuating embed/rerank/reward.** ⚠ Trigger as stated says "gpu0" but those three are on **GPU 1** (~0.16 util, ~15.7 GB; GPU 1 is the tight card at 0.975 / 4,336 MiB free) — confirm which he meant before executing. All three services move together because only `breeze-tts` is GPU-resident (~10.3 GiB, **growing**) while `bragi` and `tts-gateway` are CPU proxies, and co-location is what avoids a cross-site hop per TTS call. **breeze-tts sizing — original recommendation NOT to move it.** ~**10.3 GiB** measured under load at 53 min uptime, **up from 9.2 GiB** shortly after warm-up (it grows; n=2, plateau unmeasured) — so GPU 0's 11,982 MiB free is a **1.7 GB margin and shrinking**, on the live chat serving path. ⚠ Two measurement traps: it reports **nothing at idle on the wrong card** (`BREEZE_GPU_DEVICES=0` = the **3090**, not the A6000), and an early reading understates it. ⭐ The real objection is **topology**: `tts-gateway` is on irv-ml1 and reaches it same-box, so moving breeze alone adds a cross-site hop to every TTS call against a 478 ms first-sample budget. GPU 3 would fit it but spends the reserve. → `persistent-memory.d/2026-09-15-breeze-placement-sizing.md`
|
||||
|
||||
- `[2026-09-13]` **STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint.** → `persistent-memory.d/2026-09-13-standing-policy-operator-cap-gpu-power.md`
|
||||
|
||||
- `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** → `persistent-memory.d/2026-08-25-fused-moe-kernel-path-deferred-tracked-at-park-fused-moe.md`
|
||||
|
||||
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md`
|
||||
|
||||
_197 older entries archived to archival-memory.md._
|
||||
_214 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-10-06]` **Restarting a user-facing service after an idle check that only PRINTS** — `nvidia-smi … | wc -l` said 1 and the same command restarted Scriberr anyway, killing Prime's transcript run. Make the check gate (`&&` / exit). auto-memory `feedback_idle_check_must_gate_the_restart`.
|
||||
- `[2026-10-06]` **Aging PBS snapshots by directory mtime** — verify jobs and notes edits rewrite `index.json.blob`, so a 10-04 snapshot read as 10-06. Age by the snapshot NAME. auto-memory `reference_pbs_snapshot_mtime_moves_on_verify`.
|
||||
- `[2026-10-06]` **`deploy-stack.sh` on a stack whose host dir holds rollback copies** (albok-service `compose.yaml.pre-*`) — its rsync `--delete` removes them. Upload the single changed file with elway instead.
|
||||
- `[2026-10-05]` **Sizing a vLLM seat's KV from `--gpu-memory-utilization` on a shared card** — co-booting neighbours corrupt the profile (non-torch −0.56 GiB), so the KV is a lottery per boot. Pin `--kv-cache-memory`. auto-memory `reference_vllm_util_kv_coboot_race`.
|
||||
- `[2026-10-05]` **Reading GPU headroom just after a restart** — 10-01's "Free ~1.05 GB" was really a ~60 MiB plateau 2 days later. Size against Beszel history. auto-memory `feedback_vllm_headroom_at_plateau_not_boot`.
|
||||
- `[2026-10-04]` **`apt-get` run in-band through `qm guest exec`** — qemu-guest-agent was in the upgrade set, restarted itself, and systemd killed apt mid-dpkg on BOTH PBS VMs. Run apt in a `systemd-run` unit. (auto-memory `feedback_no_inband_apt_through_guest_agent`)
|
||||
- `[2026-10-04]` **Pulling files with `ssh host 'sudo cat …'`** — sudo ≥ 1.9.14 truncated the piped stream at exit (Debian 13 hosts), and `log_output` re-recorded the whole file as a new session. Read without sudo (group-readable file).
|
||||
- `[2026-10-01]` **Size + mtime scans to pick reclaimable "staging" on a model store** → `persistent-memory.d/2026-10-01-size-mtime-scans-to-pick-reclaimable-staging-on-a.md`
|
||||
|
||||
Reference in New Issue
Block a user