diff --git a/persistent-memory.md b/persistent-memory.md index a9e9b19..34e599a 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -125,6 +125,7 @@ _As of 2026-10-05 ~2035 PT. The newest subsection is first; older subsections ca - **Worldtree memory gate: the 20261004T143011Z FAIL did not recur.** That run filed the forbidden `u-negative-ephemeral` fact once (run 1, negative 1/2), and an esh-pve reboot overlapped its run 1. The next two batches PASSED: 20261005T143010Z and 20261006T143010Z (user filing 11/12 = 0.92 in both runs, floor 0.80, negative 2/2). Read from `Worldtree/docs/eval/memory_acceptance_runs//summary.md` on 10-06 after infra-hermes' 0747 PASS report. That is 2 passes after 1 failure on a stochastic gate: a single filing slip is not ruled out, so keep reading the daily verdict rather than calling it fixed. - **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm. - **fv-ml1 GPU 0/1 REBALANCED 2026-10-05 2226–2249 (Prime: "accept all that" + "context up to published maximums if at all possible"; `playbooks/fv-ml1-rebalance-20261005.yaml`, elway all green, gateway smoke tests pass).** It follows the 2131 KV pins (meromero 22 GiB, coder pinned, gen-small 5.5 GiB). Why: on a shared card util-sized KV is a co-boot lottery, and GPU 0 creeps to ~60 MiB free after a boot. Rule: KV ≥ 1.25× the seat's max context, rounded up to 0.5 GiB. Two exceptions: meromero and gen-small keep Prime's numbers, and voices gets 1.07× (short prose traffic). Usage basis: 7 days of LiteLLM logs (gen-small 22,396 calls; sec 400; char-rp-fast 55; char-rp 14) plus the engines' own KV logs. **Layout:** GPU 0 = cyberprev, erp-seat, voices, coder, parakeet-nemo (4,036 MiB free right after; ~2.3 GB spare at parakeet's cap; cyberprev may still creep). GPU 1 = gen-small, meromero-rp, intern-decision (10,864 free; ~3.3 GB spare at the 32k peak once gen-small settles). Contexts raised: coder 16,384 → **32,768** (card max), voices 8,192 → **40,960** (native max). YaRN extensions were NOT enabled, and **Prime confirmed 2026-10-06: leave YaRN off for gen-small** (gen-small/cyberprev to ~1M, voices to 131k). They don't fit, and Qwen's own cards warn static YaRN hurts short prompts. intern-decision stays at 32,768 (card default 8,192; base config 262k; ~0.21 MiB/token of VRAM). **gen-small default output cap 32,768** (`--override-generation-config`; an explicit max_tokens still wins, verified in vLLM `get_max_tokens`). It answers 9 runaway ~260k-token replies in 7 days (albok-nemi ×6, all-agents-local ×3, all from nh3-dev); albok-dev told 2250 and closed it 10-06 0000. Nemi now sends max_tokens 8192 (albok-nemi 0.1.2, `tagging.max_tokens`), and its nh3-dev harness configs moved off the shared all-agents-local key onto vault `albok/litellm-key-nemi` + `albok/litellm-key`. The loops came from gen-small at temperature 0 on some memory entries; a capped reply degrades to Nemi's rule-based tagger. ⚠ **Verify both cards' plateaus ~10-07/08** from Beszel (`stats.g..mu` × 1.024). .env backups `*.env.bak-20261005-rebal`. Earlier the same night: KV pins 2131–2139 (`playbooks/fv-ml1-kv-pin-20261005.yaml`; diagnosis in commit 269c80f). Park 75 dropped by Prime 2115. +- **gen-large contention (2026-10-06 1432):** the `jekyll` Hermes seat (nh3-dev, shared all-agents-local key) was sending 165–183k-token prompts every few seconds while Prime ran ~140k from his MacBook. flash-next's KV (344k tokens) overflowed: 539 → 819 preemptions, decode 2–16 tok/s, calls of 221–244 s. **Prime moved jekyll to `sec-reasoning` (cyberprev, GPU 0) at ~1435.** At 1439 gen-large was back to 90–143 tok/s with 1 running; sec had KV at 34% and 0 preemptions. ⚠ Watch: sec's KV is 337k tokens, so jekyll plus a long sec prompt (up to 192k) can overflow it too. jekyll's context grows ~1k/call toward the 262k ceiling. Heavier long-context traffic may grow cyberprev's footprint and eat GPU 0's ~2.3 GB spare; check at the 10-07/08 plateau review. - **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in. ### Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01)