vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: mirror retention set; YaRN stays off (Prime)

Browse Source
This commit is contained in:
vh
2026-10-06 13:20:29 -07:00
parent 5896a7479c
commit eaa32d6249
1 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+2 -2
View File
@@ -119,12 +119,12 @@ _As of 2026-10-05 ~2035 PT. The newest subsection is first; older subsections ca
- **Nothing is mid-action.** No claims held (`scripts/ops-log claims`); eshpfi pushed through `1210b4a`.
- **New SVOS on nh3-svos (VM 111, nh3-pve-2, 10.100.50.43): DEPLOYED and ACCEPTED by svos-dev 2026-10-05 ~2024** (thread `01M47ET6XH0M7JZSH98Z19SKR4` closed). Separate from the High Seat (`svos.service` :8770 on nh3-dev) and Miranda, which are NOT migrated. Front door `https://svos.phasefinal.com`, mesh only, Basic auth login `Vuong` (Prime has the password; vault `nh3-svos/basic-auth-vuong`). ⚠ **After every SVOS release** (`/opt/svos/current` moves): `sudo /usr/local/sbin/svos-front-inventory`, and keep the previous release dir while it is "previous" (svos-dev's runbook carries both). Only open item is Prime's: whether his iPhone home-screen app keeps the login. → `servers/nh3-svos/README.md`
- **Backup check (2026-10-06): the 0800 run was clean apart from one false STALE, now FIXED.** nh3-pve-2 coverage read clean, svos drill B was ACKNOWLEDGED, nothing was DEGRADED. The false STALE was `pbs/ana-pve/vm/110`, pfi-ana-webhost, destroyed 10-04 with `qm destroy --purge`. Three fixes landed in `scripts/check-backup-freshness.sh`. infra-hermes, `0a348e5`: destroyed guests are now reported as policy residue (GUEST DESTROYED), and an ssh in the read loop now uses `-n`, because without it the whole PBS layer silently vanished. infra-ops, `d920cf3`: snapshot age now comes from the snapshot NAME, not the directory mtime, which PBS bumps at every verify and every 30-day re-verify. Verified exit 0; positive control at 5h flags 10. Residue PRUNED on Prime's word (2026-10-06 1307): on pbs-ana, ana-pve/vm/110 (8 snapshots) and esh-pve/vm/102 (7, including the protected final 09-10 image, unprotected first; Prime's 09-19 ruling said that VM had no needed state); on the pbs-nh3 mirror, 162 + 10. The checker now reads 0 GUEST DESTROYED, exit 0. ⚠ **Open for Prime: the mirror keeps everything forever.** Its prune job has no keep-* rules and its sync has remove-vanished false: 3,932 snapshots against 266 on the primary, 5.49 TiB of chunks against 0.93 TiB, on an nh3-nas volume at 79%.
- **Backup check (2026-10-06): the 0800 run was clean apart from one false STALE, now FIXED.** nh3-pve-2 coverage read clean, svos drill B was ACKNOWLEDGED, nothing was DEGRADED. The false STALE was `pbs/ana-pve/vm/110`, pfi-ana-webhost, destroyed 10-04 with `qm destroy --purge`. Three fixes landed in `scripts/check-backup-freshness.sh`. infra-hermes, `0a348e5`: destroyed guests are now reported as policy residue (GUEST DESTROYED), and an ssh in the read loop now uses `-n`, because without it the whole PBS layer silently vanished. infra-ops, `d920cf3`: snapshot age now comes from the snapshot NAME, not the directory mtime, which PBS bumps at every verify and every 30-day re-verify. Verified exit 0; positive control at 5h flags 10. Residue PRUNED on Prime's word (2026-10-06 1307): on pbs-ana, ana-pve/vm/110 (8 snapshots) and esh-pve/vm/102 (7, including the protected final 09-10 image, unprotected first; Prime's 09-19 ruling said that VM had no needed state); on the pbs-nh3 mirror, 162 + 10. The checker now reads 0 GUEST DESTROYED, exit 0. **Mirror retention SET (Prime "match history", 2026-10-06 1310):** pbs-nh3 prune job `default-backups-mirror-1b7d4123-` now keeps last 3 / daily 14 / weekly 8 / monthly 12 / yearly 5, the same as pbs-ana. It was keep-all: 3,932 snapshots against 266, 5.49 TiB against 0.93 TiB, nh3-nas at 79%. Dry-run: keep 738, remove 3,194. First run at 00:00 on 10-07, then daily GC. remove-vanished stays false (deliberate). ⚠ **Check on 10-07** that the prune ran and GC freed the space. Old cfg: `/root/prune.cfg.bak-20261006` on pbs-nh3.
- **Waiting on Prime (updated 2026-10-05 2035):** push Booth commit `ce095c1` (touch-layout fix, `~/development/booth`, browser test + mutation); a trusted HTTPS name for ESH Homepage (https://10.0.50.45 serves Traefik's default cert; the ESH ACME resolver, Route53, is dead); nh3-dev reboot (runs kernel 6.1.0-49, 6.1.0-53 installed; its engine is ALREADY 29.8.2 since 10-05 1413, so the old "reboot + engine bump" wording was stale, checked 2035); Docker engine bumps held on nh3-docker (carries the post office) and esh-docker-vm (NFS D-state history); fv-ml1's held ZFS 2.3.9, nvidia-container-toolkit 1.20, and a new kernel only together with a newer NVIDIA driver (6.12.111 broke the 580.65 DKMS build). ⚠ **irv-ml1: DO NOT PATCH (Prime).** ana-nas is out of scope.
- **Scheduled / to verify:** 10-05 0815 + 0900 infra-hermes jobs (sfsrv-ana `pre-pbs4` snapshot delete after its 04:00 backup; drop `pve/root-pre-pve9` on pfi-pve + esh-pve) — outcome not checked in this session, read `scripts/ops-log tail`; esh-nas-pve `nvme/ROOT@pre-pve9` destroy ~10-10 (ours); 10-11 0540 first weekly sudo-io archiver run (check it), 10-11 0900 infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet.
- **Worldtree memory gate: the 20261004T143011Z FAIL did not recur.** That run filed the forbidden `u-negative-ephemeral` fact once (run 1, negative 1/2), and an esh-pve reboot overlapped its run 1. The next two batches PASSED: 20261005T143010Z and 20261006T143010Z (user filing 11/12 = 0.92 in both runs, floor 0.80, negative 2/2). Read from `Worldtree/docs/eval/memory_acceptance_runs/<stamp>/summary.md` on 10-06 after infra-hermes' 0747 PASS report. That is 2 passes after 1 failure on a stochastic gate: a single filing slip is not ruled out, so keep reading the daily verdict rather than calling it fixed.
- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm.
- **fv-ml1 GPU 0/1 REBALANCED 2026-10-05 2226–2249 (Prime: "accept all that" + "context up to published maximums if at all possible"; `playbooks/fv-ml1-rebalance-20261005.yaml`, elway all green, gateway smoke tests pass).** It follows the 2131 KV pins (meromero 22 GiB, coder pinned, gen-small 5.5 GiB). Why: on a shared card util-sized KV is a co-boot lottery, and GPU 0 creeps to ~60 MiB free after a boot. Rule: KV ≥ 1.25× the seat's max context, rounded up to 0.5 GiB. Two exceptions: meromero and gen-small keep Prime's numbers, and voices gets 1.07× (short prose traffic). Usage basis: 7 days of LiteLLM logs (gen-small 22,396 calls; sec 400; char-rp-fast 55; char-rp 14) plus the engines' own KV logs. **Layout:** GPU 0 = cyberprev, erp-seat, voices, coder, parakeet-nemo (4,036 MiB free right after; ~2.3 GB spare at parakeet's cap; cyberprev may still creep). GPU 1 = gen-small, meromero-rp, intern-decision (10,864 free; ~3.3 GB spare at the 32k peak once gen-small settles). Contexts raised: coder 16,384 → **32,768** (card max), voices 8,192 → **40,960** (native max). YaRN extensions were NOT enabled (gen-small/cyberprev to ~1M, voices to 131k). They don't fit, and Qwen's own cards warn static YaRN hurts short prompts. intern-decision stays at 32,768 (card default 8,192; base config 262k; ~0.21 MiB/token of VRAM). **gen-small default output cap 32,768** (`--override-generation-config`; an explicit max_tokens still wins, verified in vLLM `get_max_tokens`). It answers 9 runaway ~260k-token replies in 7 days (albok-nemi ×6, all-agents-local ×3, all from nh3-dev); albok-dev told 2250 and closed it 10-06 0000. Nemi now sends max_tokens 8192 (albok-nemi 0.1.2, `tagging.max_tokens`), and its nh3-dev harness configs moved off the shared all-agents-local key onto vault `albok/litellm-key-nemi` + `albok/litellm-key`. The loops came from gen-small at temperature 0 on some memory entries; a capped reply degrades to Nemi's rule-based tagger. ⚠ **Verify both cards' plateaus ~10-07/08** from Beszel (`stats.g.<n>.mu` × 1.024). .env backups `*.env.bak-20261005-rebal`. Earlier the same night: KV pins 2131–2139 (`playbooks/fv-ml1-kv-pin-20261005.yaml`; diagnosis in commit 269c80f). Park 75 dropped by Prime 2115.
- **fv-ml1 GPU 0/1 REBALANCED 2026-10-05 2226–2249 (Prime: "accept all that" + "context up to published maximums if at all possible"; `playbooks/fv-ml1-rebalance-20261005.yaml`, elway all green, gateway smoke tests pass).** It follows the 2131 KV pins (meromero 22 GiB, coder pinned, gen-small 5.5 GiB). Why: on a shared card util-sized KV is a co-boot lottery, and GPU 0 creeps to ~60 MiB free after a boot. Rule: KV ≥ 1.25× the seat's max context, rounded up to 0.5 GiB. Two exceptions: meromero and gen-small keep Prime's numbers, and voices gets 1.07× (short prose traffic). Usage basis: 7 days of LiteLLM logs (gen-small 22,396 calls; sec 400; char-rp-fast 55; char-rp 14) plus the engines' own KV logs. **Layout:** GPU 0 = cyberprev, erp-seat, voices, coder, parakeet-nemo (4,036 MiB free right after; ~2.3 GB spare at parakeet's cap; cyberprev may still creep). GPU 1 = gen-small, meromero-rp, intern-decision (10,864 free; ~3.3 GB spare at the 32k peak once gen-small settles). Contexts raised: coder 16,384 → **32,768** (card max), voices 8,192 → **40,960** (native max). YaRN extensions were NOT enabled, and **Prime confirmed 2026-10-06: leave YaRN off for gen-small** (gen-small/cyberprev to ~1M, voices to 131k). They don't fit, and Qwen's own cards warn static YaRN hurts short prompts. intern-decision stays at 32,768 (card default 8,192; base config 262k; ~0.21 MiB/token of VRAM). **gen-small default output cap 32,768** (`--override-generation-config`; an explicit max_tokens still wins, verified in vLLM `get_max_tokens`). It answers 9 runaway ~260k-token replies in 7 days (albok-nemi ×6, all-agents-local ×3, all from nh3-dev); albok-dev told 2250 and closed it 10-06 0000. Nemi now sends max_tokens 8192 (albok-nemi 0.1.2, `tagging.max_tokens`), and its nh3-dev harness configs moved off the shared all-agents-local key onto vault `albok/litellm-key-nemi` + `albok/litellm-key`. The loops came from gen-small at temperature 0 on some memory entries; a capped reply degrades to Nemi's rule-based tagger. ⚠ **Verify both cards' plateaus ~10-07/08** from Beszel (`stats.g.<n>.mu` × 1.024). .env backups `*.env.bak-20261005-rebal`. Earlier the same night: KV pins 2131–2139 (`playbooks/fv-ml1-kv-pin-20261005.yaml`; diagnosis in commit 269c80f). Park 75 dropped by Prime 2115.
- **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in.
### Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01)