memory: fv-ml1 tight-card diagnosis (meromero util-sized KV co-boot race on GPU 1; GPU 0 plateau ~0 free); park 75 dropped

2026-10-05 21:22:41 -07:00
parent 1273163173
commit 269c80f3ca
1 changed files with 1 additions and 0 deletions
@@ -124,6 +124,7 @@ _As of 2026-10-05 ~2035 PT. The newest subsection is first; older subsections ca
 - **Scheduled / to verify:** 10-05 0815 + 0900 infra-hermes jobs (sfsrv-ana `pre-pbs4` snapshot delete after its 04:00 backup; drop `pve/root-pre-pve9` on pfi-pve + esh-pve) — outcome not checked in this session, read `scripts/ops-log tail`; esh-nas-pve `nvme/ROOT@pre-pve9` destroy ~10-10 (ours); 10-11 0540 first weekly sudo-io archiver run (check it), 10-11 0900 infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet.
 - **Worldtree memory gate 20261004T143011Z FAILED** (one run filed the forbidden `u-negative-ephemeral` fact; an esh-pve reboot overlapped run 1). Whether the 2026-10-05 0730 run repeated it was not checked here; it is a regression check with worldtree-dev, not a gate.
 - **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm.
+- **fv-ml1 tight cards (dug in 2026-10-05 ~2120, read-only; fixes await Prime's go).** GPU 1: a +2.4–2.6 GB STEP at the 0234 Docker-move restart (Beszel 2 h windows: flat 88,566–88,779 for 4 days → flat 91,082 for 16 h; Beszel `mu` = MiB / 1.024). Cause: `vllm-meromero-rp` is the only big GPU-1 seat sized by `--gpu-memory-utilization` (0.52) rather than `--kv-cache-memory`, and vLLM's profile is corrupted when neighbours co-boot on the card: non-torch read 8.11 (09-14) → 1.74 → **−0.56 GiB** (10-05 run 2), KV 20.37 → 26.74 → **29.04 GiB** (320,774 → 457,315 tokens). Free now 3,981 vs the 6,625 intern-decision's 32k budget assumed → a max-length call is ~2.4 GB short (no OOM logged; 293 calls on 10-05, all 200). `vllm-coder` (0.055, unpinned) also crashes once per co-boot on negative KV, then lands on 1.03 GiB every time. **Proposed fix:** pin meromero `--kv-cache-memory` 24 GiB (≈377,950 tokens, 1.44×; Free → ~9.1 GB, ~2.7 GB spare at intern-decision's peak) and coder at 1.03 GiB; profiling is skipped, so no trial boot. GPU 0: no step; it CREEPS after every boot and plateaus near ZERO free (Beszel: ~60 MiB free 10-03→10-05, before the restart). The 10-01 "Free ~1.05 GB at rest" was a just-after-restart reading, the trap `stacks/meromero-charrp/.env.example` already warns about. Now 495 free and falling; gen-small 37,302 (10-01: 36,116), cyberprev 46,504 (09-14 plateau 47,104). Parakeet: 16 short jobs 10-02/03 OK; a long file needs ~920 MiB above its 2,108 rest. Options for Prime: move parakeet-nemo to GPU 3 (recommended), trim gen-small KV (he declined on 10-01), or leave it. **Park 75 (TTS stack → fv-ml1) DROPPED** on Prime's call 2026-10-05 2115 (comment 63 has the reasoning).
 - **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in.

 ### Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01)