fix(fv-ml1): pin vLLM KV in bytes on meromero (22 GiB) and coder; gen-small KV 8 -> 5.5 GiB

meromero-rp sized its KV from --gpu-memory-utilization, and vLLM's profile
counts co-booting neighbours as its own non-torch memory. Three boots gave
20.37 / 26.74 / 29.04 GiB of KV, and the 2026-10-05 Docker-move restart cut
GPU 1 free memory to 3,981 MiB, below intern-decision's 32k-token budget.
Coder read negative KV on every co-boot. Both are now pinned, so util only
gates the boot check (meromero util 0.52 -> 0.47, just above its footprint).

GPU 0 plateaus at ~60 MiB free 1-2 days after a boot (Beszel), so
gen-small's KV drops to 5.5 GiB (461,215 tokens, 1.76x) and parakeet-nemo
stays on GPU 0 with room for its 3,840 MiB cap. GPU 3 stays clear.

Applied with playbooks/fv-ml1-kv-pin-20261005.yaml: GPU 1 free 3,981 ->
11,801 MiB, GPU 0 495 -> 5,039 MiB right after boot; all nine seats healthy.

persistent-memory.md

@@ -124,7 +124,7 @@ _As of 2026-10-05 ~2035 PT. The newest subsection is first; older subsections ca
 - **Scheduled / to verify:** 10-05 0815 + 0900 infra-hermes jobs (sfsrv-ana `pre-pbs4` snapshot delete after its 04:00 backup; drop `pve/root-pre-pve9` on pfi-pve + esh-pve) — outcome not checked in this session, read `scripts/ops-log tail`; esh-nas-pve `nvme/ROOT@pre-pve9` destroy ~10-10 (ours); 10-11 0540 first weekly sudo-io archiver run (check it), 10-11 0900 infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet.
 - **Worldtree memory gate 20261004T143011Z FAILED** (one run filed the forbidden `u-negative-ephemeral` fact; an esh-pve reboot overlapped run 1). Whether the 2026-10-05 0730 run repeated it was not checked here; it is a regression check with worldtree-dev, not a gate.
 - **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm.
-- **fv-ml1 tight cards (dug in 2026-10-05 ~2120, read-only; fixes await Prime's go).** GPU 1: a +2.4–2.6 GB STEP at the 0234 Docker-move restart (Beszel 2 h windows: flat 88,566–88,779 for 4 days → flat 91,082 for 16 h; Beszel `mu` = MiB / 1.024). Cause: `vllm-meromero-rp` is the only big GPU-1 seat sized by `--gpu-memory-utilization` (0.52) rather than `--kv-cache-memory`, and vLLM's profile is corrupted when neighbours co-boot on the card: non-torch read 8.11 (09-14) → 1.74 → **−0.56 GiB** (10-05 run 2), KV 20.37 → 26.74 → **29.04 GiB** (320,774 → 457,315 tokens). Free now 3,981 vs the 6,625 intern-decision's 32k budget assumed → a max-length call is ~2.4 GB short (no OOM logged; 293 calls on 10-05, all 200). `vllm-coder` (0.055, unpinned) also crashes once per co-boot on negative KV, then lands on 1.03 GiB every time. **Proposed fix:** pin meromero `--kv-cache-memory` 24 GiB (≈377,950 tokens, 1.44×; Free → ~9.1 GB, ~2.7 GB spare at intern-decision's peak) and coder at 1.03 GiB; profiling is skipped, so no trial boot. GPU 0: no step; it CREEPS after every boot and plateaus near ZERO free (Beszel: ~60 MiB free 10-03→10-05, before the restart). The 10-01 "Free ~1.05 GB at rest" was a just-after-restart reading, the trap `stacks/meromero-charrp/.env.example` already warns about. Now 495 free and falling; gen-small 37,302 (10-01: 36,116), cyberprev 46,504 (09-14 plateau 47,104). Parakeet: 16 short jobs 10-02/03 OK; a long file needs ~920 MiB above its 2,108 rest. Options for Prime: move parakeet-nemo to GPU 3 (recommended), trim gen-small KV (he declined on 10-01), or leave it. **Park 75 (TTS stack → fv-ml1) DROPPED** on Prime's call 2026-10-05 2115 (comment 63 has the reasoning).
+- **fv-ml1 KV pins APPLIED 2026-10-05 2131–2139 (Prime's call; `playbooks/fv-ml1-kv-pin-20261005.yaml`, elway all green).** Diagnosis: on GPU 1, `vllm-meromero-rp` sized KV from util 0.52, and vLLM's profile is corrupted by co-booting neighbours (non-torch 8.11 → 1.74 → −0.56 GiB; KV 20.37 → 26.74 → 29.04 GiB). The 0234 Docker-move restart cost GPU 1 ~2.5 GB (Beszel step), so intern-decision's 32k calls were ~2.4 GB short. GPU 0 has no step: it CREEPS for 1–2 days after a boot and plateaued at ~60 MiB free (Beszel 10-03→10-05). The 10-01 "~1.05 GB free" was a just-after-restart read. Changes: meromero KV pinned **22 GiB** (346,436 tokens, 1.32×), util 0.52 → 0.47 (boot check only); coder pinned **1,104,674,816 B** (77,056 tokens, unchanged); gen-small KV **8 → 5.5 GiB** (461,215 tokens, 1.76×) so parakeet-nemo STAYS on GPU 0. **GPU 3 stays clear for ad-hoc work (Prime).** Result: GPU 1 Free 3,981 → **11,801** MiB (~5.4 GB spare at intern-decision's 32k peak); GPU 0 Free 495 → 5,039 right after boot, projected plateau ~2.6 GB, which covers parakeet's 3,840 MiB cap with ~0.9 GB to spare. ⚠ **Verify the GPU 0 plateau ~10-07/08** from Beszel (`stats.g.0.mu` × 1.024). Downtime: char-rp ~2 min, coder ~1 min, gen-small ~4 min (57 s compile miss). .env backups `*.env.bak-20261005-kvpin`. **Park 75 (TTS stack → fv-ml1) DROPPED** by Prime 2026-10-05 2115 (comment 63).
 - **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in.
 
 ### Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01)
@@ -152,11 +152,12 @@ _As of 2026-10-05 ~2035 PT. The newest subsection is first; older subsections ca
 - **Legacy archive:** DESTROY it whole by **2026-10-30**, or at retirement-done, or on a subject-erasure request, whichever comes first. The runbook is in the detail file.
 - **TODO:** re-sweep both api logs after real traffic. After the b193 push, remove the retired config keys (list: detail file, "Post-b193 config retirements").
 
-### fv-ml1 GPU layout (as of 2026-10-01)
+### fv-ml1 GPU layout (as of 2026-10-05)
 
-- **GPU 0:** cyberprev (47.1 GB), gen-small (35.3 GB), voices (10.8 GB), parakeet-nemo (3.6 GB steady). Free 385 MiB, FULL.
+- **GPU 0:** cyberprev (KV 16 GiB pinned), gen-small (KV 5.5 GiB pinned since 2026-10-05), voices (KV 1.5 GiB), parakeet-nemo (rest 2,108 MiB, cap 3,840). Plateau target ~2.6 GB free (verify ~10-07/08).
-- **GPU 1:** vllm-coder, erp-seat, meromero-rp, plus intern-decision (cap 14.4 GiB, 32k tokens, peak 15,220 of a 15,437 MiB budget). FULL.
+- **GPU 1:** vllm-coder, erp-seat, meromero-rp (all KV byte-pinned since 2026-10-05), plus intern-decision (cap 14.4 GiB, 32k tokens, peak 15,220). Free 11,801 MiB at rest → ~5.4 GB spare at its peak.
-- **GPU 3:** the full-size-seat reserve (Flash-Next is parked). On-demand tenants: Blender, Scriberr (0 idle, ~5.5 GB per job), and MIA auto-rig (one-shot `scripts/mia-run`, peak 3.4 GB, added 2026-10-01 for dread-dev). When a full-size seat claims GPU 3, Scriberr steps aside to **irv-ml1's A6000**, not back to GPU 1.
+- **GPU 2:** `vllm-flash-next` alone (gen / gen-large and 6 other aliases, util 0.96, ~96.4 GB). Live since 2026-09-14; henge park 49 is stale.
+- **GPU 3:** clear for ad-hoc work (Prime, 2026-10-05); no always-resident tenant goes here. On-demand tenants: Blender, Scriberr (0 idle, ~5.5 GB per job), and MIA auto-rig (one-shot `scripts/mia-run`, peak 3.4 GB, added 2026-10-01 for dread-dev). When a full-size seat claims GPU 3, Scriberr steps aside to **irv-ml1's A6000**, not back to GPU 1.
 
 ### intern-decision (replaced SemIf on 2026-09-30)
 

playbooks/fv-ml1-kv-pin-20261005.yaml

@@ -0,0 +1,186 @@
+# Pin the KV cache in bytes on three fv-ml1 vLLM seats (2026-10-05, Prime's call).
+#
+#   GPU 1  vllm-meromero-rp  util-sized (0.52) -> pinned 22 GiB, util 0.47 (boot check only)
+#   GPU 1  vllm-coder        util-sized         -> pinned 1,104,674,816 B (77,056 tokens, unchanged)
+#   GPU 0  vllm-gen-small    pinned 8 GiB       -> pinned 5.5 GiB, so parakeet-nemo stays on GPU 0
+#
+# Why: on a shared card vLLM's memory profile counts co-booting neighbours as its own, so
+# util-sized KV is a lottery per boot (meromero 20.37 / 26.74 / 29.04 GiB), and the 0234
+# restart left intern-decision's 32k calls ~2.4 GB short. GPU 0 creeps to ~60 MiB free.
+# Detail: persistent-memory.md, in-flight 2026-10-05.
+#
+# One seat at a time; each is guarded by the boot-check arithmetic (Free + own resident
+# >= util x 97,249 MiB + 512) and must come back healthy with the expected KV line before
+# the next one is touched. elway halts on the first failure.
+#   scripts/elway infra-ops@10.251.50.54 --playbook playbooks/fv-ml1-kv-pin-20261005.yaml
+# Rollback per seat: restore its .env.bak-20261005-kvpin and the previous compose.yaml from
+# git, then `docker compose up -d --no-deps <service>` in the stack dir.
+vars:
+  mero: /opt/docker/compose/meromero-charrp
+  vllm: /opt/docker/compose/vllm
+  gens: /opt/docker/compose/gen-small-seat
+  mero_kv: "23622320128"
+  coder_kv: "1104674816"
+  gens_kv: "5905580032"
+
+steps:
+  # ---------------- GPU 1: meromero-rp ----------------
+  - name: meromero — back up .env once
+    sudo: true
+    shell: cp -p {{ mero }}/.env {{ mero }}/.env.bak-20261005-kvpin
+    creates: "{{ mero }}/.env.bak-20261005-kvpin"
+
+  - name: meromero — util 0.52 -> 0.47
+    sudo: true
+    shell: sed -i 's/^MEROMERO_GPU_MEM_UTIL=.*/MEROMERO_GPU_MEM_UTIL=0.47/' {{ mero }}/.env
+    when: "! grep -qx 'MEROMERO_GPU_MEM_UTIL=0.47' {{ mero }}/.env"
+
+  - name: meromero — add MEROMERO_KV_CACHE_MEMORY
+    sudo: true
+    shell: sed -i '$a MEROMERO_KV_CACHE_MEMORY={{ mero_kv }}' {{ mero }}/.env
+    when: "! grep -q '^MEROMERO_KV_CACHE_MEMORY=' {{ mero }}/.env"
+
+  - name: meromero — upload compose.yaml
+    sudo: true
+    upload:
+      src: stacks/meromero-charrp/compose.yaml
+      dest: "{{ mero }}/compose.yaml"
+      mode: "0644"
+      owner: "lkraven:lkraven"
+
+  - name: meromero — dry-parse, and the rendered args carry the pin
+    sudo: true
+    shell: cd {{ mero }} && docker compose config -q && docker compose config | grep -q '{{ mero_kv }}'
+    changed_when: "false"
+
+  - name: meromero — boot-check arithmetic (Free + own >= 0.47 x 97249 + 512)
+    shell: |
+      own=0; for p in $(docker top vllm-meromero-rp -eo pid | tail -n +2); do
+        m=$(nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader,nounits | awk -F', ' -v p="$p" '$1==p{print $2}')
+        own=$((own + ${m:-0})); done
+      free=$(nvidia-smi -i 1 --query-gpu=memory.free --format=csv,noheader,nounits)
+      need=$(awk 'BEGIN{printf "%d", 0.47*97249+512}')
+      echo "GPU1 free=$free own=$own need=$need"; [ $((free + own)) -ge "$need" ]
+    when: "! docker inspect vllm-meromero-rp | grep -q '{{ mero_kv }}'"
+    changed_when: "false"
+
+  - name: meromero — recreate the seat
+    sudo: true
+    shell: cd {{ mero }} && docker compose up -d --no-deps vllm-meromero-rp
+    when: "! docker inspect vllm-meromero-rp | grep -q '{{ mero_kv }}'"
+
+  - name: meromero — wait for healthy (up to 20 min)
+    shell: |
+      i=0; while [ $i -lt 120 ]; do
+        [ -n "$(docker ps -q --filter name=^vllm-meromero-rp$ --filter health=healthy)" ] && exit 0
+        i=$((i + 1)); sleep 10; done; exit 1
+    changed_when: "false"
+
+  - name: meromero — KV is the pinned 22 GiB
+    shell: docker logs vllm-meromero-rp 2>&1 | grep -E 'reserved 22.0 GiB memory for KV Cache|GPU KV cache size' | tail -2 && docker logs vllm-meromero-rp 2>&1 | grep -q 'reserved 22.0 GiB memory for KV Cache'
+    changed_when: "false"
+
+  # ---------------- GPU 1: coder ----------------
+  - name: coder — back up .env once
+    sudo: true
+    shell: cp -p {{ vllm }}/.env {{ vllm }}/.env.bak-20261005-kvpin
+    creates: "{{ vllm }}/.env.bak-20261005-kvpin"
+
+  - name: coder — add CODER_KV_CACHE_MEMORY
+    sudo: true
+    shell: sed -i '$a CODER_KV_CACHE_MEMORY={{ coder_kv }}' {{ vllm }}/.env
+    when: "! grep -q '^CODER_KV_CACHE_MEMORY=' {{ vllm }}/.env"
+
+  - name: coder — upload compose.yaml
+    sudo: true
+    upload:
+      src: stacks/vllm/compose.yaml
+      dest: "{{ vllm }}/compose.yaml"
+      mode: "0644"
+      owner: "root:docker"
+
+  - name: coder — dry-parse, and the rendered args carry the pin
+    sudo: true
+    shell: cd {{ vllm }} && docker compose config -q && docker compose config | grep -q '{{ coder_kv }}'
+    changed_when: "false"
+
+  - name: coder — recreate the seat
+    sudo: true
+    shell: cd {{ vllm }} && docker compose up -d --no-deps vllm-coder
+    when: "! docker inspect vllm-coder | grep -q '{{ coder_kv }}'"
+
+  - name: coder — wait for healthy (up to 10 min)
+    shell: |
+      i=0; while [ $i -lt 60 ]; do
+        [ -n "$(docker ps -q --filter name=^vllm-coder$ --filter health=healthy)" ] && exit 0
+        i=$((i + 1)); sleep 10; done; exit 1
+    changed_when: "false"
+
+  - name: coder — KV is the pinned 77,056 tokens
+    shell: |
+      docker logs vllm-coder 2>&1 | grep -E 'GPU KV cache size' | tail -1
+      docker logs vllm-coder 2>&1 | grep -q 'GPU KV cache size: 77,056 tokens'
+    changed_when: "false"
+
+  # ---------------- GPU 0: gen-small ----------------
+  - name: gen-small — back up .env once
+    sudo: true
+    shell: cp -p {{ gens }}/.env {{ gens }}/.env.bak-20261005-kvpin
+    creates: "{{ gens }}/.env.bak-20261005-kvpin"
+
+  - name: gen-small — KV 8 GiB -> 5.5 GiB
+    sudo: true
+    shell: sed -i 's/^GEN_SMALL_KV_CACHE_MEMORY=.*/GEN_SMALL_KV_CACHE_MEMORY={{ gens_kv }}/' {{ gens }}/.env
+    when: "! sudo -n grep -qx 'GEN_SMALL_KV_CACHE_MEMORY={{ gens_kv }}' {{ gens }}/.env"
+
+  - name: gen-small — upload compose.yaml (compose default follows the .env)
+    sudo: true
+    upload:
+      src: stacks/gen-small-seat/compose.yaml
+      dest: "{{ gens }}/compose.yaml"
+      mode: "0644"
+      owner: "root:root"
+
+  - name: gen-small — dry-parse, the rendered args carry the pin, util is 0.33
+    sudo: true
+    shell: cd {{ gens }} && docker compose config -q && docker compose config | grep -q '{{ gens_kv }}' && grep -qx 'GEN_SMALL_GPU_MEM_UTIL=0.33' .env
+    changed_when: "false"
+
+  - name: gen-small — boot-check arithmetic (Free + own >= 0.33 x 97249 + 512)
+    shell: |
+      own=0; for p in $(docker top vllm-gen-small -eo pid | tail -n +2); do
+        m=$(nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader,nounits | awk -F', ' -v p="$p" '$1==p{print $2}')
+        own=$((own + ${m:-0})); done
+      free=$(nvidia-smi -i 0 --query-gpu=memory.free --format=csv,noheader,nounits)
+      need=$(awk 'BEGIN{printf "%d", 0.33*97249+512}')
+      echo "GPU0 free=$free own=$own need=$need"; [ $((free + own)) -ge "$need" ]
+    when: "! docker inspect vllm-gen-small | grep -q '{{ gens_kv }}'"
+    changed_when: "false"
+
+  - name: gen-small — recreate the seat
+    sudo: true
+    shell: cd {{ gens }} && docker compose up -d --no-deps vllm-gen-small
+    when: "! docker inspect vllm-gen-small | grep -q '{{ gens_kv }}'"
+
+  - name: gen-small — wait for healthy (up to 20 min)
+    shell: |
+      i=0; while [ $i -lt 120 ]; do
+        [ -n "$(docker ps -q --filter name=^vllm-gen-small$ --filter health=healthy)" ] && exit 0
+        i=$((i + 1)); sleep 10; done; exit 1
+    changed_when: "false"
+
+  - name: gen-small — KV is the pinned 5.5 GiB
+    shell: docker logs vllm-gen-small 2>&1 | grep -E 'reserved 5.5 GiB memory for KV Cache|GPU KV cache size' | tail -2 && docker logs vllm-gen-small 2>&1 | grep -q 'reserved 5.5 GiB memory for KV Cache'
+    changed_when: "false"
+
+verify:
+  - name: per-GPU free memory (GPU 1 must clear intern-decision's 32k need of ~6,400 MiB + margin)
+    shell: nvidia-smi --query-gpu=index,memory.used,memory.free --format=csv,noheader && [ "$(nvidia-smi -i 1 --query-gpu=memory.free --format=csv,noheader,nounits)" -ge 8000 ]
+    changed_when: "false"
+  - name: .env owners and modes unchanged
+    sudo: true
+    shell: test "$(stat -c '%U %a' {{ mero }}/.env)" = "lkraven 644" && test "$(stat -c '%U %a' {{ vllm }}/.env)" = "lkraven 644" && test "$(stat -c '%U %a' {{ gens }}/.env)" = "root 600"
+    changed_when: "false"
+  - name: all nine fv-ml1 GPU seats healthy
+    shell: for c in vllm-cyberprev vllm-gen-small vllm-voices parakeet-nemo intern-decision vllm-meromero-rp vllm-erp-seat vllm-coder vllm-flash-next; do docker ps -q --filter name=^$c$ --filter health=healthy | grep -q . || { echo "NOT healthy $c"; exit 1; }; done
+    changed_when: "false"

stacks/gen-small-seat/.env.example

@@ -24,8 +24,12 @@ GEN_SMALL_SERVED_NAME_THINK=gen-small-thinking
 # the boot check (Free at start must be >= util x total). Lowering it freed GPU 0 slack for parakeet-nemo
 # at zero KV cost (still 670,142 tokens / 2.56x). Before ANY restart, check the arithmetic:
 # util x 95.6 GiB <= measured Free + this seat's own resident memory. Do not trial-boot.
+# 2026-10-05 (Prime): KV 8 -> 5.5 GiB (~461k tokens, ~1.76x) so parakeet-nemo can stay on GPU 0.
+# The "Free ~1.05 GB at rest" on 10-01 was a just-after-restart read: per Beszel the card plateaus
+# at ~60 MiB free after 1-2 days of creep. 2.5 GiB back covers parakeet's hard cap (MEM_CAP_MIB
+# 3840, i.e. +1,732 MiB over its 2,108 rest) with ~0.9 GB spare at the plateau.
 GEN_SMALL_GPU_MEM_UTIL=0.33
-GEN_SMALL_KV_CACHE_MEMORY=8589934592
+GEN_SMALL_KV_CACHE_MEMORY=5905580032
 GEN_SMALL_MAX_MODEL_LEN=262144
 GEN_SMALL_MAX_NUM_SEQS=16
 GEN_SMALL_MAX_NUM_BATCHED_TOKENS=4096

stacks/gen-small-seat/compose.yaml

@@ -58,7 +58,7 @@ services:
       # avoids the util-fraction fight between co-resident seats. fp8 KV + an A3B's small
       # attention footprint make this cheap; tuned to a sane concurrency band, not max.
       - --kv-cache-memory
-      - "${GEN_SMALL_KV_CACHE_MEMORY:-8589934592}"
+      - "${GEN_SMALL_KV_CACHE_MEMORY:-5905580032}"
       - --max-model-len
       - "${GEN_SMALL_MAX_MODEL_LEN:-262144}"
       - --max-num-seqs

stacks/intern-decision/README.md

@@ -87,6 +87,12 @@ endpoint to hit 32k tokens if possible"). With scriberr gone, GPU 1 holds only t
 service. This container may therefore use its rest (8,812 MiB) plus GPU 1's nvidia-smi `Free` (6,625 MiB), which is
 **15,437 MiB**. The 14.4 GiB cap plus the ~660 MiB outside the allocator puts the card ceiling at ~15,408 MiB.
 
+> ⚠ **2026-10-05:** the 0234 Docker-move restart broke this budget. `vllm-meromero-rp` sized its KV from
+> `--gpu-memory-utilization`, re-profiled during a co-boot, and came back 8.7 GiB larger, so `Free` fell to
+> 3,981 MiB, about 2.4 GB short of a 32k call. Fixed 2131–2135 the same night: meromero and coder now pin KV in
+> bytes (`playbooks/fv-ml1-kv-pin-20261005.yaml`), and `Free` reads **11,801 MiB**, a 20,613 MiB budget, so
+> ~5.4 GB is spare at the 15,220 peak. The static seats can no longer drift across restarts.
+
 Measured on the live service (per-process nvidia-smi every 0.1 s; single `noul` question unless noted; n=3, deterministic):
 
 | call tokens | card peak MiB | wall (warm) |

stacks/meromero-charrp/.env.example

@@ -26,6 +26,15 @@
 # stopped fitting on the 24th": nothing about this seat changed, gen crept up
 # underneath it. Headroom arithmetic done against a long-running gen is
 # measuring a moving number.
+#
+# 2026-10-05: the seat lives on fv-ml1 GPU 1 (MEROMERO_GPU_ID=1 on the host),
+# shared with erp-seat, coder and intern-decision. KV is now PINNED in bytes
+# (Prime: 22 GiB, ~346k tokens, ~1.32x at 262k). Util-sized KV was a co-boot
+# lottery (20.37 / 26.74 / 29.04 GiB on three boots) and the 29.04 boot ate
+# intern-decision's 32k headroom. Util now only gates the boot check, so it
+# sits just above the real footprint (~43.6 GiB). Before ANY restart:
+# util x 95.0 GiB <= measured Free + this seat's own resident memory.
 MEROMERO_GPU_MEM_UTIL=0.47
+MEROMERO_KV_CACHE_MEMORY=23622320128
 MEROMERO_MAX_MODEL_LEN=262144
 MEROMERO_GPU_ID=0

stacks/meromero-charrp/compose.yaml

@@ -64,7 +64,14 @@ services:
       - --max-num-seqs
       - "${MEROMERO_MAX_NUM_SEQS:-32}"
       - --gpu-memory-utilization
-      - "${MEROMERO_GPU_MEM_UTIL:-0.52}"
+      - "${MEROMERO_GPU_MEM_UTIL:-0.47}"
+      # KV pinned in bytes (2026-10-05, Prime: 22 GiB). Util-sized KV was a lottery on
+      # this shared card: vLLM's profile counts co-booting neighbours as its own non-torch
+      # memory, so three boots gave 20.37 / 26.74 / 29.04 GiB of KV. With this set, vLLM
+      # skips profiling and util only gates the boot check (Free at start >= util x total),
+      # so util stays just above the real footprint (~43.6 GiB).
+      - --kv-cache-memory
+      - "${MEROMERO_KV_CACHE_MEMORY:-23622320128}"
       - --kv-cache-dtype
       - fp8
       - --trust-remote-code

stacks/vllm/.env.example

@@ -88,7 +88,9 @@ CODER_MODEL=Qwen/Qwen2.5-Coder-1.5B
 CODER_SERVED_NAME=qwen2.5-coder-1.5b
 CODER_MAX_MODEL_LEN=8192
 CODER_KV_CACHE_DTYPE=fp8
-CODER_GPU_MEM_UTIL=0.06
+CODER_GPU_MEM_UTIL=0.055
+# KV pinned 2026-10-05 (77,056 tokens; see compose.yaml). Util now only gates the boot check.
+CODER_KV_CACHE_MEMORY=1104674816
 CODER_MAX_NUM_SEQS=32
 
 # === LFM2.5-2.6B — RETIRED PERMANENTLY 2026-08-20 (operator directive) ===

stacks/vllm/compose.yaml

@@ -79,6 +79,11 @@ services:
       - "8000"
       - --gpu-memory-utilization
       - ${CODER_GPU_MEM_UTIL}
+      # KV pinned in bytes (2026-10-05): GPU 1 is shared, and util-sized profiling read
+      # negative KV on every co-boot (crash, then a retry). 1,104,674,816 B = 4,816 blocks
+      # = 77,056 tokens, the size every successful boot had landed on.
+      - --kv-cache-memory
+      - ${CODER_KV_CACHE_MEMORY:-1104674816}
       - --max-model-len
       - ${CODER_MAX_MODEL_LEN}
       - --max-num-seqs